diff --git a/quest/m1/subscribe-ranges/README.md b/quest/m1/subscribe-ranges/README.md index 0ef6ae40a02..d3051609410 100644 --- a/quest/m1/subscribe-ranges/README.md +++ b/quest/m1/subscribe-ranges/README.md @@ -102,7 +102,6 @@ every sequence delivered or dropped. ## Required -- [Fetch as a range](/quest/m1/subscribe-ranges/fetch-range.md) - `fetch_group` becomes a one-range subscription that ends once its range is accounted for - [Lite-07 ranges](/quest/m1/subscribe-ranges/lite.md) - the lite-07 wire, Rust publisher and subscriber, and the draft - [JS ranges](/quest/m1/subscribe-ranges/js.md) - `@moq/net` model and lite-07 parity, and the on-demand request surface - [Remove FETCH from lite-07](/quest/m1/subscribe-ranges/lite07-fetch.md) - FETCH removed from lite-07 in Rust, JS, and the draft diff --git a/quest/m1/subscribe-ranges/fetch-range.md b/quest/m1/subscribe-ranges/fetch-range.md deleted file mode 100644 index 36fb438a660..00000000000 --- a/quest/m1/subscribe-ranges/fetch-range.md +++ /dev/null @@ -1,182 +0,0 @@ -# [L] Fetch as a range - -## Goal - -`fetch_group` is gone: every caller reads a past group through a one-range -subscription, which ends once its range is accounted for. That covers the -relay, moq-archive, hls, moq-ffi and every binding wrapper with its `doc/lib` -page, and `moq fetch` through `moq_relay::fetch_group`. - -## Plan - -Split out of dynamic range requests, which replaced `requested_group` with -`requested_fill`, by the maintainer on 2026-10-20 -([moq-dev/moq#6123](https://github.com/moq-dev/moq/pull/6123)), so each PR -breaks one API once. Resized from [M] to [L] on 2026-10-24, when the first -attempt ([moq-dev/moq#6722](https://github.com/moq-dev/moq/pull/6722)) found -the questions decided below and landed only the `track_range_filling` benchmark. -The second attempt ([moq-dev/moq#7545](https://github.com/moq-dev/moq/pull/7545)) -landed the bounded-read end on a plain cursor, the parking wire publishers, -the cached-backfill rule, the bounded miss grace, and the unfed moq-transport -copy; what remains is below. - -- **Ending a bounded read** (decided 2026-10-20). `recv_group` returns `None` - once every sequence in a bounded window is delivered, dropped, stale, or - unfillable. Landed for a track's own cache: a `with_range` cap never ends a - read, since it may rise again. A front's logical track (a routed track, - `model/resume.rs`) still parks, because each copy's reader parks and the - routes judge the end. Its `Delivered` counts what is no longer owed as - handed out (`track::Subscriber::skip_ordered`), so it cannot account for the - window alone, and `resume_model` asserts a reader never ends before a route - declares an end, which ending there has to change. `Ok(None)` before any group arrives is a miss, which the - moq-ffi fetch wrapper reports as `NotFound`; after one, it is the range's - end. A one-range read surfaces no `NotFetchable` or `Old` result, and a - peer's reset with either names the group dropped. Every fetch boundary - already treats `NotFetchable` as not found (relay HTTP 404, hls - `is_cache_miss`, moq-ffi `map_fetch_error`, moq-c `NOT_FOUND`); a fetched - `Old` becomes a miss too, where relay HTTP answers 500 and moq-c reports - `OLD` today. Both stay as wire stream errors, and `Old` still aborts a cut - group mid-read. The Impact bullet says how to migrate each affected package. -- A one-range subscription for a cached live group missing the frames its - range starts at needs a head fill. Frame bounds - ([moq-dev/moq#6129](https://github.com/moq-dev/moq/pull/6129)) hand such a - group out positioned at the asked frame, so reading the missing head returns - `Error::Lagged` rather than the group being skipped (maintainer, - 2026-10-20). Only `fetch_group` requests a head fill today, so the one-range - read has to request it instead of letting `Lagged` reach the caller. - `fill::Ask` already carries a start frame, so the miss judge can count a - headless slot as a hole for a bounded window. -- **Wire publishers keep parking** (landed). Every wire publisher reads its - subscription through `track::Subscriber::parked`: the IETF publisher and - lite publishers before lite-07 keep parking on a bounded downstream - SUBSCRIBE rather than FIN it, and a lite-07 publisher FINs a covered bounded - subscription without SUBSCRIBE_END - ([moq-dev/moq#6807](https://github.com/moq-dev/moq/pull/6807)) until the - explicit end below removes that FIN. -- **Route recovery keeps a frame start.** `resume::Recover` fetches the rest of - a group from frame N on the new route (`group::Fetch::frame_start`), which a - whole-group range cannot say, so a crate-private fetch stays for it. -- **Ends are signaled, not inferred** (maintainer, 2026-11-01, on - [moq-dev/moq#7545](https://github.com/moq-dev/moq/pull/7545)): "we should - have a SUBSCRIBE_END message or something to signal the last group of a - track? we were using QUIC FIN to signal that but that prevents updates in the - future, so we should have a signal instead". The follow-up answers, the same - day: - - "do we need a SUBSCRIBE_COVERED? We have SUBSCRIBE_DROP so the peer can - determine that the bounded range has been received? SUBSCRIBE_END (rename - to FINISH?) would signal the final group of the track maybe, but would be - independent of the bounded range?" No new message: a subscriber judges a - bounded range complete from the groups it received, SUBSCRIBE_DROP, the - Range Starts in SUBSCRIBE_OK, and SUBSCRIBE_END. Drops and groups name - absolute sequences, so the subscriber judges against its latest ranges and - an update in flight changes nothing it has counted; the FIN's race with an - unread update goes away. One gap: before SUBSCRIBE_OK a lite-07 publisher - holds back what it gave up (`TrackRun::skipped`) and answers only once a - group stream resolves the start, so a bounded subscription none of whose - groups it can serve gets neither, and only the covered FIN ends it today. - Fix: once such a subscription is covered, the publisher sends SUBSCRIBE_OK - and then the drops it held. Datagrams are left out: they become their own - track kind, which takes no ranges (maintainer, 2026-11-01). - - "Keep stream open": SUBSCRIBE_END (or FINISH) names the track's final group - and no longer FINs the Subscribe Stream, so an update can still reach an - ended track's history. The subscriber FINs once it wants nothing more, and - the publisher FINs in reply. The covered FIN goes too. - - ✅ Rename SUBSCRIBE_END to SUBSCRIBE_FINISH in lite-07 (maintainer, - 2026-11-01). It matches TRACKS_FINISH and `track::Producer::finish`, and - "END" would read as closing the stream, which it no longer does. - - The settled lite-07 wire design, together: - - SUBSCRIBE_FINISH (formerly SUBSCRIBE_END) names the track's final group and - no longer FINs the Subscribe Stream. - - A covered bounded subscription is not FINed either: the subscriber judges - it complete from the groups it received, SUBSCRIBE_DROP, the Range Starts in - SUBSCRIBE_OK, and SUBSCRIBE_FINISH. - - A publisher that held back SUBSCRIBE_OK (`TrackRun::skipped`) sends it, then - the drops it held, once such a subscription is covered. - - The subscriber FINs the stream once it wants nothing more, and the - publisher FINs in reply. - - Datagrams take no ranges, since they become their own track kind. The - datagram-tracks plan - ([moq-dev/moq#7575](https://github.com/moq-dev/moq/pull/7575)) touches - `lite.md` too, and should take this design and the rename as given. - - This wire half belongs to [lite-07 ranges](/quest/m1/subscribe-ranges/lite.md) - (Rust and the draft) and [JS ranges](/quest/m1/subscribe-ranges/js.md), not - to this quest. It is recorded here while - [moq-dev/moq#7487](https://github.com/moq-dev/moq/pull/7487) rewrites that - quest, and moves there once it lands. -- **The live edge is told, never inferred** (maintainer, 2026-11-01, on - [moq-dev/moq#7545](https://github.com/moq-dev/moq/pull/7545)): "If the relay - doesn't know the live edge, it shouldn't say anything. It should be a - separate message if it's async or the OK should block. NEVER use the cache - as the live edge." The OK blocks, since the wire already works that way: - lite-07 SUBSCRIBE_OK carries Largest Group and waits until each Range's - start resolves, a relay's `TrackRun` waits on its source's start - (`track::Subscriber::poll_start`) before answering, and an idle copy reads - nothing until its route answers. The draft's exception for a start past the - publisher's largest position should say that a relay knows that position - only from its upstream's answer (lite-07 ranges' draft change). A reader - asks fills only below the newest group the live feed produced (decision 2 - on [moq-dev/moq#6123](https://github.com/moq-dev/moq/pull/6123)), so with - nothing produced it waits for the answer that its own demand's upstream - subscription brings. Earlier (maintainer, 2026-10-25): while a track has no - live feed, holes in a bounded range are misses up to its end, and treating - "no group produced" as "no feed" asks too early - (`subscribe_ranges::sparse_ranges_reach_the_publisher_on_lite07` reads - `[0]` on lite-06). moq-archive needs nothing new: its consumer declares the - newest stored sequence with `start_at`. -- **The session marks a copy unfed** (maintainer, 2026-11-01, on - [moq-dev/moq#7545](https://github.com/moq-dev/moq/pull/7545); landed). ✅ - The session marks a copy it holds no upstream subscription for unfed - (`track::Producer::set_unfed`), from its own state, never from the cache. - A bounded read's holes there are misses up to its end at once, per the - 2026-10-25 rule, a refusal is final, and the read takes what the cache holds - in its window even while the copy is idle; an open read still waits for a - feed. The moq-transport subscriber marks a lingering copy unfed, such as one - learned from TRACK_STATUS for a fetch, and keeps a subscription whose ranges - all end below the publisher's reported Largest Object out of the upstream - SUBSCRIBE (`fetch_only.rs`). The rejected alternative, a one-range read always - subscribing so its OK names the edge, broke the fetch-only rule below. -- **Every hole ends** (maintainer, 2026-11-01, on - [moq-dev/moq#7545](https://github.com/moq-dev/moq/pull/7545)): "uhh the - publisher is supposed to emit the start/end bounds if the subscriber didn't - ask for one? I don't understand, but parking forever is terrible?" A hole is - asked as a miss, refused, or covered by the publisher's bounds, and a - bounded read never parks for good. A hole under the start the publisher - resolved is asked at once, and a refusal there is held while the feed may - still deliver the group late (maintainer, 2026-10-21, on - [moq-dev/moq#6171](https://github.com/moq-dev/moq/pull/6171)). Nothing is - inferred from the first arrival, since a local producer may still write a - lower group (`late_lower_group`). Landed: that hold, and the wait before a - hole with no start under it is asked, last the arrival grace, or - `ledger::GRACE` when neither a budget nor a max age bounds it - (`TrackState::miss_grace`), and a held refusal wakes its reader at its - deadline. -- **Cached backfill is owed** (maintainer, 2026-10-25; landed). A cached - backfill group inside a reader's window is taken as filled, whether or not - the live feed may still deliver it, so a relay behaves the same warm or - cold. Re-filling a cached group was rejected: it costs an upstream FETCH for - a group already in the cache. -- **Relay demand before lite-07** (maintainer, 2026-10-25). A one-range read's - ranges join the track's aggregate, so a relay would put a bounded SUBSCRIBE - upstream where `fetch_group` puts a FETCH. `fetch_only.rs` asserts that a - moq-transport fetch never subscribes, and a lite SUBSCRIBE before lite-05 - cannot name a missing group. A range wholly below the live edge stays out of - the upstream SUBSCRIBE on any wire without range SUBSCRIBE and is filled by - FETCH, keeping today's wire behavior: since - [moq-dev/moq#6724](https://github.com/moq-dev/moq/pull/6724) a run of misses - is one range FETCH on moq-transport, and per-group FETCH is lite-only, until - [lite-07 ranges](/quest/m1/subscribe-ranges/lite.md) land. A bounded - upstream SUBSCRIBE accounted by SUBSCRIBE_START and SUBSCRIBE_DROP was - rejected: it works on lite-05 and later only. Landed for a lingering - moq-transport copy whose subscription lies wholly below the reported Largest - Object (the unfed copy above). Still to do: a first request with a bounded - subscription, which SUBSCRIBEs before any TRACK_STATUS names an edge; a - subscription only partly below the edge; and lite before lite-07. A - subscription to a copy no SUBSCRIBE accepts waits for the track's info - today, so this also needs a one-range read to resolve from the info its fill - installs. -- **The fill path goes linear first** (maintainer, 2026-10-25). One-range - reads put every past read on the fill path, which `track_range_filling` - measured as superlinear in range count and span. It is flat since - [moq-dev/moq#6995](https://github.com/moq-dev/moq/pull/6995). diff --git a/quest/m1/subscribe-ranges/js.md b/quest/m1/subscribe-ranges/js.md index c11c3f0db65..3461e56f57e 100644 --- a/quest/m1/subscribe-ranges/js.md +++ b/quest/m1/subscribe-ranges/js.md @@ -17,11 +17,16 @@ the ranges model, anchors, `delay: { max: null }`, and the on-demand request surface. Rust's lite-07 SUBSCRIBE already carries ranges ([moq-dev/moq#6715](https://github.com/moq-dev/moq/pull/6715)). -Mirror lite-07 bounded completion in the publisher: a covered bounded -subscription FINs without SUBSCRIBE_END. The JS subscriber already accepts that -FIN, closing the request's track once its tail settles, and failing it when an -update moved the range and the groups that arrived leave it uncovered. Reopen -that demand on the same track instead, as the Rust copy does. +Mirror lite-07 bounded completion as the Rust copy implements it when this +lands. The target is [Lite-07 ranges](/quest/m1/subscribe-ranges/lite.md)' +"Ends are signaled" design: a covered bounded subscription sends +SUBSCRIBE_FINISH, keeps the Subscribe Stream open, and never FINs it. This +quest does not wait on that one, so if it lands first, the wire still FINs a +covered bounded subscription without SUBSCRIBE_END. The JS subscriber already +accepts that FIN, closing the request's track once its tail settles, and +failing it when an update moved the range and the groups that arrived leave it +uncovered; until SUBSCRIBE_FINISH ships it reopens that demand on the same +track instead, as the Rust copy does. Mirror the Rust names, including `Time(pts)` ends and their refusal on an untimed track. The publisher resolves no relative or time start, serving it from the oldest diff --git a/quest/m1/subscribe-ranges/lite.md b/quest/m1/subscribe-ranges/lite.md index 3a7e229f7d0..c07dfe0af5a 100644 --- a/quest/m1/subscribe-ranges/lite.md +++ b/quest/m1/subscribe-ranges/lite.md @@ -1,8 +1,11 @@ -# [L] Lite-07 ranges +# [XL] Lite-07 ranges ## Goal -A relay fills a lite-07 upstream by subscribing to the missing ranges. FETCH stays on lite-07 until +A relay fills a lite-07 upstream by subscribing to the missing ranges, and +recovers a dropped group with a one-group SUBSCRIBE. SUBSCRIBE_FINISH signals +a track's end and the Subscribe Stream stays open, so one-range reads back to +back on one lite-07 session complete. FETCH stays on lite-07 until [Remove FETCH from lite-07](/quest/m1/subscribe-ranges/lite07-fetch.md) removes it from both implementations. @@ -17,27 +20,27 @@ range fill work without it. Decided 2026-11-01 (maintainer, on [moq-dev/moq#7487](https://github.com/moq-dev/moq/pull/7487)): this quest -waits for [Fetch as a range](/quest/m1/subscribe-ranges/fetch-range.md)'s miss -judging (a miss with no feed frontier, and holes below where a live feed -started) and its cached backfill rule. Without them a range fill is a -functional regression from FETCH, which answers a miss at once. Both landed -in [moq-dev/moq#7545](https://github.com/moq-dev/moq/pull/7545) on -2026-11-02, so this quest is unblocked. Fetch as a range's remaining work, -such as the explicit end signal, doesn't block it. +waited for Fetch as a range's miss judging (a miss with no feed frontier, and +holes below where a live feed started) and its cached backfill rule. Without +them a range fill is a functional regression from FETCH, which answers a miss +at once. Both landed in +[moq-dev/moq#7545](https://github.com/moq-dev/moq/pull/7545) on 2026-11-02. +Fetch as a range finished in +[moq-dev/moq#7625](https://github.com/moq-dev/moq/pull/7625), which kept +`track::Consumer::fetch_group` public as the single-group fetch and moved its +remaining lite-07 work here (maintainer, 2026-11-02): the explicit end, the +back-to-back hang, and the one-group recovery below. ### Range fill through the subscription Lite-07 SUBSCRIBE and SUBSCRIBE_UPDATE carry the range list (`lite::Range`, typed anchors), and the publisher serves it from the model. Older wires map at the codec, and a merged request goes upstream as one span, from group 0 when -its starts mix (`wire_span`). A lite-07 publisher FINs a covered bounded -subscription without SUBSCRIBE_END; the Rust copy settles its tail, keeps the -cached copy, and subscribes again only for demand the covered ranges leave -out. The maintainer has since directed an explicit end signal (SUBSCRIBE_END -or similar) in place of that FIN, since a FIN prevents later updates (on -[moq-dev/moq#7545](https://github.com/moq-dev/moq/pull/7545)). -[Fetch as a range](/quest/m1/subscribe-ranges/fetch-range.md) owns that -design; build the fill on whatever it settles, not on the FIN. +its starts mix (`wire_span`). Today a lite-07 publisher FINs a covered bounded +subscription without SUBSCRIBE_END, and the Rust copy settles its tail, keeps +the cached copy, and subscribes again only for demand the covered ranges leave +out. The explicit end below replaces that FIN; build the fill on it, not on +the FIN. The first attempt ([moq-dev/moq#7487](https://github.com/moq-dev/moq/pull/7487), WIP commit @@ -74,14 +77,14 @@ whole) passed. These failed, and must pass on #7545's miss judging: ask a fill and park (`untimed::a_fetched_untimed_frame_arrives_untimed`, `fetch_cancel::a_refetch_within_the_linger_stays_on_the_relay_lite07`, `subscription_end_integrity::a_session_close_cancels_a_pending_fetch_from_either_side`). - Fetch as a range's miss with no feed frontier covers them. + #7545's miss with no feed frontier covers them. - A missing or expired group fetched with no budget parks where FETCH answered NotFound at once (`rejoin::{rejoined,taken_over}_quiet_catalog_survives_cache_expiry`). - Fetch as a range's holes below where a live feed started cover them. + #7545's holes below where a live feed started cover them. - A relay re-asks upstream for groups it cached, since its covered memory leaves with the downstream demand (`fetch_fanout::sequential_viewers_share_the_relays_fetched_groups`, two - upstream SUBSCRIBEs where FETCH needed one). Fetch as a range's cached + upstream SUBSCRIBEs where FETCH needed one). #7545's cached backfill rule covers it: a relay leaves cached groups out of a bounded upstream range. On lite-07 the test counts SUBSCRIBE streams, not FETCH. @@ -96,10 +99,12 @@ Open questions the attempt raised, each with the attempt's recommendation: datagram; the subscriber already maps that drop onto the fill. This is a wire change. The alternative keeps FETCH for a fill the first answer shows is a datagram, at a round trip and a second path. + Since settled in part: datagrams become their own track kind, which takes + no ranges (maintainer, 2026-11-01; see "Ends are signaled" below). - **Fills from partway through a group.** Route recovery (`resume::Recover`) fills from frame N, which a whole-group range cannot say. Recommended: keep - FETCH on lite-07 for a fill whose first group starts past frame 0, matching - Fetch as a range's crate-private fetch for that case. The alternative asks + FETCH on lite-07 for a fill whose first group starts past frame 0, as + `resume::Recover` does today (`group::Fetch::frame_start`). The alternative asks for the whole group and drops the frames the reader already has. - **One budget caps every range.** Merging the fills into the upstream Subscriber Max Age takes the most tolerant budget, which loosens the live @@ -129,6 +134,92 @@ bounded ranges complete upstream. A hole on a live track counts as dropped once the feed gives up on it, past the arrival grace (#7545's bounded wait, `TrackState::miss_grace`). +### Ends are signaled, not inferred + +The maintainer, 2026-11-01, on +[moq-dev/moq#7545](https://github.com/moq-dev/moq/pull/7545): "we should have +a SUBSCRIBE_END message or something to signal the last group of a track? we +were using QUIC FIN to signal that but that prevents updates in the future, so +we should have a signal instead". Settled the same day: + +- No SUBSCRIBE_COVERED message: a subscriber judges a bounded range complete + from the groups it received, SUBSCRIBE_DROP, the Range Starts in + SUBSCRIBE_OK, and SUBSCRIBE_FINISH. Drops and groups name absolute + sequences, so the subscriber judges against its latest ranges and an update + in flight changes nothing it has counted; the FIN's race with an unread + update goes away. +- "Keep stream open": SUBSCRIBE_FINISH names the track's final group and no + longer FINs the Subscribe Stream, so an update can still reach an ended + track's history. A covered bounded subscription is not FINed either. +- ✅ Rename SUBSCRIBE_END to SUBSCRIBE_FINISH. It matches TRACKS_FINISH and + `track::Producer::finish`, and "END" would read as closing the stream, which + it no longer does. +- Before SUBSCRIBE_OK a lite-07 publisher holds back what it gave up + (`TrackRun::skipped`) and answers only once a group stream resolves the + start, so a bounded subscription none of whose groups it can serve gets + neither, and only the covered FIN ends it today. Once such a subscription is + covered, the publisher sends SUBSCRIBE_OK and then the drops it held. +- The subscriber FINs the stream once it wants nothing more, and the publisher + FINs in reply. +- Datagrams take no ranges, since they become their own track kind + (maintainer, 2026-11-01). The datagram-tracks plan + ([moq-dev/moq#7575](https://github.com/moq-dev/moq/pull/7575)) touches this + quest too, and should take this design and the rename as given. + +The draft's exception for a start past the publisher's largest position should +also say that a relay knows that position only from its upstream's answer, +never from its cache (maintainer, 2026-11-01: "NEVER use the cache as the live +edge"). + +### Back-to-back one-range reads + +Acceptance test (maintainer, 2026-11-02, on +[moq-dev/moq#7625](https://github.com/moq-dev/moq/pull/7625)): on lite-07, +one-range reads of the same track through one session, groups 1 then 2, and +2, 1, 0, each complete, direct and through a relay. Today each hangs on the +later read (on 2, and on 0). Lite-05/06 and moq-transport pass. It also hangs +hls's `a_live_fmp4_import_serves_segments_over_lite07` once hls reads a +segment's groups that way. Reproduce with sequential `one_range` reads in +`range_read.rs`'s fixture. + +The cause is the publisher's end-of-subscription FIN on a covered bounded +subscription, traced with tracing on: + +- When the copy widens `1..2` to `1..3` with SUBSCRIBE_UPDATE while the + publisher FINs the covered `1..2` without reading it, the copy's tail owes + `1..3` (`SubStream::covered`, `updated` set) and waits its grace, which is + `Duration::MAX` with no max delay (`Subscription::budget`). Group 2 never + comes and nothing re-subscribes. +- When the FIN lands first (as on main before + [moq-dev/moq#7625](https://github.com/moq-dev/moq/pull/7625)), the tail + settles, `TrackServe::begin` finds the lingering demand equal to the covered + ranges and keeps `Sub::None`, and the second read's demand never wakes the + serve loop again, so the copy is left with no upstream subscription. + +No SUBSCRIBE_DROP is involved, so the one-group recovery below does not reach +it, and a fresh SUBSCRIBE while group 2 may still be in flight could deliver +it twice. Fix it with the explicit end above: the Subscribe Stream stays open +and the covered FIN goes, so the publisher reads the update and serves or +drops the group. + +### Recovering a dropped group + +Decided 2026-11-02 (maintainer, on +[moq-dev/moq#7625](https://github.com/moq-dev/moq/pull/7625)): "lite-07 can +issue a one group SUBSCRIBE as a FETCH replacement (no max-delay) if an +existing SUBSCRIBE had it dropped", "and only when the group is officially +marked DROP so we don't receive two copies of it". A group a reader still +wants, once an existing subscription names it in SUBSCRIBE_DROP, is recovered +by a new SUBSCRIBE covering only that group with no max delay, never on a +guess, a timeout, or while the group may still arrive. This is lite-07's +FETCH replacement for a dropped group, so that +[Remove FETCH from lite-07](/quest/m1/subscribe-ranges/lite07-fetch.md) can +remove FETCH from Rust and JS together, as decided on 2026-10-25. Tests: a +group still in flight on the existing subscription sends no second SUBSCRIBE, +and a dropped group is recovered exactly once. + +### Draft and checks + Update `drafts/draft-lcurley-moq-lite.md` with each wire change here, including that a subscriber fills missing groups through its ranges. Run `just drafts check` and `just test interop --all`, and the `moq-net-tests` diff --git a/quest/m1/time-addressing/model.md b/quest/m1/time-addressing/model.md index 04f3e4b9fdb..184a1ae9d9b 100644 --- a/quest/m1/time-addressing/model.md +++ b/quest/m1/time-addressing/model.md @@ -51,10 +51,10 @@ a producer would need per-track memory of its last number. [moq-dev/moq#6780](https://github.com/moq-dev/moq/pull/6780); the cut removes its sequence half, leaving a re-created track to keep its PTS moving forward. -- **What it waits on** (decided 2026-10-27, maintainer): only the +- **What it waits on** (decided 2026-10-27, maintainer; Fetch as a range + finished in [moq-dev/moq#7625](https://github.com/moq-dev/moq/pull/7625)): only the subscribe-ranges children that put sequence ranges into the model and the - wire, [Fetch as a range](/quest/m1/subscribe-ranges/fetch-range.md), - [Lite-07 ranges](/quest/m1/subscribe-ranges/lite.md), and + wire, [Lite-07 ranges](/quest/m1/subscribe-ranges/lite.md) and [Delete the old start bookkeeping](/quest/m1/subscribe-ranges/start-bookkeeping.md), not the whole epic. [Drop set](/quest/m1/drop-set.md) lands first too: its runs keyed by group sequence convert mechanically to starts. @@ -117,7 +117,6 @@ Wire: no format change; legacy and lite-07-wip group IDs become start ticks. ## Required -- [Fetch as a range](/quest/m1/subscribe-ranges/fetch-range.md) - the one-range reads this rework moves onto times - [Lite-07 ranges](/quest/m1/subscribe-ranges/lite.md) - the range ends and drops this rework moves onto times - [Delete the old start bookkeeping](/quest/m1/subscribe-ranges/start-bookkeeping.md) - the lite cursor state lives in the model before the cut rekeys it - [Drop set](/quest/m1/drop-set.md) - its dropped and expired runs convert mechanically from sequences to starts diff --git a/quest/m2/intra-refresh/consumer-warmup.md b/quest/m2/intra-refresh/consumer-warmup.md index ea90f7bc11b..06643233e09 100644 --- a/quest/m2/intra-refresh/consumer-warmup.md +++ b/quest/m2/intra-refresh/consumer-warmup.md @@ -55,7 +55,3 @@ The fix is history rather than a withhold rule: ## Required - [Plan: recovery boundaries](/quest/m2/intra-refresh/plan-boundary.md) - K derives from `warmup`, which must be a safe upper bound - -## Related - -- [Fetch as a range](/quest/m1/subscribe-ranges/fetch-range.md) - the range read the history fetch uses diff --git a/rs/moq-c/src/test.rs b/rs/moq-c/src/test.rs index 74b1ecb8b6d..d917cc970f8 100644 --- a/rs/moq-c/src/test.rs +++ b/rs/moq-c/src/test.rs @@ -2192,19 +2192,36 @@ fn raw_track_subscription_options_and_update() { let consume = request_broadcast(origin, path); let frame_cb = Callback::new(); - let ranges = [moq_range { - start: moq_anchor { - kind: moq_anchor_kind::MOQ_ANCHOR_ABSOLUTE as u32, - group: 1, - timestamp: moq_timestamp::default(), + // A read whose ranges all end ends once it read them, so an open range far ahead keeps + // this one open for the update. + let ranges = [ + moq_range { + start: moq_anchor { + kind: moq_anchor_kind::MOQ_ANCHOR_ABSOLUTE as u32, + group: 1, + timestamp: moq_timestamp::default(), + }, + end: moq_anchor { + kind: moq_anchor_kind::MOQ_ANCHOR_ABSOLUTE as u32, + group: 2, + timestamp: moq_timestamp::default(), + }, + end_present: true, }, - end: moq_anchor { - kind: moq_anchor_kind::MOQ_ANCHOR_ABSOLUTE as u32, - group: 2, - timestamp: moq_timestamp::default(), + moq_range { + start: moq_anchor { + kind: moq_anchor_kind::MOQ_ANCHOR_ABSOLUTE as u32, + group: 100, + timestamp: moq_timestamp::default(), + }, + end: moq_anchor { + kind: moq_anchor_kind::MOQ_ANCHOR_ABSOLUTE as u32, + group: 0, + timestamp: moq_timestamp::default(), + }, + end_present: false, }, - end_present: true, - }]; + ]; let subscription = moq_subscription { priority: 5, max_delay_us: 25_000, @@ -2496,12 +2513,20 @@ fn raw_track_relative_update_counts_from_the_track_edge() { publish(b"zero"); let consume = request_broadcast(origin, path); - // Group 0 only: the reader's cap sits below where the track will be. - let first = [moq_range { - start: anchor(moq_anchor_kind::MOQ_ANCHOR_ABSOLUTE, 0), - end: anchor(moq_anchor_kind::MOQ_ANCHOR_ABSOLUTE, 1), - end_present: true, - }]; + // Group 0 only: the reader's cap sits below where the track will be. A read whose + // ranges all end ends once it read them, so an open range far ahead keeps it open. + let first = [ + moq_range { + start: anchor(moq_anchor_kind::MOQ_ANCHOR_ABSOLUTE, 0), + end: anchor(moq_anchor_kind::MOQ_ANCHOR_ABSOLUTE, 1), + end_present: true, + }, + moq_range { + start: anchor(moq_anchor_kind::MOQ_ANCHOR_ABSOLUTE, 100), + end: anchor(moq_anchor_kind::MOQ_ANCHOR_ABSOLUTE, 0), + end_present: false, + }, + ]; let frame_cb = Callback::new(); let consumer = id(unsafe { moq_consume_track( diff --git a/rs/moq-net/MIGRATE.md b/rs/moq-net/MIGRATE.md index 9bcc0b2ec9e..65116fcd78f 100644 --- a/rs/moq-net/MIGRATE.md +++ b/rs/moq-net/MIGRATE.md @@ -39,11 +39,13 @@ A leasing transport must report the eviction there: a paused group stream, and o ### A bounded subscription ends once its ranges are read When every one of a subscription's `ranges` has an end, `track::Subscriber::recv_group` and `track::Ordered::next_group` return `None` once each group in them was handed out, dropped, skipped as stale, or given up as unfillable, instead of waiting for an update to widen the ranges. -This holds for a track read from its own producer's cache. A track read through an origin's routes, such as a remote broadcast's, still waits past its ranges for now, so a reader there stops once it has the groups it asked for rather than waiting for `None`. +This holds for a track read through an origin's routes, such as a remote broadcast's, too: it ends once no route it left may still deliver a group in range. A reader that wants to widen its ranges later keeps one open, such as `Range::groups(n..)`, and narrows its reads with `with_range`, whose cap never ends a read. A group another reader's fetch or fill cached is now handed to every subscriber whose ranges hold it, not only fetches. A missing group the live feed may still deliver is waited for at most one second before it is asked of a fill or named dropped, when neither the readers' `delay.max` nor the track's max age bounds the wait. Before, a reader with `max_delay: None` on a track with no max age waited for it until the track ended. Over moq-transport, a subscription to a copy the session holds no upstream subscription for, such as one only fetched so far, whose ranges all end below the publisher's reported Largest Object, is served by FETCH without subscribing upstream. Each missing group is asked at once and a refusal is final, since nothing else would deliver it. +A first request whose ranges all have an absolute start and end asks TRACK\_STATUS before it subscribes, so one wholly below the Largest Object is served the same way. Since moq-transport publishers here answer only a one-group FETCH, such a read of a run of missing groups from one of them is refused and names the run dropped; keep a range that reaches the Largest Object to have it subscribed instead. +A cached group missing its first frames, such as one a relay joined midway, is handed to a bounded subscription only once a fill answered it from its first frame, and is given up when none can, rather than read as `Error::Lagged`. ### `broadcast::Demand` reads like `track::Demand` diff --git a/rs/moq-net/src/ietf/subscriber.rs b/rs/moq-net/src/ietf/subscriber.rs index de9a5194b45..5316a6109ad 100644 --- a/rs/moq-net/src/ietf/subscriber.rs +++ b/rs/moq-net/src/ietf/subscriber.rs @@ -2885,12 +2885,14 @@ impl Subscriber { // Demand with nobody subscribing (fetches, or a TRACK_STATUS downstream) learns the // track from TRACK_STATUS rather than a SUBSCRIBE: a finished track refuses one, and - // one made only to learn the track races the fetches it was for. + // one made only to learn the track races the fetches it was for. So does a bounded + // subscription, which may read only history below the Largest Object the status + // reports, as a fetch would: `linger` subscribes once it reaches past that. let mut target = match request.subscription() { subscription - if subscription - .as_ref() - .is_none_or(|subscription| subscription.ranges.is_empty()) => + if subscription.as_ref().is_none_or(|subscription| { + subscription.ranges.is_empty() || subscription.start().is_some() && subscription.end().is_some() + }) => { let Some(idle) = self .track_status(&broadcast_path, &track_name, request, &mut group_fetches, &mut gate) diff --git a/rs/moq-net/src/model/delivered.rs b/rs/moq-net/src/model/delivered.rs index ec631e54c53..dd1f1ce79a3 100644 --- a/rs/moq-net/src/model/delivered.rs +++ b/rs/moq-net/src/model/delivered.rs @@ -109,6 +109,15 @@ impl Delivered { gaps } + /// The parts of `groups` none of `taken` handed out, in order. + pub fn gaps_in_all(taken: &[&Self], groups: std::ops::Range) -> Vec> { + let mut gaps = Vec::from([groups]); + for taken in taken { + gaps = gaps.into_iter().flat_map(|gap| taken.gaps(gap)).collect(); + } + gaps + } + /// Count every sequence below `floor` as handed out. pub fn forget_below(&mut self, floor: u64) { let Some(below) = floor.checked_sub(1) else { diff --git a/rs/moq-net/src/model/resume.rs b/rs/moq-net/src/model/resume.rs index 8620b33ca12..f2ce2682f5c 100644 --- a/rs/moq-net/src/model/resume.rs +++ b/rs/moq-net/src/model/resume.rs @@ -588,9 +588,14 @@ impl Copy { if !reader.groups_read { return Retains::No; } - // Not yet bounded, it may still feed a group the new copy never has. + // Not yet bounded, it may still feed a group the new copy never has, unless the reader + // already holds every group of a bounded window. Then it is kept rather than waited + // on, so the read ends, and a widening judges it again. if self.until.is_none() && self.done.is_none() { - return Retains::Wait; + return match reader.holds_window() { + true => Retains::Keep, + false => Retains::Wait, + }; } // Superseded once the serving copy has a group past the bound. if self.since.is_none() @@ -669,6 +674,22 @@ struct Retain<'a> { waiter: &'a kio::Waiter, } +impl Retain<'_> { + /// Whether the reader handed out every group its bounded, resolved window names. + fn holds_window(&self) -> bool { + let Some(end) = self.window.end().filter(|_| !self.window.is_pending()) else { + return false; + }; + self.window + .runs(self.window.floor().unwrap_or(0)) + .take_while(|run| *run.start() < end) + .all(|run| { + let run = *run.start()..run.end().saturating_add(1).min(end); + self.delivered.gaps(run).is_empty() + }) + } +} + /// How a reader still needs a replaced copy; see [`Copy::retains`]. enum Retains { /// It may still deliver a group the reader lacks. @@ -723,20 +744,30 @@ impl Mirror { impl Subscriber { /// Reserve the lowest next group not in `delivered` across copies, without handing it /// out: in arrival order, or in sequence order for `ordered`. + /// Unless it `parks`, a bounded reader ends once its window has nothing left on any copy; + /// see [`Reader::exhausted`]. pub(super) fn poll_peek_group( &mut self, ordered: bool, + parks: bool, delivered: &Delivered, waiter: &kio::Waiter, ) -> Poll>> { let mut reader = self.reader(); loop { - if let Some(result) = reader.poll_lowest(ordered, delivered, waiter) { + if let Some(result) = reader.poll_lowest(ordered, parks, delivered, waiter) { return result; } } } + /// Whether this reader's bounded window has nothing left for the readers that handed out + /// `taken`, the first of which is its own; see [`Reader::exhausted`]. + pub(super) fn exhausted(&mut self, taken: &[&Delivered], waiter: &kio::Waiter) -> bool { + let mut reader = self.reader(); + !reader.let_go(taken[0], waiter) && reader.exhausted(taken, waiter) + } + /// Hand out the group a [`Self::poll_peek_group`] reserved. pub(super) fn take(&mut self, candidate: Candidate) -> Result { self.reader().take(candidate) @@ -1433,6 +1464,7 @@ impl Reader { fn poll_lowest( &mut self, ordered: bool, + parks: bool, delivered: &Delivered, waiter: &kio::Waiter, ) -> Option>>> { @@ -1530,6 +1562,9 @@ impl Reader { } let drained = self.bound_at_end(); let waits = self.let_go(delivered, waiter); + if !parks && !waits && self.exhausted(&[delivered], waiter) { + return Some(Poll::Ready(Ok(None))); + } match self.poll_end(waiter, |copy| copy.done.clone()) { // The serving copy drained, but a copy it replaced may still deliver a group below // its end: the track ends once that copy has. @@ -1538,6 +1573,52 @@ impl Reader { } } + /// Whether a bounded window has nothing left to deliver, once no replaced copy may still + /// deliver a group under its bound. The serving copy reads every group of the window from + /// its resume point, so its cursor judges those: each was handed out (in any of `taken`, or + /// by that copy), dropped, skipped as stale, or given up as unfillable. What lies under its + /// resume point was the copies' it replaced, each let go once it had nothing more, and a + /// backfill still out is not judged yet. A route between copies has nothing to judge. + /// `taken` holds the logical readers' records, the reader's own first. + fn exhausted(&mut self, taken: &[&Delivered], waiter: &kio::Waiter) -> bool { + if self.resumed.is_some() { + return false; + } + let (lowest, cap) = self.groups; + let resumed = self.copies.last().and_then(|copy| copy.floor); + let window = self.window(); + let Some(end) = window.end().filter(|_| !window.is_pending()) else { + return false; + }; + // A group under the serving copy's resume point that the local floor or cap keeps + // out stays owed, as on one cursor, though no copy reads it. + if let Some(resumed) = resumed { + let until = resumed.min(end); + for run in window + .runs(window.floor().unwrap_or(0)) + .take_while(|run| *run.start() < until) + { + let run = *run.start()..run.end().saturating_add(1).min(until); + if Delivered::gaps_in_all(taken, run) + .iter() + .any(|gap| gap.start < lowest || cap.is_some_and(|cap| gap.end > cap)) + { + return false; + } + } + } + match self.copies.last_mut() { + Some(Copy { + generation, + replaced: false, + done: None, + sub: Sub::Ready(sub), + .. + }) if Some(*generation) == self.generation => sub.exhausted_for(taken, waiter), + _ => false, + } + } + /// Whether the track ended, once the serving copy's channel (as `done` reads it) ran /// out. `None` when the route moved since it was read: the caller looks again. fn poll_end( @@ -4069,6 +4150,55 @@ mod test { } } + /// A bounded read whose whole window was handed out ends after a route switch, to a copy + /// with nothing or only a group already read, though the replaced copy is still live. + #[test] + fn a_read_range_ends_after_a_switch_to_a_repeating_route() { + for ordered in [false, true] { + for repeats in [false, true] { + let case = format!("ordered={ordered}, repeats={repeats}"); + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + routes.serve(a.consume()); + let subscription = Subscription::default() + .with_delay(Delay::default().with_max(HOUR)) + .with_range(0..1); + let mut sub = logical + .consume() + .subscribe(subscription) + .now_or_never() + .unwrap() + .unwrap(); + write(&a, 0); + assert_eq!(drain(&mut sub, ordered), [0], "{case}"); + let b = copy(); + if repeats { + write(&b, 0); + } + routes.serve(b.consume()); + let res = match ordered { + true => sub.poll_next_group(&kio::Waiter::noop()), + false => sub.poll_recv_group(&kio::Waiter::noop()), + }; + let res = res.map(|res| res.map(|group| group.map(|group| group.sequence))); + assert!( + matches!(res, Poll::Ready(Ok(None))), + "{case}: the read ends, got {res:?}" + ); + // The replaced copy was kept, so a widening still reads what only it has. + write(&a, 1); + sub.update( + Subscription::default() + .with_delay(Delay::default().with_max(HOUR)) + .with_range(0..2), + ) + .unwrap(); + assert_eq!(drain(&mut sub, ordered), [1], "{case}"); + } + } + } + /// A replaced copy is bounded only once the new copy delivers its first group: a group /// buffered on it must be read before a later group on the serving copy. #[test] diff --git a/rs/moq-net/src/model/resume_model.rs b/rs/moq-net/src/model/resume_model.rs index c9ba43285eb..1f2f2128155 100644 --- a/rs/moq-net/src/model/resume_model.rs +++ b/rs/moq-net/src/model/resume_model.rs @@ -413,6 +413,11 @@ impl Model { }; reader.ranges = range; let _ = reader.control.update(subscription(reader.spec.budget, range)); + // A read that ended at its range's end reads on once the range widens. + if matches!(reader.end, Some(Ok(()))) && !self.instances[reader.instance].ended { + reader.end = None; + reader.carried.clear(); + } self.poll(index)?; } Op::Bounds { reader: index, range } => { @@ -679,12 +684,18 @@ impl Model { ); } Poll::Ready(Ok(None)) => { - if !instance.ended { + // A bounded read ends once its range is read, a local cap holding nothing + // back: what lands in it afterwards is not owed. + if !instance.ended && reader.ranges.is_none() { return Err(fail(index, "the track ended cleanly though no route declared an end")); } + let covers = |group: u64| match instance.ended { + true => reader.covers(group), + false => reader.ranges.is_some_and(|(start, end)| (start..end).contains(&group)), + }; // A group landing below the end afterwards is not owed: a reader stops at it. reader.carried = (0..GROUPS) - .filter(|group| reader.covers(*group) && instance.live().all(|copy| copy.complete(*group))) + .filter(|group| covers(*group) && instance.live().all(|copy| copy.complete(*group))) .collect(); reader.end = Some(Ok(())); } @@ -840,15 +851,25 @@ impl Model { if reader.doomed { return match &reader.end { Some(Err(_)) => Ok(()), - Some(Ok(())) if instance.ended => Ok(()), + Some(Ok(())) if instance.ended || reader.ranges.is_some() => Ok(()), end => Err(fail(index, format!("outlived its instance's routes: {end:?}"))), }; } // What every live route carries in full. let carried = |group: u64| instance.live().all(|copy| copy.complete(group)); + // A read that ended its range owed every group in it, whatever its cap, but an + // ordered one passes what its floor keeps out once it reads past it. + let read_range = matches!(reader.end, Some(Ok(()))) && !instance.ended; + let covers = |group: u64| match read_range { + true => { + reader.ranges.is_some_and(|(start, end)| (start..end).contains(&group)) + && (!reader.spec.ordered || reader.bounds.is_none_or(|(start, _)| group >= start)) + } + false => reader.covers(group), + }; let mut owed: Vec = (0..GROUPS) .filter(|group| reader.floor.is_none_or(|floor| *group > floor)) - .filter(|group| reader.covers(*group) && carried(*group)) + .filter(|group| covers(*group) && carried(*group)) .collect(); if reader.spec.ordered { owed.retain(|group| before.is_none_or(|before| *group > before)); @@ -881,7 +902,7 @@ impl Model { } match (&reader.end, finish && instance.live().next().is_some()) { (None, false) | (Some(Ok(())), true) => {} - (Some(Ok(())), false) if instance.ended => {} + (Some(Ok(())), false) if instance.ended || reader.ranges.is_some() => {} (end, _) => { let _ = write!(problems, "track end {end:?}; "); } @@ -1080,6 +1101,59 @@ fn regressions() { ], finish: false, }, + // An ordered reader whose ranges narrowed to groups only a standby route's head + // fill can complete reads them and ends with the track. + Case { + ops: vec![ + Op::Up { instance: 0, slot: 1 }, + Op::Join { + reader: 1, + spec: spec(false, Budget::Hour, None, false), + }, + Op::Up { instance: 0, slot: 2 }, + Op::Join { + reader: 0, + spec: spec(true, Budget::Hour, None, false), + }, + Op::Ranges { + reader: 0, + range: Some((8, 11)), + }, + Op::Write { + instance: 0, + slot: 1, + group: Some(8), + frames: 1, + }, + ], + finish: true, + }, + // Groups in range under the resume point that the local floor keeps out stay owed, + // so the read does not end once the copies read everything above them. + Case { + ops: vec![ + Op::Join { + reader: 1, + spec: spec(false, Budget::Hour, None, false), + }, + Op::Write { + instance: 0, + slot: 0, + group: Some(6), + frames: 1, + }, + Op::Up { instance: 0, slot: 1 }, + Op::Ranges { + reader: 1, + range: Some((0, 8)), + }, + Op::Bounds { + reader: 1, + range: Some((3, 8)), + }, + ], + finish: false, + }, ]; for case in cases { if let Err(err) = run(&case) { diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index e4b719f2540..b5ee4fdf2ac 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -5313,10 +5313,6 @@ impl Consumer { options, end: frame_end, } = request; - let frames = group::Bounds { - start: options.frame_start, - end: frame_end, - }; // One fetch per calling context, counted here (coalesced upstream work is // still one request served). Independent of `subscriptions` and the viewer @@ -5335,6 +5331,7 @@ impl Consumer { frame_end, priority: options.priority, max_delay: options.max_delay, + judges_drift: true, result: None, hit: Some(Box::new(group)), stats: self.stats.clone(), @@ -5344,7 +5341,6 @@ impl Consumer { let state = self.resolve(None).0; let resume = state.read().routes.clone(); - let mut result = None; // A front's logical track caches nothing: the serving route answers. if let Some(resume) = resume { @@ -5356,6 +5352,7 @@ impl Consumer { frame_end, priority: options.priority, max_delay: options.max_delay, + judges_drift: true, result: None, hit: None, stats: self.stats.clone(), @@ -5370,45 +5367,15 @@ impl Consumer { }); } - // Queue a request only when the group isn't already resolvable from the track - // (cached, aborted, or past-final all resolve through `Fetching::poll` without - // a queue entry). - let (fills, cached) = { - let state = state.read(); - (state.fills.clone(), state.poll_fetch_cached(sequence, frames)) - }; - let unresolved = cached.is_pending(); - // Hold a hit until the caller polls: the held consumer keeps the group wanted, - // so an abandoned fetch filling it can't abort it in between. - let hit = match cached { - Poll::Ready(Ok(group)) => Some(Box::new(group)), - _ => None, - }; - - if unresolved { - result = queue( - &fills, - sequence, - &FetchOptions { - options: options.clone(), - end: frame_end, - }, - ); - } - - kio::Pending::new(Fetching { + kio::Pending::new(Fetching::cached_or_queued( state, - fills, sequence, - frame_start: options.frame_start, - frame_end, - priority: options.priority, - max_delay: options.max_delay, - result, - hit, - stats: self.stats.clone(), - resume: None, - }) + FetchOptions { + options, + end: frame_end, + }, + self.stats.clone(), + )) } /// The sequences of `span` the cache holds, lowest first. @@ -5711,6 +5678,9 @@ pub struct Fetching { priority: i8, // This caller's own age cap. None means only the publisher's max age. max_delay: Option, + // Whether this fetch refuses a group a newer cached one superseded, judged across the + // whole cache. A cursor's head fill leaves it to the cursor, which judges its window. + judges_drift: bool, // The fill this fetch joined; `None` on a cache hit, or when no handler existed to // queue on. Boxed: a miss already allocates, and the handle stays small. result: Option>, @@ -5724,6 +5694,59 @@ pub struct Fetching { resume: Option>, } +impl Fetching { + /// A fetch of `sequence` from the track's own cache, queueing a fill for it only when + /// the cache can't resolve it (cached, aborted, or past-final all resolve through + /// [`Self::poll`] without one). + fn cached_or_queued( + state: kio::Consumer, + sequence: u64, + request: FetchOptions, + stats: stats::Scope, + ) -> Self { + let frames = group::Bounds { + start: request.options.frame_start, + end: request.end, + }; + let (fills, cached) = { + let state = state.read(); + (state.fills.clone(), state.poll_fetch_cached(sequence, frames)) + }; + let result = match cached.is_pending() { + true => queue(&fills, sequence, &request), + false => None, + }; + // Hold a hit until the caller polls: the held consumer keeps the group wanted, + // so an abandoned fetch filling it can't abort it in between. + let hit = match cached { + Poll::Ready(Ok(group)) => Some(Box::new(group)), + _ => None, + }; + Self { + state, + fills, + sequence, + frame_start: request.options.frame_start, + frame_end: request.end, + priority: request.options.priority, + max_delay: request.options.max_delay, + judges_drift: true, + result, + hit, + stats, + resume: None, + } + } + + /// Leave staleness to the caller, a cursor filling a head: it judges the group against + /// its own window when it hands it out, where a newer group outside that window + /// supersedes nothing. + fn unjudged(mut self) -> Self { + self.judges_drift = false; + self + } +} + impl kio::Task for Fetching { type Output = Result; @@ -5762,7 +5785,7 @@ impl kio::Task for Fetching { let state = state.read(); let budget = clamp_max_delay(self.max_delay.unwrap_or(Duration::MAX), state.max_age_bound()); let _ = state.cache.wakes().watch_landing(sequence, waiter); - if state.poll_drifted(sequence, &Servable::ALL, budget, None, waiter) { + if self.judges_drift && state.poll_drifted(sequence, &Servable::ALL, budget, None, waiter) { return Poll::Ready(Err(Error::Old)); } state.cache.clone() @@ -6647,6 +6670,8 @@ struct Cursor { drops: Option>, /// What this cursor misses and asks fills for, once it first reads. misses: Option>, + /// The heads a bounded window asked for, once it first finds a group missing one. + heads: Option>, /// The group read that last parked, answered again without the walk until something /// it parked on fires, once it first parks. idle: Option>, @@ -6679,6 +6704,18 @@ enum Taken { Owed, /// What a fill answered. Filled, + /// A head a bounded window asked for. + Head, +} + +/// The groups a bounded window read missing their head, which it hands out only once a +/// fill answered from their first frame: a whole-group read never starts short. +#[derive(Default)] +struct Heads { + /// Asked, by sequence. + fetching: BTreeMap, + /// Answered and not handed out yet. + filled: BTreeMap, } /// The sequences a [`DropCursor`] reports from: a cursor's span, narrowed to its window. @@ -6882,6 +6919,7 @@ impl Cursor { skipped: Skipped::default(), drops: None, misses: None, + heads: None, idle: None, finished: None, } @@ -7044,10 +7082,10 @@ impl Cursor { } /// Whether a bounded window has nothing left to deliver: every group in its ranges was - /// handed out (`taken`), dropped, skipped as stale, or given up as unfillable. An open - /// window, or one with a fill still out, always has more, and so does one its cap - /// keeps a group out of, since a reader may raise a cap again. - fn is_exhausted(&self, taken: &Delivered) -> bool { + /// handed out (in any of `taken`), dropped, skipped as stale, or given up as unfillable. + /// An open window, or one with a fill still out, always has more, and so does one its + /// cap keeps a group out of, since a reader may raise a cap again. + fn is_exhausted(&self, taken: &[&Delivered]) -> bool { let Some(end) = self.filter.window.end() else { return false; }; @@ -7081,7 +7119,7 @@ impl Cursor { .take_while(|run| *run.start() < end) .all(|run| { let run = *run.start()..run.end().checked_add(1).map_or(end, |next| next.min(end)); - taken.gaps(run).into_iter().all(&accounted) + Delivered::gaps_in_all(taken, run).into_iter().all(&accounted) }) } @@ -7610,7 +7648,9 @@ impl Cursor { /// Whether a fill still owes this cursor a group, so the end of the feed is not its end. fn awaits_fill(&self) -> bool { - self.misses.as_ref().is_some_and(|misses| misses.is_waiting()) || self.filter.window.lookups().next().is_some() + self.misses.as_ref().is_some_and(|misses| misses.is_waiting()) + || self.filter.window.lookups().next().is_some() + || self.heads.as_ref().is_some_and(|heads| !heads.fetching.is_empty()) } /// Find the next group in arrival order, or in sequence order for `ordered`, without @@ -7659,18 +7699,23 @@ impl Cursor { /// [`Self::poll_peek_recv`] once synced, judging against `drift`. fn poll_recv_with(&mut self, mut drift: Drift, waiter: &kio::Waiter) -> Poll>> { + self.poll_heads(waiter); + self.expire_heads(&drift, waiter); loop { // Re-offer what a widening newly covers ahead of the arrival walk, in the // subscription's order: it is the oldest thing still owed. A group the walk has // yet to reach comes again from it, and the subscriber's delivered set drops the // repeat. let owed = match self.next_owed() { - Some(producer) => Some((producer, Taken::Owed)), + Some(producer) => Some((producer.consume(), Taken::Owed)), // A fill answered what the walk never reaches: backfill is not in it. - None => self.next_filled().map(|producer| (producer, Taken::Filled)), + None => match self.next_filled() { + Some(producer) => Some((producer.consume(), Taken::Filled)), + None => self.next_head().map(|group| (group, Taken::Head)), + }, }; let (consumer, taken) = match owed { - Some((producer, taken)) => (producer.consume(), taken), + Some((consumer, taken)) => (consumer, taken), None => { let floor = self.floor(); let polled = self.poll(waiter, |state| { @@ -7680,6 +7725,7 @@ impl Cursor { if polled.is_pending() && self.watch_time(waiter) { self.admit(); drift = ready!(self.poll_drift(&self.servable, waiter))?; + self.expire_heads(&drift, waiter); continue; } let Some((producer, found_index)) = ready!(polled)? else { @@ -7721,22 +7767,12 @@ impl Cursor { // poll walks a whole backlog off rather than handing it out group by group. if ready!(self.poll_stale(&consumer, &drift, waiter))? { self.advance_arrival(consumer.sequence, taken); - self.stale.add(consumer.content()); - if let Some(drops) = &mut self.drops - && let Some(end) = consumer.sequence.checked_add(1) - { - drops.pending.insert(consumer.sequence..end, StreamError::Old); - } - match drift.bounded { - // A re-offered group stale at the bound is final. - true => self.skipped.remove(consumer.sequence), - false => self.skip(consumer.sequence, drift.budget), - } - // Whatever the window is now: narrowed to a bounded one later, it accounts for - // this group by it, even once the group left the cache. - if let Some(end) = consumer.sequence.checked_add(1) { - self.filter.written_off.insert(consumer.sequence..end); - } + self.write_off_stale(consumer.sequence, consumer.content(), &drift); + continue; + } + if self.lacks_head(&consumer) { + self.advance_arrival(consumer.sequence, taken); + self.ask_head(consumer.sequence, waiter); continue; } return Poll::Ready(Ok(Some(Peeked { @@ -7746,6 +7782,27 @@ impl Cursor { } } + /// Write off group `sequence`, which the drift budget gave up on, as dropped `Old`: + /// skipped for a wider budget to offer again, unless the budget is at its bound. + fn write_off_stale(&mut self, sequence: u64, content: stats::Content, drift: &Drift) { + self.stale.add(content); + if let Some(drops) = &mut self.drops + && let Some(end) = sequence.checked_add(1) + { + drops.pending.insert(sequence..end, StreamError::Old); + } + match drift.bounded { + // A re-offered group stale at the bound is final. + true => self.skipped.remove(sequence), + false => self.skip(sequence, drift.budget), + } + // Whatever the window is now: narrowed to a bounded one later, it accounts for this + // group by it, even once the group left the cache. + if let Some(end) = sequence.checked_add(1) { + self.filter.written_off.insert(sequence..end); + } + } + /// Move the arrival cursor past a group it read: from the walk, to `index`; from what /// a widening owes, out of the owed spans. fn advance_arrival(&mut self, sequence: u64, taken: Taken) { @@ -7757,9 +7814,146 @@ impl Cursor { misses.take_filled(sequence); } } + Taken::Head => { + if let Some(heads) = &mut self.heads { + heads.filled.remove(&sequence); + } + } + } + } + + /// Whether `group` is one this cursor hands out only from its first frame, which it + /// lacks: the window is bounded, so it reads whole groups as a fetch would. + fn lacks_head(&self, group: &group::Consumer) -> bool { + self.filter.window.end().is_some() && group.first_frame() > 0 + } + + /// Ask a fill for the head of `sequence`, as a fetch from frame 0 would, taking in an + /// answer already at hand. + fn ask_head(&mut self, sequence: u64, waiter: &kio::Waiter) { + let asking = self.subscription.read().clone(); + let request = FetchOptions { + options: group::Fetch { + priority: asking.priority, + max_delay: asking.delay.max, + frame_start: 0, + }, + end: None, + }; + // The cursor judges the answer against its window when it hands it out. + let mut fetching = + Fetching::cached_or_queued(self.state.clone(), sequence, request, stats::Scope::default()).unjudged(); + let heads = self.heads.get_or_insert_default(); + match kio::Task::poll(&mut fetching, waiter) { + Poll::Pending => { + heads.fetching.insert(sequence, fetching); + } + Poll::Ready(Ok(group)) => { + heads.filled.insert(sequence, group); + } + Poll::Ready(Err(_)) => self.give_up_heads(std::iter::once(sequence..sequence.saturating_add(1))), + } + } + + /// Take in the heads answered, parking `waiter` on those still out, once per read. A + /// head the reader's bounds no longer admit is let go: a widening offers its group again. + fn poll_heads(&mut self, waiter: &kio::Waiter) { + let Some(mut heads) = self.heads.take() else { return }; + let floor = self.floor(); + let admitted = |sequence: &u64| *sequence >= floor && self.admits(*sequence); + heads.fetching.retain(|sequence, _| admitted(sequence)); + heads.filled.retain(|sequence, _| admitted(sequence)); + let mut given_up = Vec::new(); + heads + .fetching + .retain(|sequence, fetching| match kio::Task::poll(fetching, waiter) { + Poll::Pending => true, + Poll::Ready(Ok(group)) => { + heads.filled.insert(*sequence, group); + false + } + Poll::Ready(Err(_)) => { + given_up.push(*sequence..sequence.saturating_add(1)); + false + } + }); + self.heads = Some(heads); + self.give_up_heads(given_up); + } + + /// Write off the heads still asked whose group went stale within this cursor's window, + /// so a bounded read never waits on a fill for a group it would not hand out. The cursor + /// judges what it hands out, so a head fill does not ([`Fetching::unjudged`]). + fn expire_heads(&mut self, drift: &Drift, waiter: &kio::Waiter) { + let Some(heads) = &self.heads else { return }; + if heads.fetching.is_empty() { + return; + } + let asked: Vec = heads.fetching.keys().copied().collect(); + let servable = self.servable.clone(); + let Poll::Ready(Ok(stale)) = self.poll(waiter, |state| { + let stale: Vec<(u64, stats::Content)> = asked + .iter() + .filter_map(|&sequence| { + // Parked first, so a group landing above while this judges wakes it. + let _ = state.cache.wakes().watch_landing(sequence, waiter); + // Judged whether or not the cache still holds the headless copy: an evicted + // one goes stale all the same, and counts as a group of unknown size. + state + .poll_drifted(sequence, &servable, drift.budget, None, waiter) + .then(|| { + let content = match state.lookup.get(&sequence) { + Some(slot) => slot.group.consume().content(), + None => stats::Content { + groups: 1, + ..Default::default() + }, + }; + (sequence, content) + }) + }) + .collect(); + Poll::Ready(Ok(stale)) + }) else { + return; + }; + for (sequence, content) in stale { + if let Some(heads) = &mut self.heads { + heads.fetching.remove(&sequence); + } + self.write_off_stale(sequence, content, drift); + } + } + + /// Give up the groups whose head no fill can serve, as a refused miss is. + fn give_up_heads(&mut self, given_up: impl IntoIterator>) { + let mut given_up = given_up.into_iter().peekable(); + if given_up.peek().is_none() { + return; + } + let misses = self.misses.get_or_insert_with(Box::default); + let mut runs = Vec::new(); + for run in given_up { + misses.give_up(run, &mut runs); + } + if let Some(drops) = &mut self.drops { + for run in runs { + drops.give_up(run); + } } } + /// The next head answered and not handed out yet, in the subscription's order. + fn next_head(&self) -> Option { + let heads = self.heads.as_ref()?; + let mut filled = heads.filled.values(); + match self.subscription.read().order { + Order::Ascending => filled.next(), + Order::Descending => filled.next_back(), + } + .cloned() + } + /// The next datagram inside this subscriber's range, dropping the ones outside it. /// /// The floor, cap and time/group ranges select whole datagram groups here; frame @@ -7803,6 +7997,17 @@ impl Cursor { if !self.seek_pending.is_empty() { self.seek_pending = next.map_or_else(BTreeMap::new, |next| self.seek_pending.split_off(&next)); } + self.forget_heads_below(next); + } + + /// Let go of the heads asked below `next`, which an ordered read never goes back for: + /// one still out would keep the read waiting on a fill it no longer polls. + fn forget_heads_below(&mut self, next: Option) { + if let Some(heads) = &mut self.heads { + let next = next.unwrap_or(u64::MAX); + heads.fetching = heads.fetching.split_off(&next); + heads.filled = heads.filled.split_off(&next); + } } /// The lowest servable group past the last one returned, walking off everything the @@ -7830,6 +8035,7 @@ impl Cursor { waiter: &kio::Waiter, ) -> Poll>> { let end = self.end_sequence; + self.poll_heads(waiter); // Where the gap below the candidate starts: groups stepped over as stale are cached. let mut from = floor; let group = loop { @@ -7894,6 +8100,36 @@ impl Cursor { // until the budget gives up on it. Its landing, its drop, or that verdict wakes us. ready!(self.poll_owed(from..group.sequence, &drift, waiter))?; + // A group missing its head comes from the fill asked for it, in its turn. + let group = match self.lacks_head(&group) { + false => group, + true => { + let sequence = group.sequence; + if self.heads.as_ref().is_none_or(|heads| { + !heads.fetching.contains_key(&sequence) && !heads.filled.contains_key(&sequence) + }) && self + .misses + .as_ref() + .is_none_or(|misses| misses.gone(sequence).is_none()) + { + self.ask_head(sequence, waiter); + } + match self.heads.as_ref().and_then(|heads| heads.filled.get(&sequence)) { + Some(head) => head.clone(), + // Given up: past it, as past any group no fill can serve. + None if self + .misses + .as_ref() + .is_some_and(|misses| misses.gone(sequence).is_some()) => + { + floor = sequence.saturating_add(1); + continue; + } + None => return Poll::Pending, + } + } + }; + // A conviction the budget walked back: the group is handed over after all, // so it must never reach the stale count. self.seek_pending.remove(&group.sequence); @@ -7909,6 +8145,7 @@ impl Cursor { if let Some(misses) = &mut self.misses { misses.take_filled(group.sequence); } + self.forget_heads_below(self.next_sequence); // The delivery commits everything the seek stepped over to reach this group. self.commit_seek_stale(self.next_sequence.unwrap_or(u64::MAX)); group.cache_refresh(); @@ -8144,8 +8381,8 @@ impl Subscriber { /// was handed out, dropped, skipped as stale, or given up as unfillable, rather than /// waiting for an update to widen them: a reader that wants more keeps a range open. A /// [`Self::with_range`] cap never ends a read, since it may rise again, and a group it - /// keeps out stays owed. - /// A track an origin serves from whichever of its routes is up still parks there for now. + /// keeps out stays owed. A track an origin serves from whichever of its routes is up ends + /// the same way, once no route it left may still deliver a group in range. /// /// A group one of [`Subscription::ranges`] starts or ends inside reads only the frames /// they cover there. One missing the frames asked for reads [`Error::Lagged`] rather @@ -8193,7 +8430,7 @@ impl Subscriber { Poll::Pending => { // A read that only skipped since its last hand-out settles its write-offs too. cursor.retire_written_off(&mut self.delivered); - if !self.parks && cursor.is_exhausted(&self.delivered) { + if !self.parks && cursor.is_exhausted(&[&self.delivered]) { return Poll::Ready(Ok(None)); } return Poll::Pending; @@ -8209,13 +8446,25 @@ impl Subscriber { // A replay advances only this copy's cursor, as an ordinary read would. cursor.take(peeked); }, - // A front's bounded read parks past its end for now, as each copy's reader does. Inner::Resume(resume, _) => resume - .poll_peek_group(ordered, &self.delivered, waiter) + .poll_peek_group(ordered, self.parks, &self.delivered, waiter) .map(|res| res.map(|candidate| candidate.map(|candidate| Reserved::Resume(Box::new(candidate))))), } } + /// Whether this route copy's bounded window has nothing left for the logical readers that + /// handed out `logical`: what this copy took itself, wrote off, or gave up counts too. + pub(super) fn exhausted_for(&mut self, logical: &[&Delivered], waiter: &kio::Waiter) -> bool { + let mut taken = Vec::with_capacity(logical.len() + 1); + taken.push(&self.delivered); + taken.extend_from_slice(logical); + match &mut self.inner { + Inner::Plain(cursor) => cursor.is_exhausted(&taken), + // A copy that is another front's logical track judges across its own copies. + Inner::Resume(resume, _) => resume.exhausted(&taken, waiter), + } + } + /// Hand out the group a [`Self::poll_peek_group`] on this subscriber reserved. pub(super) fn take(&mut self, reserved: Reserved) -> Result { let meter = self.stats.meter(); @@ -17728,6 +17977,228 @@ mod test { ); } + /// A bounded read of a live group the feed started partway into asks its head as a fill, + /// and reads the filled copy from its first frame rather than lagging. + #[kio::test] + #[cfg(feature = "async")] + async fn a_bounded_read_fills_the_head_of_a_live_group() { + let producer = track_producer("test", Info::timed()); + let dynamic = producer.requests(); + + let mut live = producer.create_group(group::Info { sequence: 0 }).unwrap(); + live.start_at(1).unwrap(); + live.write_frame(Timestamp::ZERO, bytes::Bytes::from_static(b"delta")) + .unwrap(); + live.finish().unwrap(); + + let mut sub = producer.subscribe(replay().with_range(0..1)); + let reading = kio::clock::sim::spawn(async move { + let mut group = sub.recv_group().await.unwrap().expect("the group"); + let mut frames = Vec::new(); + while let Some(frame) = group.read_frame().await.unwrap() { + frames.push(frame.payload); + } + (frames, sub.recv_group().await.unwrap().is_none()) + }); + let request = kio::clock::timeout(std::time::Duration::from_secs(1), dynamic.requested_fill_group()) + .await + .expect("the head is asked") + .unwrap(); + assert_eq!(request.frame_start(), 0); + let mut head = request.accept(Info::timed()).unwrap(); + for payload in [&b"snapshot"[..], b"delta"] { + head.write_frame(Timestamp::ZERO, bytes::Bytes::from_static(payload)) + .unwrap(); + } + head.finish().unwrap(); + let (frames, ended) = reading.await.unwrap(); + assert_eq!(frames, [&b"snapshot"[..], b"delta"]); + assert!(ended, "the read ends with its range"); + } + + /// A head fill is judged against the reader's window, as the cursor judges a cached + /// group: a newer group outside the window leaves it fresh, even with no max delay. + #[kio::test] + #[cfg(feature = "async")] + async fn a_head_fill_is_judged_against_the_reader_window() { + let producer = track_producer("test", Info::timed()); + let dynamic = producer.requests(); + + let mut live = producer.create_group(group::Info { sequence: 0 }).unwrap(); + live.start_at(1).unwrap(); + live.write_frame(Timestamp::ZERO, bytes::Bytes::from_static(b"delta")) + .unwrap(); + live.finish().unwrap(); + let mut newer = producer.create_group(group::Info { sequence: 1 }).unwrap(); + newer + .write_frame(Timestamp::from_secs(1).unwrap(), bytes::Bytes::from_static(b"x")) + .unwrap(); + newer.finish().unwrap(); + + let mut sub = producer.subscribe(Subscription::default().with_range(0..1)); + let reading = kio::clock::sim::spawn(async move { + let group = sub.recv_group().await.unwrap().map(|group| group.sequence); + (group, sub.recv_group().await.unwrap().is_none()) + }); + let request = kio::clock::timeout(std::time::Duration::from_secs(1), dynamic.requested_fill_group()) + .await + .expect("the head is asked") + .unwrap(); + let mut head = request.accept(Info::timed()).unwrap(); + for payload in [&b"snapshot"[..], b"delta"] { + head.write_frame(Timestamp::ZERO, bytes::Bytes::from_static(payload)) + .unwrap(); + } + head.finish().unwrap(); + assert_eq!(reading.await.unwrap(), (Some(0), true)); + } + + /// A head still asked when a newer group in the reader's window supersedes it is + /// written off, so the bounded read ends without waiting on the fill, whether or not the + /// cache still holds the headless copy. + #[kio::test] + #[cfg(feature = "async")] + async fn a_head_still_asked_goes_stale_within_the_window() { + for evict in [false, true] { + let producer = track_producer("test", Info::timed()); + let dynamic = producer.requests(); + + let mut live = producer.create_group(group::Info { sequence: 0 }).unwrap(); + live.start_at(1).unwrap(); + live.write_frame(Timestamp::ZERO, bytes::Bytes::from_static(b"delta")) + .unwrap(); + live.finish().unwrap(); + + let mut sub = producer.subscribe(Subscription::default().with_range(0..2)); + assert!(sub.recv_group().now_or_never().is_none(), "the head is still asked"); + let request = dynamic.requested_fill_group().now_or_never().unwrap().unwrap(); + assert_eq!(request.sequence(), 0); + + let mut newer = producer.create_group(group::Info { sequence: 1 }).unwrap(); + newer + .write_frame(Timestamp::from_millis(1).unwrap(), bytes::Bytes::from_static(b"x")) + .unwrap(); + newer.finish().unwrap(); + if evict { + producer.modify().unwrap().evict_group(0, Error::Evicted); + } + let group = kio::clock::timeout(std::time::Duration::from_secs(1), sub.recv_group()) + .await + .expect("group 1 timed out") + .unwrap() + .expect("group 1"); + assert_eq!(group.sequence, 1, "evict={evict}"); + let end = kio::clock::timeout(std::time::Duration::from_secs(1), sub.recv_group()) + .await + .unwrap_or_else(|_| panic!("evict={evict}: the read never ended")) + .unwrap(); + assert!(end.is_none(), "evict={evict}: the read ends with group 0 written off"); + drop(request); + } + } + + /// Answered heads go out in the subscription's order, as owed and filled groups do. + #[kio::test] + #[cfg(feature = "async")] + async fn answered_heads_go_out_in_the_subscription_order() { + for (order, expected) in [(Order::Ascending, [0, 1]), (Order::Descending, [1, 0])] { + let producer = track_producer("test", Info::default()); + let dynamic = producer.requests(); + for sequence in 0..2 { + let mut live = producer.create_group(group::Info { sequence }).unwrap(); + live.start_at(1).unwrap(); + live.write_frame(None, bytes::Bytes::from_static(b"delta")).unwrap(); + live.finish().unwrap(); + } + + let mut sub = producer.subscribe(replay().with_order(order).with_range(0..2)); + assert!( + sub.recv_group().now_or_never().is_none(), + "{order:?}: the heads are asked" + ); + let mut requests = Vec::new(); + while let Some(request) = dynamic.requested_fill_group().now_or_never() { + requests.push(request.unwrap()); + } + assert_eq!(requests.len(), 2, "{order:?}: both heads are asked"); + for request in requests { + let mut head = request.accept(Info::default()).unwrap(); + head.write_frame(None, bytes::Bytes::from_static(b"snapshot")).unwrap(); + head.finish().unwrap(); + } + + let mut got = Vec::new(); + for _ in 0..2 { + let group = sub.recv_group().now_or_never().unwrap().unwrap().expect("a head"); + got.push(group.sequence); + } + assert_eq!(got, expected, "{order:?}"); + assert!( + matches!(sub.recv_group().now_or_never(), Some(Ok(None))), + "{order:?}: the read ends" + ); + } + } + + /// A head answered after the reader's ranges moved off its group is not handed out. + #[kio::test] + #[cfg(feature = "async")] + async fn a_head_answered_outside_the_ranges_is_not_handed_out() { + let producer = track_producer("test", Info::default()); + let dynamic = producer.requests(); + + let mut live = producer.create_group(group::Info { sequence: 0 }).unwrap(); + live.start_at(1).unwrap(); + live.write_frame(None, bytes::Bytes::from_static(b"delta")).unwrap(); + live.finish().unwrap(); + framed(&producer, 1, 1).finish().unwrap(); + + let mut sub = producer.subscribe(replay().with_range(0..2)); + let group = sub.recv_group().now_or_never().unwrap().unwrap().unwrap(); + assert_eq!(group.sequence, 1); + let request = dynamic.requested_fill_group().now_or_never().unwrap().unwrap(); + assert_eq!(request.sequence(), 0); + + sub.update(replay().with_range(1..2)).unwrap(); + let mut head = request.accept(Info::default()).unwrap(); + head.write_frame(None, bytes::Bytes::from_static(b"snapshot")).unwrap(); + head.finish().unwrap(); + assert!( + matches!(sub.recv_group().now_or_never(), Some(Ok(None))), + "the narrowed read ends without group 0" + ); + } + + /// An ordered bounded read whose head no fill serves gives that group up and reads on, + /// rather than handing it out to lag. + #[kio::test] + #[cfg(feature = "async")] + async fn a_bounded_read_gives_up_a_group_whose_head_is_refused() { + let producer = track_producer("test", Info::default()); + let dynamic = producer.requests(); + + let mut live = producer.create_group(group::Info { sequence: 0 }).unwrap(); + live.start_at(1).unwrap(); + live.write_frame(None, bytes::Bytes::from_static(b"delta")).unwrap(); + live.finish().unwrap(); + framed(&producer, 1, 1).finish().unwrap(); + + let mut sub = producer.subscribe(replay().with_range(0..2)).ordered(); + let reading = kio::clock::sim::spawn(async move { + let mut read = Vec::new(); + while let Some(group) = sub.next_group().await.unwrap() { + read.push(group.sequence); + } + read + }); + let request = kio::clock::timeout(std::time::Duration::from_secs(1), dynamic.requested_fill_group()) + .await + .expect("the head is asked") + .unwrap(); + request.reject(Error::NotFound); + assert_eq!(reading.await.unwrap(), [1]); + } + /// A refused mid-group end caps none of its group's copies, even one that fit under it. #[kio::test] #[cfg(feature = "async")] diff --git a/rs/moq-net/src/model/track/miss.rs b/rs/moq-net/src/model/track/miss.rs index d06541d3577..5fd4639dd4b 100644 --- a/rs/moq-net/src/model/track/miss.rs +++ b/rs/moq-net/src/model/track/miss.rs @@ -1107,7 +1107,7 @@ impl Misses { } } - fn give_up(&mut self, groups: ops::Range, given_up: &mut GivenUp) { + pub(super) fn give_up(&mut self, groups: ops::Range, given_up: &mut GivenUp) { if groups.is_empty() { return; } diff --git a/rs/moq-net/tests/integration/fetch_only.rs b/rs/moq-net/tests/integration/fetch_only.rs index d1da60a31e4..3e38135fae9 100644 --- a/rs/moq-net/tests/integration/fetch_only.rs +++ b/rs/moq-net/tests/integration/fetch_only.rs @@ -247,11 +247,9 @@ async fn bounded_direct(version: &str) { let remote = resolve(&client).await; let copy = remote.track("video").unwrap(); - copy.info().await.expect("TRACK_STATUS resolves the track"); - // A front's bounded read still parks past its end, as each copy's reader does, so this - // reads the group and stops there. let mut sub = subscribe_history(©).await; assert_eq!(next_frames(&mut sub).await, Some(frames(1)), "{version}"); + assert_eq!(next_frames(&mut sub).await, None, "{version}: the read ends"); assert!( !subscribed.load(Ordering::SeqCst), "{version}: the bounded read subscribed" @@ -260,8 +258,9 @@ async fn bounded_direct(version: &str) { drop((sub, copy, remote, pair, broadcast, client, publisher)); } -/// A copy the session holds no subscription for is unfed: a bounded read of the history -/// below the publisher's Largest Object fetches its holes at once, never subscribing. +/// A bounded read learns the track from TRACK_STATUS, and of the history below the +/// publisher's Largest Object, which a finished track's SUBSCRIBE would refuse, fetches its +/// holes at once, never subscribing. #[kio::test] async fn a_bounded_read_of_history_fetches_without_subscribing() { let mut failures = Vec::new(); @@ -405,13 +404,19 @@ async fn fetched_copy(upstream_version: Version, version: Version) -> FetchedCop async fn history_through_fetched_copy(upstream: Version, version: Version) { let copy = fetched_copy(upstream, version).await; - // A front's bounded read parks past its end, so this reads the group and stops there. + // The fresh session asks TRACK_STATUS first, which the relay answers without an edge, + // so the read SUBSCRIBEs once that answer arrives, after `subscribe` returns. let mut sub = subscribe_history(©.fresh).await; + assert_eq!(next_frames(&mut sub).await, Some(frames(1)), "{upstream} to {version}"); + assert_eq!( + next_frames(&mut sub).await, + None, + "{upstream} to {version}: the read ends" + ); assert!( copy.track.subscription().is_some(), "{upstream} to {version}: answered without subscribing upstream" ); - assert_eq!(next_frames(&mut sub).await, Some(frames(1)), "{upstream} to {version}"); // The copy still serves its other reader. assert_eq!( fetch(©.fetched, 0).await.expect("fetch after the SUBSCRIBE"), diff --git a/rs/moq-net/tests/integration/main.rs b/rs/moq-net/tests/integration/main.rs index 276d64b423e..43d70c43bf5 100644 --- a/rs/moq-net/tests/integration/main.rs +++ b/rs/moq-net/tests/integration/main.rs @@ -46,6 +46,7 @@ mod multi_cdn; mod partial_group; mod quiet_catalog; mod range_anchors; +mod range_read; mod received_rewind; mod rejoin; mod relay_upstream_close; diff --git a/rs/moq-net/tests/integration/range_read.rs b/rs/moq-net/tests/integration/range_read.rs new file mode 100644 index 00000000000..c1253847722 --- /dev/null +++ b/rs/moq-net/tests/integration/range_read.rs @@ -0,0 +1,162 @@ +#![cfg(feature = "async")] + +//! A one-range read is how a past group is fetched: it gets each group of its range, then +//! ends, on every wire, from the publisher directly or through a relay. A range reaching +//! past the live edge ends once the groups up to its end arrive. + +use crate::support; + +use std::time::Duration; + +use moq_net::track::{Delay, Order, Subscription}; +use moq_net::{Hop, Timestamp, Version}; +use support::harness::{MockConnectOptions, connect_mock}; + +const TIMEOUT: Duration = Duration::from_secs(5); + +fn produce_origin(hop: u64) -> moq_net::origin::Producer { + let (producer, driver) = moq_net::origin::Producer::new(moq_net::origin::Config::new(Hop::new(hop).unwrap())); + support::harness::spawn(driver); + producer +} + +fn write_group(track: &mut moq_net::track::Producer, sequence: u64) { + let mut group = track.append_group().unwrap(); + assert_eq!(group.sequence, sequence); + for frame in 0..2u64 { + let at = Timestamp::from_millis(sequence * 100 + frame).unwrap(); + group.write_frame(at, payload(sequence, frame)).unwrap(); + } + group.finish().unwrap(); +} + +fn payload(sequence: u64, frame: u64) -> Vec { + format!("{sequence}-{frame}").into_bytes() +} + +/// Each group a one-range read of `range` gets, read whole, until it ends. +async fn read(sub: &mut moq_net::track::Subscriber) -> Vec { + let mut read = Vec::new(); + while let Some(mut group) = kio::clock::timeout(TIMEOUT, sub.recv_group()) + .await + .expect("the read neither ended nor got a group") + .expect("recv group") + { + let mut frames = Vec::new(); + while let Some(frame) = kio::clock::timeout(TIMEOUT, group.read_frame()) + .await + .expect("read timed out") + .expect("read frame") + { + frames.push(frame.payload.to_vec()); + } + assert_eq!(frames, [payload(group.sequence, 0), payload(group.sequence, 1)]); + read.push(group.sequence); + } + read +} + +/// A live track with groups 0 to 2 written, read on `version` through a relay hop when +/// `relay` is set, as the reader's broadcast and the publisher's track. +async fn fixture( + version: Version, + relay: bool, +) -> (moq_net::broadcast::Consumer, moq_net::track::Producer, impl Sized) { + let publisher = produce_origin(1); + let broadcast = publisher.create_broadcast("bcast").unwrap(); + let mut track = broadcast + .create_track( + "video", + moq_net::track::Info::default().with_timescale(moq_net::Timescale::MILLI), + ) + .unwrap(); + broadcast.announce(Default::default()).unwrap(); + for sequence in 0..3 { + write_group(&mut track, sequence); + } + + let middle = produce_origin(2); + let client = produce_origin(3); + let mut sessions = Vec::new(); + if relay { + let mut options = MockConnectOptions::new(version); + options.server_publish = Some(publisher.consume()); + options.client_subscribe = Some(middle.clone()); + sessions.push(connect_mock(options).await); + } + let mut options = MockConnectOptions::new(version); + options.server_publish = Some(match relay { + true => middle.consume(), + false => publisher.consume(), + }); + options.client_subscribe = Some(client.clone()); + sessions.push(connect_mock(options).await); + + let remote = kio::clock::timeout(TIMEOUT, client.consume().follow("bcast", None).unwrap().available()) + .await + .expect("resolve timeout") + .expect("broadcast resolves"); + (remote, track, (sessions, broadcast, publisher, middle, client)) +} + +fn one_range(groups: std::ops::Range) -> Subscription { + Subscription::default() + .with_range(groups) + .with_delay(Delay::default().with_max(None)) + .with_order(Order::Ascending) +} + +async fn history(version: Version, relay: bool, finished: bool) { + let (remote, track, _held) = fixture(version, relay).await; + if finished { + track.finish().unwrap(); + } + let mut sub = remote.track("video").unwrap().subscribe(one_range(1..2)).await.unwrap(); + assert_eq!(read(&mut sub).await, [1], "{version} relay={relay} finished={finished}"); +} + +/// A range wholly below the live edge reads its group and ends, live or finished. +#[kio::test] +async fn a_range_of_history_reads_its_groups_and_ends() { + let mut failures = Vec::new(); + for version in support::every_version() { + for relay in [false, true] { + for finished in [false, true] { + if kio::clock::sim::spawn(history(version, relay, finished)).await.is_err() { + failures.push((version, relay, finished)); + } + } + } + } + assert!(failures.is_empty(), "failed: {failures:?}"); +} + +async fn straddling(version: Version, relay: bool) { + let (remote, mut track, _held) = fixture(version, relay).await; + let mut sub = remote.track("video").unwrap().subscribe(one_range(1..5)).await.unwrap(); + let reading = kio::clock::sim::spawn(async move { read(&mut sub).await }); + kio::clock::sleep(Duration::from_millis(50)).await; + for sequence in 3..6 { + write_group(&mut track, sequence); + } + // Arrival order: an older wire sends a backlog in no particular order. + let mut read = reading.await.unwrap(); + read.sort_unstable(); + assert_eq!(read, [1, 2, 3, 4], "{version} relay={relay}"); +} + +/// A range from below the live edge to past it reads the history and the live groups up +/// to its end, then ends. Draft-18 has no bounded SUBSCRIBE, so it joins live and the +/// reader stops at its own end. +#[kio::test] +async fn a_range_across_the_live_edge_reads_up_to_its_end() { + let mut failures = Vec::new(); + for version in support::every_version() { + for relay in [false, true] { + if kio::clock::sim::spawn(straddling(version, relay)).await.is_err() { + failures.push((version, relay)); + } + } + } + assert!(failures.is_empty(), "failed: {failures:?}"); +} diff --git a/rs/moq-net/tests/integration/subscribe_ranges.rs b/rs/moq-net/tests/integration/subscribe_ranges.rs index c54ce34b6d6..ed5b2d73e9c 100644 --- a/rs/moq-net/tests/integration/subscribe_ranges.rs +++ b/rs/moq-net/tests/integration/subscribe_ranges.rs @@ -291,29 +291,33 @@ async fn initial_mixed_ranges_keep_the_absolute_history() { async fn draft22_updates_both_bounds_across_cached_groups() { for relay in [false, true] { let mut fixture = range_fixture(relay).await; + // Reaching the live group 5, so the copy subscribes: history alone is fetched. let initial = Subscription::default() - .with_range(2..4) + .with_ranges([Range::groups(2..4), Range::groups(5..6)]) .with_delay(Delay::default().with_max(None)) .with_order(moq_net::track::Order::Ascending); let mut reader = subscribe(&fixture.remote, initial.clone()).await; - let mut seen = [next(&mut reader).await.unwrap(), next(&mut reader).await.unwrap()]; + let mut seen = Vec::new(); + for _ in 0..3 { + seen.push(next(&mut reader).await.unwrap()); + } seen.sort_unstable(); - assert_eq!(seen, [2, 3]); + assert_eq!(seen, [2, 3, 5]); reader.update(initial.clone().with_range(0..6)).unwrap(); let mut got = Vec::new(); - for _ in 0..4 { + for _ in 0..3 { let seen = next(&mut reader).await; got.push(seen.expect("a widened group never arrived")); } got.sort_unstable(); - assert_eq!(got, [0, 1, 4, 5], "relay={relay}"); + assert_eq!(got, [0, 1, 4], "relay={relay}"); reader.update(initial.clone().with_range(4..5)).unwrap(); kio::clock::timeout( TIMEOUT, kio::wait(|waiter| { assert!( - reader.poll_recv_group(waiter).is_pending(), - "the narrowed reader has no unread group" + matches!(reader.poll_recv_group(waiter), std::task::Poll::Ready(Ok(None))), + "the narrowed reader read its range" ); loop { if fixture @@ -349,14 +353,18 @@ async fn draft22_updates_both_bounds_across_cached_groups() { async fn draft22_resolves_cached_time_ends() { for relay in [false, true] { let mut fixture = range_fixture(relay).await; + // Reaching the live group 5, so the copy subscribes: history alone is fetched. let initial = Subscription::default() - .with_range(2..4) + .with_ranges([Range::groups(2..4), Range::groups(5..6)]) .with_delay(Delay::default().with_max(None)) .with_order(moq_net::track::Order::Ascending); let mut reader = subscribe(&fixture.remote, initial.clone()).await; - let mut seen = [next(&mut reader).await.unwrap(), next(&mut reader).await.unwrap()]; + let mut seen = Vec::new(); + for _ in 0..3 { + seen.push(next(&mut reader).await.unwrap()); + } seen.sort_unstable(); - assert_eq!(seen, [2, 3]); + assert_eq!(seen, [2, 3, 5]); let unknown = initial.clone().with_ranges([Range { start: moq_net::track::Anchor::Absolute(0), end: Some(moq_net::track::Anchor::Time(Timestamp::from_millis(10_000).unwrap())), @@ -524,8 +532,9 @@ async fn draft18_updates_narrow_then_widen() { } } -/// Empty ranges retire only demand no other reader holds. A later widening reuses -/// the copy and joins the new live group, without disturbing a sibling reader. +/// Empty ranges end the read and retire only demand no other reader holds. A later +/// widening reuses the copy and joins the new live group, without disturbing a sibling +/// reader. #[kio::test] async fn empty_ranges_retire_the_wire_and_can_widen_again() { for version in ["moq-transport-18", "moq-transport-22"] { @@ -537,7 +546,10 @@ async fn empty_ranges_retire_the_wire_and_can_widen_again() { assert_eq!(next(&mut reader).await, Some(5)); assert_eq!(next(&mut sibling).await, Some(5)); reader.update(live.clone().with_ranges([])).unwrap(); - assert!(reader.poll_recv_group(&kio::Waiter::noop()).is_pending()); + assert!(matches!( + reader.poll_recv_group(&kio::Waiter::noop()), + std::task::Poll::Ready(Ok(None)) + )); let mut group = fixture .track .create_group(moq_net::group::Info { sequence: 6 }) @@ -554,7 +566,10 @@ async fn empty_ranges_retire_the_wire_and_can_widen_again() { kio::clock::timeout( TIMEOUT, kio::wait(|waiter| { - assert!(reader.poll_recv_group(waiter).is_pending()); + assert!(matches!( + reader.poll_recv_group(waiter), + std::task::Poll::Ready(Ok(None)) + )); loop { if fixture.track.subscription().is_none() { return std::task::Poll::Ready(());