From 85a1691c0094bd358e175402854d600cd6a5c399 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 2 Nov 2026 21:43:47 -0800 Subject: [PATCH 01/68] chore(quest): claim quest/m1/drop-set.md Quest-Claim: 70de50bce7f314b6 From 757a0721d2f5dcbb33266b870ec097ab4e765459 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 2 Nov 2026 21:55:36 -0800 Subject: [PATCH 02/68] docs(quest): record drop-set accounting and range-term blockers Co-Authored-By: GPT-6.1 Sol --- quest/m1/drop-set.md | 40 +++++++++++++++++++++++++++++++++++----- 1 file changed, 35 insertions(+), 5 deletions(-) diff --git a/quest/m1/drop-set.md b/quest/m1/drop-set.md index df29ff5092a..cab51ef8499 100644 --- a/quest/m1/drop-set.md +++ b/quest/m1/drop-set.md @@ -168,11 +168,10 @@ the set never outgrows what the pool would hold, with no new constant. An evicted run's sequences become unknown again. `js/net` has no byte pool and gets no new cap: its shared and per-cursor runs expire with the same idle window as the groups they sit between, and expiry applies the same -fold-or-`TooFarBehind` rule as a Rust eviction. A run sits between cached -sequences, so there are at most about one per cached sequence (alternating -drop and arrival), and JS run state grows no faster than the JS cache -itself. The JS byte bound for both is the 32 MiB group tail that -[moq-dev/moq#7411](https://github.com/moq-dev/moq/pull/7411) landed. +fold-or-`TooFarBehind` rule as a Rust eviction. The implementation audit below +found that neither a cached-sequence count nor the existing group-tail limit +bounds drop-only runs; the JS memory policy needs the maintainer's decision +under `Waiting` before implementation. Only a final run folds into a cursor's floor: a delivered run, a run skipped at the track's max age bound (no widening reaches past it), or a @@ -303,6 +302,37 @@ a cancelled fetch below a track's end still holds the end open until the cache expiry scan removes it, so how fast a subscription ends depends on expiry timing. Settle it here, where cancelled and expired runs get a reason. +## Waiting + +- Maintainer: settle the JS run memory policy. The 2026-11-02 implementation + audit found that `js/net/src/group.ts` applies its 32 MiB limit to each + group's frame tail, not to track metadata or cursor runs. A mock-time + probe retained 64, 256, and 1,024 disjoint OLD runs with zero cached groups + and zero elapsed time. Unknown gaps prevent coalescing or folding these + into a cursor's final floor; idle expiry alone does not bound a flood + inside its window. Choose shared byte accounting for JS cache metadata + and runs, with eviction and the planned fold-or-`TooFarBehind` behavior + (recommended; settle its budget and ownership), or a different ownership + or backpressure bound that also covers drop-only tracks. The previous + assertion that every run sits between cached sequences is false when + drops name groups this relay never cached, a required case of this quest. +- Maintainer: settle how a received stale drop identifies its judged range + after SUBSCRIBE_UPDATE changes ranges without changing Max Age. A drop + sent under the old broad range can arrive after the relay requests a + narrower aggregate range; that reader now covers the aggregate but not + the range used for the verdict. Mock-time cursors over groups at 0, 1, + and 2 seconds confirm the difference: at Max Age 0 the open range serves + group 2, whereas a range capped before group 1 serves group 0. Max Age + alone cannot distinguish an old broad-range verdict from a new one. + Choose an explicit range-terms identifier echoed in stale drops and + treat superseded identifiers as unknown without keeping unbounded term + history (recommended; settle the lite-07 fields), or conservatively + qualify drops against all possibly judged ranges and refill whenever + that cannot prove applicability. The latter must specify how to bound + that state and when a reader is guaranteed a prompt verdict. This is + separate from the settled budget tag and from the deferred lite-ranges + design work. No wire or production change has been implemented. + ## Related - [Subscribe ranges](/quest/m1/subscribe-ranges/README.md) - fills the cache miss an evicted run leaves From cf95e8fa4985bf35557827915d5f10eee4c1b3af Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Mon, 2 Nov 2026 21:58:07 -0800 Subject: [PATCH 03/68] docs(quest): align drop-set plan and tests with open decisions Co-Authored-By: GPT-6.1 Sol --- quest/m1/drop-set.md | 87 +++++++++++++++++++++++--------------------- 1 file changed, 45 insertions(+), 42 deletions(-) diff --git a/quest/m1/drop-set.md b/quest/m1/drop-set.md index cab51ef8499..6643e3b4ed3 100644 --- a/quest/m1/drop-set.md +++ b/quest/m1/drop-set.md @@ -12,6 +12,38 @@ waits, and a widening re-offers what its new budget covers. On lite-07 this restores "growing means I want it": a group reported dropped under a narrower budget is served after a widening, as a cold relay would. +## Waiting + +- Maintainer: settle the JS run memory policy. The 2026-11-02 implementation + audit found that `js/net/src/group.ts` applies its 32 MiB limit to each + group's frame tail, not to track metadata or cursor runs. A mock-time + probe retained 64, 256, and 1,024 disjoint OLD runs with zero cached groups + and zero elapsed time. Unknown gaps prevent coalescing or folding these + into a cursor's final floor; idle expiry alone does not bound a flood + inside its window. Choose shared byte accounting for JS cache metadata + and runs, with eviction and the planned fold-or-`TooFarBehind` behavior + (recommended; settle its budget and ownership), or a different ownership + or backpressure bound that also covers drop-only tracks. The previous + assertion that every run sits between cached sequences is false when + drops name groups this relay never cached, a required case of this quest. +- Maintainer: settle how a received stale drop identifies its judged range + after SUBSCRIBE_UPDATE changes ranges without changing Max Age. A drop + sent under the old broad range can arrive after the relay requests a + narrower aggregate range; that reader now covers the aggregate but not + the range used for the verdict. Mock-time cursors over groups at 0, 1, + and 2 seconds confirm the difference: at a zero-millisecond budget + (wire Max Age 1) the open range serves group 2, whereas a range capped + before group 1 serves group 0. Max Age alone cannot distinguish an old + broad-range verdict from a new one. + Choose an explicit range-terms identifier echoed in stale drops and + treat superseded identifiers as unknown without keeping unbounded term + history (recommended; settle the lite-07 fields), or conservatively + qualify drops against all possibly judged ranges and refill whenever + that cannot prove applicability. The latter must specify how to bound + that state and when a reader is guaranteed a prompt verdict. This is + separate from the settled budget tag and from the deferred lite-ranges + design work. No wire or production change has been implemented. + ## Plan Decided 2026-10-27 (maintainer): this runs after the range count cost work, @@ -57,7 +89,8 @@ disposition record to this quest. (`Edge::cap`), so a narrower reader treats the run as unknown and fills it by range, where the upstream judges it under that narrower range. The cost is an extra range request, only for narrower readers, when a - stale drop arrives. + stale drop arrives. This qualification is incomplete when an update + changes the aggregate range; settle the range-terms item under `Waiting`. - A missing run this relay judges itself is dropped for a reader under whose own budget and range `TrackState::drifted` convicts it. `drifted` judges a missing sequence by its successor here, and every sequence in @@ -165,13 +198,11 @@ byte accounting like a group: the Rust `Pool` through a `Charge` at a fixed per-run cost. Under pressure the pool evicts the oldest runs as it evicts groups, and a run older than the cache's retention window is forgotten, so the set never outgrows what the pool would hold, with no new constant. An -evicted run's sequences become unknown again. `js/net` has no byte pool and -gets no new cap: its shared and per-cursor runs expire with the same idle -window as the groups they sit between, and expiry applies the same -fold-or-`TooFarBehind` rule as a Rust eviction. The implementation audit below -found that neither a cached-sequence count nor the existing group-tail limit -bounds drop-only runs; the JS memory policy needs the maintainer's decision -under `Waiting` before implementation. +evicted run's sequences become unknown again. `js/net` has no byte pool; +its run memory policy is open under `Waiting`. Neither a cached-sequence +count nor its existing per-group frame-tail limit bounds drop-only runs. +Expiry and eviction must preserve the same fold-or-`TooFarBehind` behavior +as Rust. Only a final run folds into a cursor's floor: a delivered run, a run skipped at the track's max age bound (no widening reaches past it), or a @@ -222,7 +253,9 @@ Test on mock time through a relay: - A reader whose range is narrower than the relay's aggregate range is not settled by a received stale drop: it fills G by range and gets G or a drop judged under its own range, while a reader covering the aggregate - range is settled at once. + range is settled at once. Add the in-flight broad-range drop after a + narrowing from the range-terms item under `Waiting`; its qualification + remains open. - On lite-07, a cursor that delivered F, reported its neighbour G stale while G stayed cached, and delivered H, gets only G after a widening that covers all three: G is re-offered and F and H are never resent. @@ -234,8 +267,9 @@ Test on mock time through a relay: - A flood of fragmented runs stays within the pool in Rust, evicting the oldest shared runs first, and a sequence whose shared run was evicted is judged locally or filled by range, never waited on indefinitely. In - `js/net` the same flood holds at most one run more than its cached - sequences, and the runs expire with the idle window. + `js/net`, include a drop-only flood with no cached groups and check the + bound chosen under `Waiting`, including eviction or expiry of a + nonfinal cursor run. That memory-policy decision blocks this assertion. - A single run spanning billions of sequences is settled by each cursor in one step, whether dropped, skipped by a wider reader, or evicted and judged again. @@ -302,37 +336,6 @@ a cancelled fetch below a track's end still holds the end open until the cache expiry scan removes it, so how fast a subscription ends depends on expiry timing. Settle it here, where cancelled and expired runs get a reason. -## Waiting - -- Maintainer: settle the JS run memory policy. The 2026-11-02 implementation - audit found that `js/net/src/group.ts` applies its 32 MiB limit to each - group's frame tail, not to track metadata or cursor runs. A mock-time - probe retained 64, 256, and 1,024 disjoint OLD runs with zero cached groups - and zero elapsed time. Unknown gaps prevent coalescing or folding these - into a cursor's final floor; idle expiry alone does not bound a flood - inside its window. Choose shared byte accounting for JS cache metadata - and runs, with eviction and the planned fold-or-`TooFarBehind` behavior - (recommended; settle its budget and ownership), or a different ownership - or backpressure bound that also covers drop-only tracks. The previous - assertion that every run sits between cached sequences is false when - drops name groups this relay never cached, a required case of this quest. -- Maintainer: settle how a received stale drop identifies its judged range - after SUBSCRIBE_UPDATE changes ranges without changing Max Age. A drop - sent under the old broad range can arrive after the relay requests a - narrower aggregate range; that reader now covers the aggregate but not - the range used for the verdict. Mock-time cursors over groups at 0, 1, - and 2 seconds confirm the difference: at Max Age 0 the open range serves - group 2, whereas a range capped before group 1 serves group 0. Max Age - alone cannot distinguish an old broad-range verdict from a new one. - Choose an explicit range-terms identifier echoed in stale drops and - treat superseded identifiers as unknown without keeping unbounded term - history (recommended; settle the lite-07 fields), or conservatively - qualify drops against all possibly judged ranges and refill whenever - that cannot prove applicability. The latter must specify how to bound - that state and when a reader is guaranteed a prompt verdict. This is - separate from the settled budget tag and from the deferred lite-ranges - design work. No wire or production change has been implemented. - ## Related - [Subscribe ranges](/quest/m1/subscribe-ranges/README.md) - fills the cache miss an evicted run leaves From edb3320b7f613a726d66d426b1cdeda9bf64d781 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 13:46:14 -0800 Subject: [PATCH 04/68] feat(net)!: split lite-07 holes into SUBSCRIBE_SKIP and SUBSCRIBE_DROP Lite-07 names sequences never produced in SUBSCRIBE_SKIP and groups that existed in SUBSCRIBE_DROP, which gains Max Age and Terms. SUBSCRIBE and SUBSCRIBE_UPDATE carry an increasing range-terms id. A relay settles a received stale drop only for readers it covers, forgets stale runs when it sends new terms, and a lite-07 widening re-offers what was reported stale. Co-Authored-By: Claude Opus 5.5 --- drafts/draft-lcurley-moq-lite.md | 71 +++- js/net/src/internal.ts | 93 +++++- js/net/src/lite/publisher.test.ts | 16 +- js/net/src/lite/publisher.ts | 78 ++++- js/net/src/lite/subscribe.test.ts | 35 +- js/net/src/lite/subscribe.ts | 156 +++++++-- js/net/src/lite/subscriber.test.ts | 2 +- js/net/src/lite/subscriber.ts | 35 +- js/net/src/lite/version.ts | 21 ++ js/net/src/lite/widened_max_age.test.ts | 56 +++- js/net/src/resume.model.test.ts | 2 +- js/net/src/resume.test.ts | 19 +- js/net/src/resume.ts | 4 +- js/net/src/track.test.ts | 84 ++++- js/net/src/track.ts | 147 +++++++-- quest/m1/drop-set.md | 71 ++-- rs/moq-net/src/fuzz.rs | 4 + rs/moq-net/src/ietf/subscriber.rs | 2 +- rs/moq-net/src/lite/publisher.rs | 175 +++++++--- rs/moq-net/src/lite/subscribe.rs | 230 +++++++++++-- rs/moq-net/src/lite/subscriber.rs | 57 +++- rs/moq-net/src/lite/version.rs | 11 + rs/moq-net/src/model/resume.rs | 16 +- rs/moq-net/src/model/resume_model.rs | 2 +- rs/moq-net/src/model/track.rs | 418 ++++++++++++++++++++---- rs/moq-net/src/model/track/fill.rs | 2 +- rs/moq-net/src/model/track/hole.rs | 52 +++ rs/moq-net/src/model/track/miss.rs | 56 +++- 28 files changed, 1595 insertions(+), 320 deletions(-) create mode 100644 rs/moq-net/src/model/track/hole.rs diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index c1440ba7a00..3241764408b 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -335,9 +335,9 @@ Sent when resetting a stream (RESET_STREAM), or when refusing to receive one (ST | ------- | ------------- | ----------- | | 0x32 | GROUP_TOO_LARGE | The group grew past the publisher's cache budget and was aborted. | | ------- | ------------- | ----------- | -| 0x33 | NOT_FOUND | The requested group, track, or broadcast is not here. | +| 0x33 | NOT_FOUND | The requested group, track, or broadcast is not here. A sequence that never held a group is named by SUBSCRIBE_SKIP instead. | | ------- | ------------- | ----------- | -| 0x34 | OLD | The group was superseded by a newer group and dropped. | +| 0x34 | OLD | The group was superseded by a newer group and dropped. In SUBSCRIBE_DROP with a `Max Age`, a widening past that budget re-offers it. | | ------- | ------------- | ----------- | | 0x35 | EVICTED | The group was dropped under memory pressure. Unlike OLD it was still current, so it can be re-fetched. | | ------- | ------------- | ----------- | @@ -510,11 +510,11 @@ The subscriber needs the track's TRACK_INFO (notably its timescale) to interpret The publisher sends SUBSCRIBE_OK once the start is resolved, and SUBSCRIBE_END as soon as the exclusive final group is known (see [SUBSCRIBE_OK](#subscribe-ok) and [SUBSCRIBE_END](#subscribe-end)). The publisher closes its side of the stream (FIN) after SUBSCRIBE_END, once every Group Stream has finished or been reset. -A subscription whose every Range has an `End` other than None is covered once every Group in its Ranges has been sent as a Group Stream or a datagram, or named by SUBSCRIBE_DROP. +A subscription whose every Range has an `End` other than None is covered once every Group in its Ranges has been sent as a Group Stream or a datagram, or named by SUBSCRIBE_SKIP or SUBSCRIBE_DROP. The publisher MAY then close its side (FIN) without SUBSCRIBE_END, once every Group Stream has finished or been reset, rather than wait for a SUBSCRIBE_UPDATE to widen it. That FIN ends the subscription, not the Track: the subscriber MUST NOT treat it as the Track's end, and opens a new Subscribe Stream to receive more. It can arrive after the subscriber widened the Ranges in a SUBSCRIBE_UPDATE that the publisher never read. -After reading the publisher's FIN, the subscriber MUST close its side (FIN) once its tail accounting has settled: every received Group Stream has been read to its FIN, reset, or dropped, and every sequence in range has arrived, been named by SUBSCRIBE_DROP, or exhausted the subscriber's grace period. +After reading the publisher's FIN, the subscriber MUST close its side (FIN) once its tail accounting has settled: every received Group Stream has been read to its FIN, reset, or dropped, and every sequence in range has arrived, been named by SUBSCRIBE_SKIP or SUBSCRIBE_DROP, or exhausted the subscriber's grace period. A publisher closing its session gracefully MUST wait for the subscriber's FIN or reset on every Subscribe Stream before closing the connection; a transport acknowledgement alone does not confirm the subscriber read the tail. A subscriber MAY cancel by closing its side (FIN); the publisher then closes its side (FIN) without owing SUBSCRIBE_END. Any other subscription stays open until SUBSCRIBE_END, and either endpoint MAY reset the stream at any time. @@ -1248,6 +1248,7 @@ SUBSCRIBE Message { Subscriber Min Age (i) Range Count (i) Range (..) ... + Terms (i) } Range { @@ -1298,7 +1299,7 @@ A value of 0 is a PROTOCOL_VIOLATION, since a subscription that asks for nothing **Range**: A span of the Track to deliver, from its `Start` up to its `End`. -The subscription is the union of its Ranges: the publisher delivers each Group inside any of them once, in `Group Order` across all of them, and accounts with SUBSCRIBE_DROP for every Group it will not deliver. +The subscription is the union of its Ranges: the publisher delivers each Group inside any of them once, in `Group Order` across all of them, and accounts with SUBSCRIBE_SKIP or SUBSCRIBE_DROP for every Group it will not deliver. **Anchor Type**: Which fields follow, and what the Anchor names: @@ -1311,6 +1312,10 @@ Which fields follow, and what the Anchor names: Any other value is a PROTOCOL_VIOLATION. The publisher resolves a Relative or Time Anchor against the Track as it holds it. +**Terms**: +An identifier for the subscription's terms: its Ranges and `Subscriber Max Age`. +Each SUBSCRIBE_UPDATE carries a larger value than the one before it, and a stale [SUBSCRIBE_DROP](#subscribe-drop) echoes the value it was judged under, so the subscriber can tell a verdict on its current terms from one an update has since replaced. + A `Start` is a floor, not a request; `Subscriber Max Age` is the only thing that asks for data. The publisher SHOULD start at the oldest group at or above the floor that [Expiration](#expiration) has not expired, and MUST NOT deliver an older one. A `Subscriber Max Age` of 0 therefore starts at the latest group, since every older group is already stale. @@ -1322,11 +1327,13 @@ An Absolute `Start` past frame 0 delivers its group from that frame; a start res A subscriber MAY name any start frame, as described in [Positions](#positions). An Absolute `End` past frame 0 delivers its group up to that frame. -## SUBSCRIBE_UPDATE +## SUBSCRIBE_UPDATE {#subscribe-update} A subscriber can modify a subscription with a SUBSCRIBE_UPDATE message. A subscriber MAY send multiple SUBSCRIBE_UPDATE messages to update the subscription. Its Ranges replace the previous ones, growing or shrinking the subscription. Lowering a `Start` covers the groups below the old one that the publisher still holds, not only those that arrive later; a group already sent is not sent again. +Growing the subscription asks for what it newly covers: the publisher MUST serve the newly covered Groups it holds and never delivered on this subscription, including Groups it named in a stale SUBSCRIBE_DROP that the new terms no longer convict, and MUST name again, under the new `Terms`, those it cannot serve. +A SUBSCRIBE_UPDATE never resets the subscription, since a reset would race Group Streams already in flight, and shrinking it sends no SUBSCRIBE_SKIP or SUBSCRIBE_DROP. ~~~ SUBSCRIBE_UPDATE Message { @@ -1337,11 +1344,13 @@ SUBSCRIBE_UPDATE Message { Subscriber Min Age (i) Range Count (i) Range (..) ... + Terms (i) Wanted Group (i) } ~~~ See [SUBSCRIBE](#subscribe) for information about each field. +A `Terms` no larger than the previous one only confuses the subscriber's own accounting of stale drops; the publisher echoes the latest value it read. Moving a `Start` forward within a group that is already being delivered does not retract frames the publisher has already sent; it only bounds what the publisher sends from here on. @@ -1553,7 +1562,8 @@ The publisher SHOULD send SUBSCRIBE_END as soon as the exclusive final group is ## SUBSCRIBE_DROP {#subscribe-drop} -A publisher names groups it will not deliver with SUBSCRIBE_DROP on the Subscribe Stream. +A publisher names groups that existed but that it will not deliver with SUBSCRIBE_DROP on the Subscribe Stream: an aborted or reset group, one evicted or expired with no copy left to serve it, or one past the subscriber's budget (OLD). +Sequences the publisher never produced are named by [SUBSCRIBE_SKIP](#subscribe-skip) instead. ~~~ SUBSCRIBE_DROP Message { @@ -1562,20 +1572,57 @@ SUBSCRIBE_DROP Message { Group Start (i) Group End (i) Error Code (i) + Max Age (i) + Terms (i) } ~~~ **Group Start** and **Group End** are the inclusive range of group sequences the publisher gave up. `Group Start = Group End + 1` is an empty range; a larger Group Start is a PROTOCOL_VIOLATION. -**Error Code** is a [stream error code](#error-codes) saying why the groups were given up, as a reset retaining the Group Stream header would: for example NOT_FOUND for a sequence the publisher never had, or OLD for one past its budget. + +**Error Code** is a [stream error code](#error-codes) saying why the groups were given up, as a reset retaining the Group Stream header would: for example OLD for one past the subscriber's budget, or EVICTED for one the publisher no longer holds. A subscriber MUST end the readers of a dropped group it holds with that error, including a group whose stream is still arriving, rather than end it cleanly. -For every sequence in the subscription's range, the publisher MUST send a Group Stream, send a datagram, or name it in SUBSCRIBE_DROP. -This includes sequences it skipped and streams reset before their headers were delivered. +**Max Age**: +The `Subscriber Max Age` a staleness verdict was judged under, in milliseconds plus one, so a verdict on a budget of zero is 1. +0 means the drop does not depend on the subscriber's budget, such as an aborted group or one evicted or expired with no copy left, and no SUBSCRIBE_UPDATE reverses it. + +**Terms**: +The [SUBSCRIBE](#subscribe) `Terms` in effect when the publisher judged the drop. +A stale drop holds only under those terms: a subscriber treats one whose `Terms` an update has since replaced as no verdict, since the update may have moved the Range the verdict was judged against, and a relay asks for such a group again rather than drop it for its own readers. +A relay that received a stale drop settles with it only a reader whose own budget is no wider than its `Max Age` and whose Ranges cover the relay's own, which the verdict was judged against. + +A SUBSCRIBE_UPDATE that widens the terms re-offers the stale drops they no longer convict, as described in [SUBSCRIBE_UPDATE](#subscribe-update). +A Group Stream for a sequence the subscriber already saw dropped, as after such a re-offer, wins over the drop: the subscriber delivers it. + +For every sequence in the subscription's range, the publisher MUST send a Group Stream, send a datagram, or name it in SUBSCRIBE_SKIP or SUBSCRIBE_DROP. +This includes streams reset before their headers were delivered. A reliably delivered reset retaining the Group Stream header accounts for that group instead. -The publisher SHOULD send SUBSCRIBE_DROP as soon as it gives a sequence up, without waiting for SUBSCRIBE_END. +The publisher SHOULD name a sequence as soon as it gives it up, without waiting for SUBSCRIBE_END. A datagram counts as sent even if it is lost; a subscriber SHOULD bound its wait for unaccounted sequences, for example by Subscriber Max Age. +Before moq-lite-07, SUBSCRIBE_DROP carries neither `Max Age` nor `Terms`, its Error Code is application-specific, and it also names the sequences moq-lite-07 names in SUBSCRIBE_SKIP; every such drop is final. + +## SUBSCRIBE_SKIP {#subscribe-skip} + +A publisher names sequences it never produced with SUBSCRIBE_SKIP on the Subscribe Stream: skipped sequence numbers, or ones past a cut. + +~~~ +SUBSCRIBE_SKIP Message { + Type (i) = 0x3 + Message Length (i) + Group Start (i) + Group End (i) +} +~~~ + +**Group Start** and **Group End** are the inclusive range of sequences that never held a group. +`Group Start = Group End + 1` is an empty range; a larger Group Start is a PROTOCOL_VIOLATION. + +A skip is final: no Group will ever exist at those sequences, whatever the terms. +It carries no error and ends no reader, since there is nothing to read; a subscriber stops waiting for those sequences. +A relay forwards it unchanged. + ## FETCH FETCH is sent by a subscriber to request a single group from a track. @@ -1899,6 +1946,8 @@ The `Message Length` describes the payload size on the wire. - Restored SUBSCRIBE_DROP (type 0x2), replacing Stream Count on SUBSCRIBE_END. Every sequence is accounted for by a Group Stream, a datagram, or a drop; a publisher SHOULD send a drop as soon as it gives a sequence up. - SUBSCRIBE_DROP's Error Code is a stream error code, which ends the dropped groups' readers, including one still arriving. - Refuse a SUBSCRIBE_DROP whose Group Start is greater than Group End plus one on every version that carries it. +- Added SUBSCRIBE_SKIP (type 0x3) for sequences the publisher never produced, which ends no reader and is final. SUBSCRIBE_DROP now names only groups that existed. +- Added `Max Age` and `Terms` to SUBSCRIBE_DROP, and `Terms` to SUBSCRIBE and SUBSCRIBE_UPDATE: a stale drop names the budget and the terms it was judged under, holds only under them, and a widening re-offers it. A Group Stream wins over its earlier drop. - SUBSCRIBE_END declares the final boundary while lower groups can still be in flight; the Subscribe Stream FIN follows once every Group Stream has finished or been reset. - A publisher MAY FIN a subscription whose every Range has an End, once each Group in them is sent or dropped, without SUBSCRIBE_END. The FIN ends the subscription, not the Track. - The subscriber FINs its Subscribe Stream after settling its tail; graceful session close waits for that FIN or reset. diff --git a/js/net/src/internal.ts b/js/net/src/internal.ts index 0be942d71c1..adceca6bbbc 100644 --- a/js/net/src/internal.ts +++ b/js/net/src/internal.ts @@ -10,7 +10,7 @@ import type { Dispose, GetPromise, Getter } from "@moq/signals"; import type { Consumer as BroadcastConsumer, Producer as BroadcastProducer } from "./broadcast.ts"; import type { Datagram } from "./datagram.ts"; import type { Delivered } from "./delivered.ts"; -import { toStreamCode } from "./error.ts"; +import { StreamCode, StreamError, toStreamCode } from "./error.ts"; import type { Frame, FrameConsumer, Consumer as GroupConsumer, Producer as GroupProducer } from "./group.ts"; import { isAnonymous, type Route } from "./hop.ts"; import type * as Instance from "./instance.ts"; @@ -175,26 +175,57 @@ export function removeRun(runs: readonly R[], start: bigint, end: ); } -/** A run of sequences given up, with the error they were given up with. */ -export interface DropRun extends Run { +/** + * Why a group sequence will not be delivered. Mirrors the Rust `track::Hole`. + * + * lite-07 carries the two kinds in their own messages, SUBSCRIBE_SKIP and SUBSCRIBE_DROP; + * earlier versions name both in SUBSCRIBE_DROP. + */ +export interface Hole { + /** The error a reader waiting on the sequence ends with: NOT_FOUND for a skip. */ error: Error; + /** Never produced, such as a skipped sequence number: final, and it ends no reader. */ + skip?: boolean; + /** + * The budget in milliseconds a staleness verdict was judged under, undefined for a drop that + * does not depend on it. Only a widening past it reverses the verdict. + */ + stale?: number; +} + +/** A sequence never produced; see {@link Hole.skip}. */ +export function skipHole(): Hole { + return { error: new StreamError(StreamCode.NotFound, { message: "group never produced" }), skip: true }; +} + +/** A group judged stale under `budget` milliseconds. */ +export function staleHole(budget: number): Hole { + return { error: new StreamError(StreamCode.Old, { message: "group is past the budget" }), stale: budget }; +} + +/** A run of sequences given up, with why. */ +export interface DropRun extends Run, Hole {} + +/** Whether two holes say the same thing, so their runs may merge. */ +function sameHole(a: Hole, b: Hole): boolean { + return toStreamCode(a.error) === toStreamCode(b.error) && !a.skip === !b.skip && a.stale === b.stale; } /** - * Sorted, disjoint `drops` with `[start, end)` given up with `error`, which replaces whatever - * it overlaps. Neighbours given up with the same stream code merge, so a wide DROP stays one - * run and a peer naming one sequence per DROP cannot grow the runs. + * Sorted, disjoint `drops` with `[start, end)` given up as `hole` says, which replaces whatever + * it overlaps. Neighbours given up alike merge, so a wide DROP stays one run and a peer naming + * one sequence per DROP cannot grow the runs. */ -export function insertDrop(drops: readonly DropRun[], start: bigint, end: bigint, error: Error): DropRun[] { +export function insertDrop(drops: readonly DropRun[], start: bigint, end: bigint, hole: Hole): DropRun[] { if (start >= end) return [...drops]; const next = removeRun(drops, start, end); let at = 0; while (at < next.length && next[at].start < start) at++; - next.splice(at, 0, { start, end, error }); + next.splice(at, 0, { start, end, error: hole.error, skip: hole.skip, stale: hole.stale }); const merged: DropRun[] = []; for (const run of next) { const last = merged[merged.length - 1]; - if (last && last.end === run.start && toStreamCode(last.error) === toStreamCode(run.error)) last.end = run.end; + if (last && last.end === run.start && sameHole(last, run)) last.end = run.end; else merged.push({ ...run }); } return merged; @@ -226,7 +257,7 @@ export type Recv = /** A group to serve. It has already left the buffer, so dropping this is dropping the group. */ | { kind: "group"; group: GroupConsumer } /** Sequences `[start, end)` this cursor gave up without serving, and why. */ - | { kind: "drop"; start: bigint; end: bigint; error: Error } + | ({ kind: "drop"; start: bigint; end: bigint } & Hole) /** Nothing readable, but the track is live and may produce more. */ | { kind: "idle" } /** The producer finished, yet groups above the cap are still held: raising it releases them. */ @@ -454,9 +485,14 @@ export const hooks: { * Record that the publisher gave up the groups `[start, end)` with `error`: none will * arrive, so a reader waiting on one moves past it, a group still arriving in them ends with * `error`, the declared end does not wait for them, and a relay serving this track forwards - * the drop. Mirrors the Rust `drop_groups`; assigned by `track.ts`. + * the drop. A skip ends no reader. Mirrors the Rust `give_up`; assigned by `track.ts`. */ - dropTrackGroups: (producer: Producer, start: bigint, end: bigint, error: Error) => void; + dropTrackGroups: (producer: Producer, start: bigint, end: bigint, hole: Hole) => void; + /** + * Forget every run a received copy holds as stale: the terms its upstream judged them under + * were replaced, so they are unknown again. Mirrors the Rust `supersede_stale`. + */ + supersedeStale: (producer: Producer) => void; /** Mark a track as fed by a wire subscription, which owes missing groups. Assigned by `track.ts`. */ receiveTrack: (producer: Producer) => void; /** Create a received group at its wire offset before publishing it to readers. Assigned by `track.ts`. */ @@ -592,6 +628,24 @@ export const hooks: { * below where SUBSCRIBE_START resolved never goes out, so a wider budget may still serve it. */ unreport: (subscriber: Subscriber, start: bigint, end: bigint) => void; + /** + * The budget in milliseconds a subscriber judges staleness under: its max delay, clamped to the + * track's max age. Mirrors the Rust crate-private `track::Subscriber::budget`. + */ + budget: (subscriber: Subscriber) => number; + /** + * Have a widening re-offer what a subscriber reported stale through dispositions, as a wire + * whose drops a widening reverses does. Returns the subscriber. Mirrors the Rust + * crate-private `track::Subscriber::reoffers_reported`. + */ + reoffersReported: (subscriber: Subscriber) => Subscriber; + /** The runs a subscriber reported stale that a widening took back since the last call. */ + takeUnreported: (subscriber: Subscriber) => Run[]; + /** + * Whether a subscriber reads a copy received from a remote publisher, which cannot tell a group + * its source never produced from one it never received. A resumed reader always does. + */ + isReceived: (subscriber: Subscriber) => boolean; /** Whether this subscriber's budget gives up on `sequence`, judged by its successor; see {@link expireGroup}. */ staleAt: (subscriber: Subscriber, sequence: bigint) => Expiry; /** A group the subscriber's track still caches, as a fresh read handle. */ @@ -807,6 +861,21 @@ export const hooks: { unreport: () => { throw new Error("track.ts not loaded"); }, + supersedeStale: () => { + throw new Error("track.ts not loaded"); + }, + budget: () => { + throw new Error("track.ts not loaded"); + }, + reoffersReported: () => { + throw new Error("track.ts not loaded"); + }, + takeUnreported: () => { + throw new Error("track.ts not loaded"); + }, + isReceived: () => { + throw new Error("track.ts not loaded"); + }, tryRecvDatagram: () => { throw new Error("track.ts not loaded"); }, diff --git a/js/net/src/lite/publisher.test.ts b/js/net/src/lite/publisher.test.ts index 1a311ecfec6..6a2c192fbb5 100644 --- a/js/net/src/lite/publisher.test.ts +++ b/js/net/src/lite/publisher.test.ts @@ -1014,7 +1014,7 @@ for (const version of [Version.DRAFT_06, Version.DRAFT_07]) { expect(await sub.servedSequence()).toBe(0n); expect("start" in (await decodeSubscribeResponse(sub.client.reader, version))).toBe(true); - hooks.dropTrackGroups(sub.track, 1n, 3n, new StreamError(StreamCode(70), { message: "cut" })); + hooks.dropTrackGroups(sub.track, 1n, 3n, { error: new StreamError(StreamCode(70), { message: "cut" }) }); const resp = await decodeSubscribeResponse(sub.client.reader, version); if (!("drop" in resp)) throw new Error("expected SUBSCRIBE_DROP"); expect([resp.drop.start, resp.drop.end, resp.drop.error]).toEqual([ @@ -1027,7 +1027,7 @@ for (const version of [Version.DRAFT_06, Version.DRAFT_07]) { } }); - test(`lite ${version}: the end sweep's SUBSCRIBE_DROP carries ${version === Version.DRAFT_07 ? "NOT_FOUND" : "no code"}`, async () => { + test(`lite ${version}: the end sweep names a group never produced ${version === Version.DRAFT_07 ? "in SUBSCRIBE_SKIP" : "in SUBSCRIBE_DROP"}`, async () => { const sub = await servedSubscription({ version, startGroup: 5n }); try { sub.serve(5n); @@ -1037,12 +1037,20 @@ for (const version of [Version.DRAFT_06, Version.DRAFT_07]) { sub.track.finish(); sub.track.close(); const codes: number[] = []; + const skips: bigint[][] = []; for (;;) { const resp = await decodeSubscribeResponseMaybe(sub.client.reader, version); if (!resp) break; if ("drop" in resp) codes.push(resp.drop.error); + if ("skip" in resp) skips.push([resp.skip.start, resp.skip.end]); + } + // The track's own publisher never produced groups 2 through 4. + if (version === Version.DRAFT_07) { + expect(codes).toEqual([]); + expect(skips).toEqual([[2n, 4n]]); + } else { + expect(codes).toEqual([0]); } - expect(codes).toEqual([version === Version.DRAFT_07 ? StreamCode.NotFound : 0]); } finally { await sub.close(); } @@ -1367,7 +1375,7 @@ for (const skipped of ["failed", "stale"] as const) { group.writeFrame(textFrame("old", Timestamp.fromMillis(5))); if (skipped === "failed") group.abort(new Error("cut")); else group.finish(); - if (skipped === "failed") hooks.dropTrackGroups(sub.track, 7n, 8n, new Error("cut")); + if (skipped === "failed") hooks.dropTrackGroups(sub.track, 7n, 8n, { error: new Error("cut") }); sub.serve(6n); const response = await decodeSubscribeResponse(sub.client.reader, Version.DRAFT_06); if (!("start" in response)) throw new Error("expected SUBSCRIBE_START"); diff --git a/js/net/src/lite/publisher.ts b/js/net/src/lite/publisher.ts index 0b084e85f15..9aea954f290 100644 --- a/js/net/src/lite/publisher.ts +++ b/js/net/src/lite/publisher.ts @@ -20,7 +20,19 @@ import { import { stored } from "../flate/group.ts"; import type * as group from "../group.ts"; import { Cost, type Hop, type Route, routesEqual } from "../hop.ts"; -import { finalSequence, hiddenBelow, hooks, insertRun, presented, type Run, runGaps } from "../internal.ts"; +import { + type DropRun, + finalSequence, + type Hole, + hiddenBelow, + hooks, + insertRun, + presented, + type Run, + removeRun, + runGaps, + skipHole, +} from "../internal.ts"; import type { Consumer as OriginConsumer } from "../origin.ts"; import * as Path from "../path.ts"; import { readSize, sendPayload } from "../payload.ts"; @@ -47,6 +59,8 @@ import { SubscribeDrop, SubscribeEnd, SubscribeOk, + type SubscribeResponse, + SubscribeSkip, SubscribeStart, SubscribeUpdate, } from "./subscribe.ts"; @@ -64,6 +78,7 @@ import { hasProbeRtt, hasRanges, hasRouteCost, + hasSkip, hasSynced, hasTrackStream, resolvesStart, @@ -919,6 +934,8 @@ export class Publisher { } // Parked past a bounded window's end: the peer reads this stream's end as the track's. const track = hooks.parked(consumer.subscribe(subscription(ranges))); + // A lite-07 widening re-offers what was reported stale; earlier versions' drops are final. + if (hasSkip(this.version)) hooks.reoffersReported(track); serving(track); if (timed) { try { @@ -1046,6 +1063,7 @@ export class Publisher { served, floor, bounds, + terms: msg.terms, }); console.debug(`publish done: broadcast=${msg.broadcast} track=${track.name}`); @@ -1211,9 +1229,12 @@ export class Publisher { // The read cursor's floor; see {@link cursorFloor}. floor: bigint; bounds: FrameBounds; + /** The SUBSCRIBE's range terms id, which a stale drop echoes until an update replaces it. */ + terms: bigint; }, ): Promise { const { sub, broadcast, stamp, publisher, compress, served, bounds } = serving; + let terms = serving.terms; // The read cursor's floor; the subscription's ranges cap it. let floor = serving.floor; const moveCursor = () => track.withRange({ start: { included: floor } }); @@ -1228,7 +1249,7 @@ export class Publisher { const emitRange = hasTrackStream(this.version); let startSent = false; // Dispositions before START refine its boundary; later ones must be named as drops. - const skipped: { start: bigint; end: bigint; error: Error }[] = []; + const skipped: DropRun[] = []; // Where SUBSCRIBE_START resolved, lowered by an update that asks for groups below it. let startSequence: bigint | undefined; // The first group and the edge seen before it was popped, held until the source @@ -1236,15 +1257,21 @@ export class Publisher { let first: { group: group.Consumer; largest: track.Position | undefined } | undefined; let endSent = false; - // Drop whatever part of `[start, end)` this subscription has not been served or dropped - // yet, given up with `err`. - const drop = async (start: bigint, end: bigint, err: Error): Promise => { + // The message naming `[start, end]` given up as `hole` says: lite-07 skips what was never + // produced, and earlier versions drop it as not found. + const holeMessage = (start: bigint, end: bigint, hole: Hole): SubscribeResponse => + hole.skip && hasSkip(this.version) + ? { skip: new SubscribeSkip({ start, end }) } + : { drop: SubscribeDrop.given(start, end, hole.error, this.version, { maxAge: hole.stale, terms }) }; + + // Give up, as `hole` says, whatever part of `[start, end)` this subscription has not been + // served or given up yet. + const drop = async (start: bigint, end: bigint, hole: Hole): Promise => { if (!served) return true; for (const gap of runGaps(served, start, end)) { insertRun(served, gap.start, gap.end); - const msg = SubscribeDrop.given(gap.start, gap.end - 1n, err, this.version); - if (!(await controls.response(encodeSubscribeResponse(stream, { drop: msg }, this.version)))) - return false; + const msg = holeMessage(gap.start, gap.end - 1n, hole); + if (!(await controls.response(encodeSubscribeResponse(stream, msg, this.version)))) return false; } return true; }; @@ -1313,7 +1340,7 @@ export class Publisher { // What was given up below the start is never named, so a wider budget may serve it. if (range.start < startSequence) hooks.unreport(track, range.start, min(range.end, startSequence)); - if (!(await drop(max(range.start, startSequence), range.end, range.error))) return false; + if (!(await drop(max(range.start, startSequence), range.end, range))) return false; } skipped.length = 0; } @@ -1340,6 +1367,7 @@ export class Publisher { throw control.error; case "update": { const update = control.update; + terms = update.terms; console.debug(`subscribe update: broadcast=${broadcast} track=${track.name}`); // `cursorFloor` pins an unfloored pre-06 SUBSCRIBE to the latest // group, so an update that drops a named floor re-pins there too. On @@ -1393,12 +1421,18 @@ export class Publisher { ? ({ kind: "group", group: first.group } as const) : hooks.tryRecvGroup(track, emitRange); first = undefined; + // A stale drop a lite-07 widening reversed is owed again: re-offered if still held, + // and dropped again under the new terms otherwise. + if (served) { + for (const run of hooks.takeUnreported(track)) + served.splice(0, served.length, ...removeRun(served, run.start, run.end)); + } switch (recv.kind) { case "error": throw recv.error; case "drop": - if (!startSent) skipped.push({ start: recv.start, end: recv.end, error: recv.error }); - else if (!(await drop(recv.start, recv.end, recv.error))) return false; + if (!startSent) skipped.push(recv); + else if (!(await drop(recv.start, recv.end, recv))) return false; continue; case "idle": // A start past everything the track has (a subscriber resuming just after @@ -1485,7 +1519,11 @@ export class Publisher { // Name every sequence from the start to the end this subscription never // got. A cap below the end bounds what the subscriber is owed, as on its side. if (startSequence !== undefined) { - const unserved = new StreamError(StreamCode.NotFound, { message: "group not served" }); + // The track's own publisher knows a group it never reported is one it never + // produced; a relay cannot tell what its source held. + const unserved: Hole = hooks.isReceived(track) + ? { error: new StreamError(StreamCode.NotFound, { message: "group not served" }) } + : skipHole(); // Only the groups inside the ranges are owed. for (const run of hooks.servingRuns(track, startSequence, boundary())) { if (!(await drop(run.start, run.end, unserved))) return false; @@ -1503,6 +1541,13 @@ export class Publisher { } const group = recv.group; + // Never a second stream for a sequence this subscription already accounted for + // (served, sent as a datagram, or dropped): before lite-07 a subscriber settles a + // drop for good. A stale drop a lite-07 widening reversed is owed again above. + if (served && runGaps(served, group.sequence, group.sequence + 1n).length === 0) { + group.close(); + continue; + } const range = frameRange(bounds, group.sequence); if (emitRange && !startSent) { @@ -1559,7 +1604,14 @@ export class Publisher { const task = opened.then(async (writer) => { const failed = writer instanceof Error ? writer : await this.#serveGroup(writer, options); if (emitRange && failed instanceof Error) { - pendingDrops.push(SubscribeDrop.given(group.sequence, group.sequence, failed, this.version)); + // A stream expired under this subscription's budget is a stale verdict. + const maxAge = toStreamCode(failed) === StreamCode.Old ? hooks.budget(track) : undefined; + pendingDrops.push( + SubscribeDrop.given(group.sequence, group.sequence, failed, this.version, { + maxAge, + terms, + }), + ); drops.update((revision) => revision + 1); } }); diff --git a/js/net/src/lite/subscribe.test.ts b/js/net/src/lite/subscribe.test.ts index 4f6f93ba313..bb0c4895516 100644 --- a/js/net/src/lite/subscribe.test.ts +++ b/js/net/src/lite/subscribe.test.ts @@ -13,6 +13,7 @@ import { SubscribeEnd, SubscribeOk, type SubscribeResponse, + SubscribeSkip, SubscribeStart, SubscribeUpdate, toWireRanges, @@ -195,10 +196,10 @@ test("lite-07 carries the span as its only range, exclusive at the end", async ( test("a lite-07 list decodes as the span covering it", async () => { // Priority, order, max delay, and an unstated min delay as SubscribeUpdate writes them, - // then the ranges and an absent wanted group. + // then the ranges, the terms, and an absent wanted group. const head = (await encodeMessage(Version.DRAFT_07, new SubscribeUpdate({ priority: 0 }))).slice(1, 5); const update = (ranges: number[]) => { - const body = [...head, ...ranges, 0]; + const body = [...head, ...ranges, 0, 0]; return SubscribeUpdate.decode( new Reader(undefined, new Uint8Array([body.length, ...body]), Version.DRAFT_07), Version.DRAFT_07, @@ -327,12 +328,38 @@ test("SubscribeEnd carries its error on draft-07", async () => { test("SubscribeDrop is restored on draft-07", async () => { const drop: SubscribeResponse = { drop: new SubscribeDrop({ start: 1n, end: 3n, error: 7 }) }; const wire = await encode(Version.DRAFT_07, drop); - expect(wire).toEqual(new Uint8Array([2, 3, 1, 3, 7])); + // A drop no budget reverses has a Max Age of 0. + expect(wire).toEqual(new Uint8Array([2, 5, 1, 3, 7, 0, 0])); const got = await responseRoundtrip(Version.DRAFT_07, drop); if (!("drop" in got)) throw new Error("expected drop"); expect([got.drop.start, got.drop.end, got.drop.error]).toEqual([1n, 3n, 7]); }); +test("a stale drop names its max age and terms on draft-07 alone", async () => { + for (const maxAge of [0, 300]) { + const drop = new SubscribeDrop({ start: 4n, end: 4n, error: 0x34, maxAge, terms: 2n }); + const got = await responseRoundtrip(Version.DRAFT_07, { drop }); + if (!("drop" in got)) throw new Error("expected drop"); + expect([got.drop.maxAge, got.drop.terms]).toEqual([maxAge, 2n]); + } + const drop = new SubscribeDrop({ start: 4n, end: 4n, error: 0, maxAge: 300, terms: 2n }); + expect(await encode(Version.DRAFT_06, { drop })).toEqual(new Uint8Array([2, 3, 4, 4, 0])); + const got = await responseRoundtrip(Version.DRAFT_06, { drop }); + if (!("drop" in got)) throw new Error("expected drop"); + expect([got.drop.maxAge, got.drop.terms]).toEqual([undefined, 0n]); +}); + +test("SubscribeSkip is type 3 on draft-07 alone", async () => { + const skip: SubscribeResponse = { skip: new SubscribeSkip({ start: 1n, end: 3n }) }; + expect(await encode(Version.DRAFT_07, skip)).toEqual(new Uint8Array([3, 2, 1, 3])); + const got = await responseRoundtrip(Version.DRAFT_07, skip); + if (!("skip" in got)) throw new Error("expected skip"); + expect([got.skip.start, got.skip.end]).toEqual([1n, 3n]); + await expect(encode(Version.DRAFT_06, skip)).rejects.toThrow(); + const malformed = new Reader(undefined, new Uint8Array([2, 3, 1]), Version.DRAFT_07); + await expect(SubscribeSkip.decode(malformed)).rejects.toThrow("invalid skipped range"); +}); + test("SubscribeDrop is type 0x2 on draft-05 and 0x1 on draft-04", async () => { const drop: SubscribeResponse = { drop: new SubscribeDrop({ start: 1n, end: 3n, error: 0 }) }; @@ -457,7 +484,7 @@ test("wanted groups refuse malformed input before writing bytes", async () => { for (const version of [Version.DRAFT_05, Version.DRAFT_06, Version.DRAFT_07]) { test(`SUBSCRIBE_DROP refuses reversed ranges while retaining empty ranges on ${version}`, async () => { const malformed = new Reader(undefined, new Uint8Array([3, 3, 1, 0]), version); - await expect(SubscribeDrop.decode(malformed)).rejects.toThrow("invalid dropped range"); + await expect(SubscribeDrop.decode(malformed, version)).rejects.toThrow("invalid dropped range"); await expect(encode(version, { drop: new SubscribeDrop({ start: 3n, end: 1n, error: 0 }) })).rejects.toThrow(); for (const [start, end] of [ [2n, 1n], diff --git a/js/net/src/lite/subscribe.ts b/js/net/src/lite/subscribe.ts index 1805ea73863..6000d0628ea 100644 --- a/js/net/src/lite/subscribe.ts +++ b/js/net/src/lite/subscribe.ts @@ -23,10 +23,21 @@ import { hasOrder, hasRanges, hasRefresh, + hasSkip, resolvesStart, Version, } from "./version.ts"; +/** Decode the range `Terms` after the ranges. A wire without the field decodes as 0. */ +async function decodeTerms(r: Reader, version: Version): Promise { + return hasSkip(version) ? await r.u62() : 0n; +} + +/** Encode the range `Terms`, dropping it on a wire without the field. */ +async function encodeTerms(w: Writer, version: Version, terms: bigint) { + if (hasSkip(version)) await w.u62(terms); +} + /** * Encode the `Group Start` field shared by SUBSCRIBE and SUBSCRIBE_UPDATE. * @@ -430,6 +441,8 @@ export class SubscribeUpdate { endFrame?: number; /** See {@link Subscribe.ranges}. */ ranges?: WireRange[]; + /** See {@link Subscribe.terms}: larger than the subscription's previous terms. */ + terms: bigint; constructor(props: { priority: number; @@ -442,6 +455,7 @@ export class SubscribeUpdate { startFrame?: number; endFrame?: number; ranges?: WireRange[]; + terms?: bigint; }) { if (props.wantedGroup !== undefined && props.wantedGroup < 0n) throw new RangeError("wanted group must be non-negative"); @@ -455,6 +469,7 @@ export class SubscribeUpdate { this.startFrame = props.startFrame ?? 0; this.endFrame = props.endFrame; this.ranges = props.ranges; + this.terms = props.terms ?? 0n; } async #encode(w: Writer, version: Version) { @@ -469,6 +484,7 @@ export class SubscribeUpdate { await w.u53(this.maxDelay); await encodeMinDelay(w, version, this.minDelay); await encodeSpan(w, version, this, this.ranges); + await encodeTerms(w, version, this.terms); if (hasRefresh(version)) await w.u62(this.wantedGroup === undefined ? 0n : this.wantedGroup + 1n); break; } @@ -485,6 +501,7 @@ export class SubscribeUpdate { const maxDelay = await r.u53(); const minDelay = await decodeMinDelay(r, version); const span = await decodeSpan(r, version); + const terms = await decodeTerms(r, version); const wanted = hasRefresh(version) ? await r.u62() : 0n; return new SubscribeUpdate({ wantedGroup: wanted === 0n ? undefined : wanted - 1n, @@ -493,6 +510,7 @@ export class SubscribeUpdate { maxDelay, minDelay, ...span, + terms, }); } } @@ -555,6 +573,13 @@ export class Subscribe { */ ranges?: WireRange[]; + /** + * The range terms id: every SUBSCRIBE_UPDATE carries a larger one, and a stale + * {@link SubscribeDrop} echoes the one it was judged under. Lite-07+; earlier wires decode + * as 0 and drop it on encode. + */ + terms: bigint; + constructor(props: { id: bigint; broadcast: Path.Valid; @@ -569,6 +594,7 @@ export class Subscribe { startFrame?: number; endFrame?: number; ranges?: WireRange[]; + terms?: bigint; }) { this.id = props.id; this.broadcast = props.broadcast; @@ -583,6 +609,7 @@ export class Subscribe { this.startFrame = props.startFrame ?? 0; this.endFrame = props.endFrame; this.ranges = props.ranges; + this.terms = props.terms ?? 0n; } async #encode(w: Writer, version: Version) { @@ -603,6 +630,7 @@ export class Subscribe { await encodeSpan(w, version, this, this.ranges); break; } + await encodeTerms(w, version, this.terms); } static async #decode(r: Reader, version: Version): Promise { @@ -621,6 +649,7 @@ export class Subscribe { const maxDelay = await r.u53(); const minDelay = await decodeMinDelay(r, version); const span = await decodeSpan(r, version); + const terms = await decodeTerms(r, version); return new Subscribe({ id, broadcast, @@ -631,6 +660,7 @@ export class Subscribe { maxDelay, minDelay, ...span, + terms, }); } } @@ -839,24 +869,53 @@ export class SubscribeEnd { } } -/// Indicates that one or more groups have been dropped. -/// -/// Draft-03+; publishers send it on Draft-05 and later. +/** + * Names groups that existed but will not be delivered: aborted or reset, evicted or expired with + * no copy left, or stale. + * + * Draft-03+; publishers send it on Draft-05 and later. Before lite-07 it also names sequences + * never produced, which lite-07 names in {@link SubscribeSkip}. + */ export class SubscribeDrop { start: bigint; end: bigint; error: number; + /** + * The budget in milliseconds a stale verdict was judged under, undefined for a drop that does + * not depend on it. Lite-07+, encoded as the value plus one; earlier wires decode as undefined. + */ + maxAge?: number; + /** The range terms id ({@link Subscribe.terms}) a stale verdict was judged under. Lite-07+. */ + terms: bigint; - constructor(props: { start: bigint; end: bigint; error: number }) { + constructor(props: { start: bigint; end: bigint; error: number; maxAge?: number; terms?: bigint }) { this.start = props.start; this.end = props.end; this.error = props.error; + this.maxAge = props.maxAge; + this.terms = props.terms ?? 0n; } - /** Name `[start, end]` given up with `err`, as `version` carries it. */ - static given(start: bigint, end: bigint, err: unknown, version: Version): SubscribeDrop { - // Before lite-07 the code is the application's, so it says nothing of why. - return new SubscribeDrop({ start, end, error: hasDropCode(version) ? toStreamCode(err) : 0 }); + /** + * Name `[start, end]` given up with `err`, as `version` carries it, judged under `maxAge` and + * `terms` when the verdict depends on the budget. + */ + static given( + start: bigint, + end: bigint, + err: unknown, + version: Version, + judged?: { maxAge?: number; terms: bigint }, + ): SubscribeDrop { + const skip = hasSkip(version); + return new SubscribeDrop({ + start, + end, + // Before lite-07 the code is the application's, so it says nothing of why. + error: hasDropCode(version) ? toStreamCode(err) : 0, + maxAge: skip ? judged?.maxAge : undefined, + terms: skip ? (judged?.terms ?? 0n) : 0n, + }); } /** @@ -872,25 +931,74 @@ export class SubscribeDrop { return fromTransport({ source: "stream", streamErrorCode: this.error }); } - async #encode(w: Writer) { + async #encode(w: Writer, version: Version) { if (this.start - this.end > 1n) throw new RangeError("invalid dropped range"); await w.u62(this.start); await w.u62(this.end); await w.u53(this.error); + if (hasSkip(version)) { + await w.u62(this.maxAge === undefined ? 0n : BigInt(this.maxAge) + 1n); + await w.u62(this.terms); + } + } + + static async #decode(r: Reader, version: Version): Promise { + const start = await r.u62(); + const end = await r.u62(); + if (start - end > 1n) throw new ProtocolViolation("invalid dropped range"); + const error = await r.u53(); + let maxAge: number | undefined; + let terms = 0n; + if (hasSkip(version)) { + const age = await r.u62(); + if (age > BigInt(Number.MAX_SAFE_INTEGER) + 1n) throw new RangeError("max age exceeds safe integer"); + maxAge = age === 0n ? undefined : Number(age - 1n); + terms = await r.u62(); + } + return new SubscribeDrop({ start, end, error, maxAge, terms }); + } + + async encode(w: Writer, version: Version): Promise { + return Message.encode(w, (w) => this.#encode(w, version)); + } + + static async decode(r: Reader, version: Version): Promise { + return Message.decode(r, (r) => SubscribeDrop.#decode(r, version)); + } +} + +/** + * Names sequences the publisher never produced, such as skipped sequence numbers or ones past a + * cut: final, with no error, ending no reader. Lite-07+; earlier versions name them in a + * {@link SubscribeDrop}. + */ +export class SubscribeSkip { + start: bigint; + end: bigint; + + constructor(props: { start: bigint; end: bigint }) { + this.start = props.start; + this.end = props.end; + } + + async #encode(w: Writer) { + if (this.start - this.end > 1n) throw new RangeError("invalid skipped range"); + await w.u62(this.start); + await w.u62(this.end); } - static async #decode(r: Reader): Promise { - const drop = new SubscribeDrop({ start: await r.u62(), end: await r.u62(), error: await r.u53() }); - if (drop.start - drop.end > 1n) throw new ProtocolViolation("invalid dropped range"); - return drop; + static async #decode(r: Reader): Promise { + const skip = new SubscribeSkip({ start: await r.u62(), end: await r.u62() }); + if (skip.start - skip.end > 1n) throw new ProtocolViolation("invalid skipped range"); + return skip; } async encode(w: Writer): Promise { return Message.encode(w, this.#encode.bind(this)); } - static async decode(r: Reader): Promise { - return Message.decode(r, SubscribeDrop.#decode); + static async decode(r: Reader): Promise { + return Message.decode(r, SubscribeSkip.#decode); } } @@ -902,12 +1010,14 @@ export class SubscribeDrop { * - Draft-03/04: `0x0` SUBSCRIBE_OK, `0x1` SUBSCRIBE_DROP. * - Draft-05+: `0x0` SUBSCRIBE_START, `0x1` SUBSCRIBE_END, `0x2` SUBSCRIBE_DROP * (SUBSCRIBE_OK was removed; acceptance is implicit). + * - Lite-07+: also `0x3` SUBSCRIBE_SKIP. */ export type SubscribeResponse = | { ok: SubscribeOk } | { start: SubscribeStart } | { end: SubscribeEnd } - | { drop: SubscribeDrop }; + | { drop: SubscribeDrop } + | { skip: SubscribeSkip }; export async function encodeSubscribeResponse(w: Writer, resp: SubscribeResponse, version: Version): Promise { switch (version) { @@ -926,7 +1036,7 @@ export async function encodeSubscribeResponse(w: Writer, resp: SubscribeResponse await resp.ok.encode(w, version); } else if ("drop" in resp) { await w.u53(0x1); - await resp.drop.encode(w); + await resp.drop.encode(w, version); } else { throw new Error("SUBSCRIBE_START/END not supported for this version"); } @@ -941,7 +1051,10 @@ export async function encodeSubscribeResponse(w: Writer, resp: SubscribeResponse await resp.end.encode(w, version); } else if ("drop" in resp) { await w.u53(0x2); - await resp.drop.encode(w); + await resp.drop.encode(w, version); + } else if ("skip" in resp && hasSkip(version)) { + await w.u53(0x3); + await resp.skip.encode(w); } else { throw new Error("subscribe response not supported for this version"); } @@ -961,7 +1074,7 @@ export async function decodeSubscribeResponse(r: Reader, version: Version): Prom case 0x0: return { ok: await SubscribeOk.decode(r, version) }; case 0x1: - return { drop: await SubscribeDrop.decode(r) }; + return { drop: await SubscribeDrop.decode(r, version) }; default: throw new ProtocolViolation(`unknown subscribe response type: ${typ}`); } @@ -974,7 +1087,10 @@ export async function decodeSubscribeResponse(r: Reader, version: Version): Prom case 0x1: return { end: await SubscribeEnd.decode(r, version) }; case 0x2: - return { drop: await SubscribeDrop.decode(r) }; + return { drop: await SubscribeDrop.decode(r, version) }; + case 0x3: + if (!hasSkip(version)) throw new ProtocolViolation(`unknown subscribe response type: ${typ}`); + return { skip: await SubscribeSkip.decode(r) }; default: throw new ProtocolViolation(`unknown subscribe response type: ${typ}`); } diff --git a/js/net/src/lite/subscriber.test.ts b/js/net/src/lite/subscriber.test.ts index 3e183003f8c..f63803d0c99 100644 --- a/js/net/src/lite/subscriber.test.ts +++ b/js/net/src/lite/subscriber.test.ts @@ -2837,7 +2837,7 @@ test("an unknown subscribe response type closes the session", async () => { await answerTrackInfo(streams[0], Version.DRAFT_07); await (await opened(1)).reading; // An unassigned response type is a session protocol violation. - streams[1].inbound.enqueue(new Uint8Array([3, 0])); + streams[1].inbound.enqueue(new Uint8Array([4, 0])); expect(await track.closed).toBeInstanceOf(ProtocolViolation); await drainReactions(); expect(closes.map((info) => info?.closeCode)).toEqual([SessionCode.ProtocolViolation]); diff --git a/js/net/src/lite/subscriber.ts b/js/net/src/lite/subscriber.ts index 02e38e65c11..5b004e3386e 100644 --- a/js/net/src/lite/subscriber.ts +++ b/js/net/src/lite/subscriber.ts @@ -22,7 +22,16 @@ import { import * as netGroup from "../group.ts"; import { Cost, type Hop, MAX_HOPS, type Route, routesEqual, UNKNOWN_HOP } from "../hop.ts"; import * as Instance from "../instance.ts"; -import { hiddenBelow, hooks, maxDelayOf, minDelayOf, scopeCaptures, scopeHead, scopeOverlaps } from "../internal.ts"; +import { + hiddenBelow, + hooks, + maxDelayOf, + minDelayOf, + scopeCaptures, + scopeHead, + scopeOverlaps, + skipHole, +} from "../internal.ts"; import type { Lister } from "../listing.ts"; import * as Path from "../path.ts"; import { receiveFrame } from "../payload.ts"; @@ -147,6 +156,9 @@ interface SubscribeEntry { starts?: (bigint | undefined)[]; // Whether a SUBSCRIBE_UPDATE went out, which the publisher may have read before answering. updated?: boolean; + // The range terms id of the latest SUBSCRIBE or SUBSCRIBE_UPDATE sent: a stale drop judged + // under an earlier one is no verdict on the current terms. Undefined is the SUBSCRIBE's 0. + terms?: bigint; // Set by the subscribe stream's FIN before lite-05, the track's only end there. finished?: boolean; // Settles once the subscribe stream's responses are read to its FIN or reset, which @@ -1814,8 +1826,19 @@ export class Subscriber { } else if ("drop" in resp) { // The publisher will never send these groups, so they are accounted for without a // stream, and a reader of one still arriving ends with the drop's error now. - entry.tail.account(resp.drop.start, resp.drop.end + 1n); - hooks.dropTrackGroups(entry.track, resp.drop.start, resp.drop.end + 1n, resp.drop.cause(this.version)); + const { start, end, maxAge, terms } = resp.drop; + entry.tail.account(start, end + 1n); + // A stale verdict under terms an update has since replaced says nothing of the + // current ones: the groups are unknown, and a reader that wants one fills it. + if (maxAge !== undefined && terms < (entry.terms ?? 0n)) continue; + hooks.dropTrackGroups(entry.track, start, end + 1n, { + error: resp.drop.cause(this.version), + stale: maxAge, + }); + } else if ("skip" in resp) { + // Never produced, so nothing is waited for and no reader ends. + entry.tail.account(resp.skip.start, resp.skip.end + 1n); + hooks.dropTrackGroups(entry.track, resp.skip.start, resp.skip.end + 1n, skipHole()); } } } @@ -1962,7 +1985,13 @@ export class Subscriber { startGroup: span.start, endGroup: inclusiveGroupEnd(span.end), ranges: hasRanges(this.version) ? toWireRanges(ranges, entry.scale) : undefined, + terms: (entry.terms ?? 0n) + 1n, }); + // A stale drop judged under the previous terms may still be in flight, and this update + // may move the range it was judged against, so the runs held as stale become unknown + // again: a reader fills them rather than trust an old verdict. + entry.terms = update.terms; + hooks.supersedeStale(entry.track); // Encoded apart from the write, so a bound the wire can't carry still fails the track. const encoded = await Writer.collect(this.version, (w) => update.encode(w, this.version)); // A failed write means the publisher stopped reading, as it does once it finishes the diff --git a/js/net/src/lite/version.ts b/js/net/src/lite/version.ts index 0ce6dac4db9..5bf69b56ceb 100644 --- a/js/net/src/lite/version.ts +++ b/js/net/src/lite/version.ts @@ -580,6 +580,27 @@ export function hasDropCode(version: Version): boolean { } } +/** + * Whether holes split into SUBSCRIBE_SKIP, for sequences never produced, and SUBSCRIBE_DROP, + * for groups that existed, whose drop names the `Max Age` and range `Terms` a stale verdict was + * judged under; SUBSCRIBE and SUBSCRIBE_UPDATE carry those `Terms`. Added in lite-07; earlier + * versions name every hole in SUBSCRIBE_DROP. + */ +export function hasSkip(version: Version): boolean { + // Explicitly list older versions so future versions keep the lite-07+ behavior. + switch (version) { + case Version.DRAFT_01: + case Version.DRAFT_02: + case Version.DRAFT_03: + case Version.DRAFT_04: + case Version.DRAFT_05: + case Version.DRAFT_06: + return false; + default: + return true; + } +} + /** * Whether SUBSCRIBE_END carries the error the track ended with, so an errored end keeps its * history across a hop. Added in lite-07; earlier versions can only reset. diff --git a/js/net/src/lite/widened_max_age.test.ts b/js/net/src/lite/widened_max_age.test.ts index d1f41fda63a..1c3c4a9df4b 100644 --- a/js/net/src/lite/widened_max_age.test.ts +++ b/js/net/src/lite/widened_max_age.test.ts @@ -10,7 +10,7 @@ import * as Path from "../path.ts"; import { Timescale, Timestamp } from "../time.ts"; import type { Subscriber as TrackSubscriber } from "../track.ts"; import { wireOf } from "../wire.ts"; -import { ALPN_07_WIP, Version } from "./version.ts"; +import { ALPN_06, ALPN_07_WIP, Version } from "./version.ts"; const url = new URL("https://localhost:4443/test"); @@ -62,3 +62,57 @@ test("a widened maxDelay re-offers what the publisher skipped below the start", client.abort(); server.abort(); }); + +for (const [version, alpn] of [ + [Version.DRAFT_06, ALPN_06], + [Version.DRAFT_07, ALPN_07_WIP], +] as const) { + const reoffers = version === Version.DRAFT_07; + test(`a widened maxDelay ${reoffers ? "re-offers" : "keeps dropped"} the groups reported stale on ${alpn}`, async () => { + const pair = createMockTransportPair(alpn); + const origin = new OriginProducer(); + const publication = origin.createBroadcast(Path.from("room")); + publication.announce(); + const broadcast = publication.broadcast; + const producer = broadcast.createTrack("video", { timescale: Timescale.MILLI }); + const write = (sequence: number, millis: number) => { + const group = producer.createGroup(BigInt(sequence)); + group.writeFrame({ payload: new Uint8Array([sequence]), timestamp: Timestamp.fromMillis(millis) }); + group.finish(); + }; + write(0, 0); + + const [client, server] = await Promise.all([ + connect({ url, transport: pair.client }), + accept({ transport: pair.server, url, version, publish: origin.consume() }), + ]); + const remote = wireOf(client).consume(Path.from("room")); + const reader = remote.track("video").subscribe({ delay: { max: Milli(1500) } }); + expect(await received(reader, 1)).toEqual([0n]); + // Groups 1 and 2 reach 2s and 3s, at least 1.5s behind the 6s edge, so they are reported + // stale after SUBSCRIBE_START. + for (const [sequence, millis] of [ + [1, 1000], + [2, 2000], + [3, 3000], + [4, 6000], + ]) { + write(sequence, millis); + } + expect(await received(reader, 2)).toEqual([3n, 4n]); + + // Lite-07 re-offers only the groups reported stale; earlier subscribers settled them for good. + reader.update({ delay: { max: Milli(10_000) } }); + if (reoffers) expect(await received(reader, 2)).toEqual([1n, 2n]); + producer.finish(); + producer.close(); + expect(await reader.recvGroup()).toBeUndefined(); + + reader.close(); + remote.close(); + broadcast.close(); + origin.close(); + client.abort(); + server.abort(); + }); +} diff --git a/js/net/src/resume.model.test.ts b/js/net/src/resume.model.test.ts index 1e0add73d51..57b194d0086 100644 --- a/js/net/src/resume.model.test.ts +++ b/js/net/src/resume.model.test.ts @@ -391,7 +391,7 @@ class Model { entry.writer.abort(giveUp()); entry.writer = undefined; } else if (!entry) { - hooks.dropTrackGroups(copy.track, BigInt(op.group), BigInt(op.group + 1), giveUp()); + hooks.dropTrackGroups(copy.track, BigInt(op.group), BigInt(op.group + 1), { error: giveUp() }); } instance.givenUp.add(op.group); break; diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts index c3458c25619..2436679a3fd 100644 --- a/js/net/src/resume.test.ts +++ b/js/net/src/resume.test.ts @@ -1141,7 +1141,8 @@ for (const resolution of ["arrival", "drop", "other copy", "budget"] as const) { tb.write(5n); tb.write(6n); } - if (resolution === "drop") hooks.dropTrackGroups(ta.producer, 5n, 6n, new StreamError(StreamCode.Old)); + if (resolution === "drop") + hooks.dropTrackGroups(ta.producer, 5n, 6n, { error: new StreamError(StreamCode.Old) }); if (resolution === "budget") await clock.advance(Nano.fromMilli(Milli(1000))); const expected = resolution === "arrival" || resolution === "other copy" ? [5n, 6n, 7n] : [6n, 7n]; expect((await held)?.sequence).toBe(expected[0]); @@ -1239,7 +1240,7 @@ for (const fails of [true, false]) { expect(tb.producer.demand().wanted.peek()).not.toBeUndefined(); const cut = new StreamError(StreamCode(70), { message: "cut" }); - hooks.dropTrackGroups(tb.producer, 0n, 1n, cut); + hooks.dropTrackGroups(tb.producer, 0n, 1n, { error: cut }); await expect(read).rejects.toBe(cut); sub.close(); @@ -1269,9 +1270,9 @@ test("a resumed reader reports a drop once and still serves a late group", async ta.write(0n); expect(recvDisposition(sub)).toBe(0n); // A route failing is no verdict: another route may still deliver the group. - hooks.dropTrackGroups(ta.producer, 1n, 2n, new StreamError(StreamCode.SessionClosed)); + hooks.dropTrackGroups(ta.producer, 1n, 2n, { error: new StreamError(StreamCode.SessionClosed) }); expect(recvDisposition(sub)).toBe("idle"); - hooks.dropTrackGroups(ta.producer, 2n, 4n, new StreamError(StreamCode(70), { message: "cut" })); + hooks.dropTrackGroups(ta.producer, 2n, 4n, { error: new StreamError(StreamCode(70), { message: "cut" }) }); expect(recvDisposition(sub)).toEqual([2n, 4n, 70]); expect(recvDisposition(sub)).toBe("idle"); // Re-created under a run already reported dropped. @@ -1295,7 +1296,7 @@ test("a separately polled drop does not deliver its run", () => { track.write(0n); expect(drain(sub)).toEqual([0n]); const error = new StreamError(StreamCode.NotFound); - hooks.dropTrackGroups(track.producer, 2n, 4n, error); + hooks.dropTrackGroups(track.producer, 2n, 4n, { error: error }); expect(hooks.tryDrop(sub)).toEqual({ kind: "drop", start: 2n, end: 4n, error }); expect(hooks.tryDrop(sub)).toBeUndefined(); track.write(3n); @@ -1348,7 +1349,7 @@ test("a drop waits while a copy the serving one replaced may still deliver it", front.serve(b.consume()); await settle(); // Only the new route gave groups 1 and 2 up; the old one is not yet bounded. - hooks.dropTrackGroups(tb.producer, 1n, 3n, new StreamError(StreamCode(70), { message: "cut" })); + hooks.dropTrackGroups(tb.producer, 1n, 3n, { error: new StreamError(StreamCode(70), { message: "cut" }) }); expect(recvDisposition(sub)).toBe("idle"); ta.write(1n); expect(recvDisposition(sub)).toBe(1n); @@ -1386,7 +1387,7 @@ test("a drop a replaced route held back is no verdict on the route serving next" front.serve(b.consume()); await settle(); // Held back: the copy b replaced may still deliver group 1. - hooks.dropTrackGroups(tb.producer, 1n, 2n, new StreamError(StreamCode(70), { message: "cut" })); + hooks.dropTrackGroups(tb.producer, 1n, 2n, { error: new StreamError(StreamCode(70), { message: "cut" }) }); expect(recvDisposition(sub)).toBe("idle"); front.serve(c.consume()); await settle(); @@ -1414,7 +1415,7 @@ test("a front serving another front reports the inner serving route's drop", asy const sub = outer.consumer.track("seq").subscribe({ delay: { max: Milli(60_000) } }); ta.write(0n); expect(recvDisposition(sub)).toBe(0n); - hooks.dropTrackGroups(ta.producer, 1n, 3n, new StreamError(StreamCode(70), { message: "cut" })); + hooks.dropTrackGroups(ta.producer, 1n, 3n, { error: new StreamError(StreamCode(70), { message: "cut" }) }); expect(recvDisposition(sub)).toEqual([1n, 3n, 70]); expect(recvDisposition(sub)).toBe("idle"); // Re-created under a run already reported dropped. @@ -2260,7 +2261,7 @@ test.each(["lands", "dropped", "expired"])( expect(drain(sub)).toEqual([]); expect(ta.producer.demand().wanted.peek()).toBeDefined(); - if (outcome === "dropped") hooks.dropTrackGroups(ta.producer, 1n, 2n, new NotFound("group")); + if (outcome === "dropped") hooks.dropTrackGroups(ta.producer, 1n, 2n, { error: new NotFound("group") }); if (outcome === "expired") await clock.advance(Nano.fromMilli(SECOND)); if (outcome !== "lands") { // The groups drained above are released once their close settles. diff --git a/js/net/src/resume.ts b/js/net/src/resume.ts index 770361aa7f4..1f81bcd679c 100644 --- a/js/net/src/resume.ts +++ b/js/net/src/resume.ts @@ -737,7 +737,7 @@ class Reader implements ResumeSource { // A route failing is no verdict: another route may still deliver the group, and // one it failed while buffered is handed out to continue through its recovery. if (routeFailed(dropped.error)) continue; - this.#drops = insertDrop(this.#drops, dropped.start, dropped.end, dropped.error); + this.#drops = insertDrop(this.#drops, dropped.start, dropped.end, dropped); } } this.#drops = this.#drops.filter((run) => !delivered.covers(run.start, run.end)); @@ -755,7 +755,7 @@ class Reader implements ResumeSource { if (open) { // Report once without marking it delivered: a late fill may still land. this.#drops = removeRun(this.#drops, open.start, open.end); - return { kind: "drop", ...open, error: run.error }; + return { kind: "drop", ...open, error: run.error, skip: run.skip, stale: run.stale }; } } } diff --git a/js/net/src/track.test.ts b/js/net/src/track.test.ts index 5eb5c49227f..770dfd05c8d 100644 --- a/js/net/src/track.test.ts +++ b/js/net/src/track.test.ts @@ -6,7 +6,7 @@ import { Decoder } from "./flate/decoder.ts"; import { Encoder } from "./flate/encoder.ts"; import type { Consumer as GroupConsumer, Producer as GroupProducer } from "./group.ts"; import { Version as IetfVersion } from "./ietf/version.ts"; -import { hooks } from "./internal.ts"; +import { hooks, skipHole, staleHole } from "./internal.ts"; import { textFrame } from "./mock.ts"; import { Timescale, Timestamp } from "./time.ts"; import { infoDefaults, Subscription, Producer as TrackProducer } from "./track.ts"; @@ -233,7 +233,7 @@ test("ordered reads pass accounted sequences and honor the feed and reader floor producer.appendDatagram(undefined, enc.encode("d")); putGroup(producer, 2n); expect(drainOrdered(reader)).toEqual([0n, 2n]); - hooks.dropTrackGroups(producer, 3n, 4n, new StreamError(StreamCode.Old)); + hooks.dropTrackGroups(producer, 3n, 4n, { error: new StreamError(StreamCode.Old) }); putGroup(producer, 4n); expect(drainOrdered(reader)).toEqual([4n]); producer.createGroup(5n).abort(new StreamError(StreamCode.Cancel)); @@ -314,7 +314,7 @@ test("ordered owed waits wake on a datagram, a drop, a declared start, or track expect(ready).toBe(false); // A datagram shares the sequence namespace, so it accounts for the hole too. if (resolve === "datagram") producer.insertDatagram(2n, undefined, new Uint8Array([2])); - if (resolve === "drop") hooks.dropTrackGroups(producer, 2n, 3n, new StreamError(StreamCode.Old)); + if (resolve === "drop") hooks.dropTrackGroups(producer, 2n, 3n, { error: new StreamError(StreamCode.Old) }); if (resolve === "start") hooks.startTrack(producer, { start: 3n }); if (resolve === "close") { producer.finish(); @@ -904,7 +904,7 @@ test("a bounded read ends once its window is read, unless a cap or a wire publis // Groups 3 and 4 may still arrive. expect(hooks.tryRecvGroup(reader, false)).toEqual({ kind: "idle" }); - hooks.dropTrackGroups(producer, 3n, 4n, DROPPED); + hooks.dropTrackGroups(producer, 3n, 4n, { error: DROPPED }); producer.createGroup(4n).finish(); expect(read(reader)).toEqual([4n]); expect(hooks.tryRecvGroup(reader, false)).toEqual({ kind: "done" }); @@ -1096,7 +1096,7 @@ test("a drop a time end settles over is reported", () => { }); expect(hooks.tryRecvGroup(reader, true)).toMatchObject({ kind: "group", group: { sequence: 0n } }); // Past the cap while group 0 has yet to start. - hooks.dropTrackGroups(producer, 1n, 2n, DROPPED); + hooks.dropTrackGroups(producer, 1n, 2n, { error: DROPPED }); expect(hooks.tryRecvGroup(reader, true)).toEqual({ kind: "idle" }); zero.writeFrame(textFrame("0", Timestamp.fromMillis(0))); producer.createGroup(2n).writeFrame(textFrame("2", Timestamp.fromMillis(2000))); @@ -3719,7 +3719,7 @@ const DROPPED = new StreamError(StreamCode.NotFound, { message: "dropped" }); test("explicit and producer-cut drops survive outside the group cache", async () => { const producer = new TrackProducer("video").accept({}); const reader = producer.subscribe({ delay: { max: Milli(60_000) } }); - hooks.dropTrackGroups(producer, 3n, 4n, DROPPED); + hooks.dropTrackGroups(producer, 3n, 4n, { error: DROPPED }); expect(hooks.tryRecvGroup(reader, true)).toEqual({ kind: "drop", start: 3n, end: 4n, error: DROPPED }); const group = producer.createGroup(4n); @@ -3768,7 +3768,7 @@ test("an upstream drop ends a group still arriving with its error", async () => expect(group?.sequence).toBe(3n); expect(await group?.readString()).toBe("3"); const cut = new StreamError(StreamCode(70), { message: "cut" }); - hooks.dropTrackGroups(producer, 3n, 5n, cut); + hooks.dropTrackGroups(producer, 3n, 5n, { error: cut }); // Settled by the drop itself, not a later reset; the finished group stays whole. await expect(group?.readString()).rejects.toBe(cut); const whole = await reader.recvGroup(); @@ -3783,7 +3783,7 @@ test("an upstream drop ends a group still arriving with its error", async () => test("a drop below the window is reported once the window lowers to it", () => { const producer = new TrackProducer("video").accept({}); const reader = producer.subscribe({ delay: { max: Milli(60_000) } }).withRange({ start: { included: 5n } }); - hooks.dropTrackGroups(producer, 2n, 3n, DROPPED); + hooks.dropTrackGroups(producer, 2n, 3n, { error: DROPPED }); expect(hooks.tryRecvGroup(reader, true)).toEqual({ kind: "idle" }); reader.withRange({ start: { included: 0n } }); @@ -3814,7 +3814,7 @@ test("a re-created cut group is served, not dropped", async () => { test("a reported drop still allows a late group, handed out once", () => { const producer = new TrackProducer("video").accept({}); const reader = producer.subscribe({ delay: { max: Milli(60_000) } }); - hooks.dropTrackGroups(producer, 2n, 4n, DROPPED); + hooks.dropTrackGroups(producer, 2n, 4n, { error: DROPPED }); expect(hooks.tryRecvGroup(reader, true)).toEqual({ kind: "drop", start: 2n, end: 4n, error: DROPPED }); expect(hooks.tryRecvGroup(reader, true)).toEqual({ kind: "idle" }); producer.createGroup(3n).finish(); @@ -3827,15 +3827,66 @@ test("a reported drop still allows a late group, handed out once", () => { producer.close(); }); +test("a received stale drop settles only the readers it covers", () => { + const producer = new TrackProducer("video").accept({}); + const reader = (max: number, start: bigint, end?: bigint) => + producer.subscribe({ + delay: { max: Milli(max) }, + ranges: [{ start: { absolute: start }, end: end === undefined ? undefined : { absolute: end } }], + }); + const covering = reader(500, 0n); + const narrower = reader(100, 0n); + const wider = reader(5000, 0n); + const capped = reader(500, 0n, 4n); + const above = reader(500, 3n); + const stale = staleHole(500); + hooks.dropTrackGroups(producer, 1n, 3n, stale); + const skip = skipHole(); + hooks.dropTrackGroups(producer, 3n, 4n, skip); + const dropped = (subscriber: typeof covering) => { + const runs: [bigint, bigint, boolean][] = []; + for (let run = hooks.tryDrop(subscriber); run; run = hooks.tryDrop(subscriber)) + runs.push([run.start, run.end, run.skip === true]); + return runs; + }; + const settled: [bigint, bigint, boolean][] = [ + [1n, 3n, false], + [3n, 4n, true], + ]; + expect(dropped(covering)).toEqual(settled); + // A narrower budget is settled too. + expect(dropped(narrower)).toEqual(settled); + // A wider budget waits, a capped range may judge otherwise, and a range above the run lacks it. + for (const subscriber of [wider, capped, above]) expect(dropped(subscriber)).toEqual([[3n, 4n, true]]); + for (const subscriber of [covering, narrower, wider, capped, above]) subscriber.close(); + producer.finish(); + producer.close(); +}); + +test("a superseded stale run is unknown again, and its group is handed out", () => { + const producer = new TrackProducer("video").accept({}); + const reader = producer.subscribe({ delay: { max: Milli(500) }, ranges: [{ start: { absolute: 0n } }] }); + hooks.dropTrackGroups(producer, 1n, 2n, staleHole(500)); + hooks.supersedeStale(producer); + expect(hooks.tryDrop(reader)).toBeUndefined(); + producer.createGroup(1n).finish(); + const recv = hooks.tryRecvGroup(reader, true); + expect(recv.kind === "group" ? recv.group.sequence : recv).toBe(1n); + if (recv.kind === "group") recv.group.close(); + reader.close(); + producer.finish(); + producer.close(); +}); + test("a wide drop is held and reported as one run", () => { const producer = new TrackProducer("video").accept({}); const reader = producer.subscribe({ delay: { max: Milli(60_000) } }); - hooks.dropTrackGroups(producer, 2n, 2n ** 40n, DROPPED); + hooks.dropTrackGroups(producer, 2n, 2n ** 40n, { error: DROPPED }); expect(hooks.tryRecvGroup(reader, true)).toEqual({ kind: "drop", start: 2n, end: 2n ** 40n, error: DROPPED }); // A wider drop reports only what this cursor has not reported yet, with its own error. const wider = new StreamError(StreamCode.Old, { message: "old" }); - hooks.dropTrackGroups(producer, 1n, 2n ** 40n + 2n, wider); + hooks.dropTrackGroups(producer, 1n, 2n ** 40n + 2n, { error: wider }); expect(hooks.tryRecvGroup(reader, true)).toEqual({ kind: "drop", start: 1n, end: 2n, error: wider }); expect(hooks.tryRecvGroup(reader, true)).toEqual({ kind: "drop", @@ -3855,12 +3906,9 @@ test("adjacent drops with the same code merge into one run", () => { const reader = producer.subscribe({ delay: { max: Milli(60_000) } }); // Each DROP decodes to its own error, so runs must merge by code, not identity. for (let sequence = 2n; sequence < 5n; sequence++) { - hooks.dropTrackGroups( - producer, - sequence, - sequence + 1n, - new StreamError(StreamCode.NotFound, { message: "dropped" }), - ); + hooks.dropTrackGroups(producer, sequence, sequence + 1n, { + error: new StreamError(StreamCode.NotFound, { message: "dropped" }), + }); } const recv = hooks.tryRecvGroup(reader, true); expect(recv.kind === "drop" ? [recv.start, recv.end, recv.error] : recv).toEqual([2n, 5n, DROPPED]); @@ -4457,7 +4505,7 @@ for (const abort of [false, true]) { const next = reader.recvGroup(); expect(await settled(next)).toBe(false); - hooks.dropTrackGroups(producer, 0n, 2n, DROPPED); + hooks.dropTrackGroups(producer, 0n, 2n, { error: DROPPED }); if (abort) producer.abort(new Error("dropped")); expect(await next).toBeUndefined(); if (abort) expect(await reader.closed).toBeNull(); diff --git a/js/net/src/track.ts b/js/net/src/track.ts index 0a9c1013844..ac2f4fe0f2f 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -25,6 +25,7 @@ import { dropAt, finalSequence, groupBounds, + type Hole, hooks, insertDrop, insertRun, @@ -39,11 +40,12 @@ import { removeRun, runGaps, type StagedTrack, + staleHole, type TrackRequestOptions, } from "./internal.ts"; import { CACHE_EXPIRY, grace, Ledger } from "./ledger.ts"; import { Timescale, type Timestamp } from "./time.ts"; -import { checkSequence, max, min } from "./util/sequence.ts"; +import { checkSequence, max, min, UNBOUNDED } from "./util/sequence.ts"; import { type Holdings, hasTime, @@ -621,8 +623,8 @@ class TrackState { /** The producer fanning into this sink, so a subscriber can mint a sibling of itself. */ producer?: Producer; groups = new Signal([]); - // Sequences the upstream publisher gave up, with the error each was given up with: an - // incoming SUBSCRIBE_DROP or a producer cut. Kept apart from the cache so reclaim cannot + // Sequences the upstream publisher gave up, with why: an incoming SUBSCRIBE_SKIP or + // SUBSCRIBE_DROP, or a producer cut. Kept apart from the cache so reclaim cannot // erase a disposition a downstream publisher still has to forward, and as runs so a wide // DROP costs one entry. Replaced rather than mutated, so a cursor can tell it changed by // identity. @@ -754,6 +756,20 @@ let startsOf: (producer: Producer) => readonly Start[]; // The exclusive end of the run of sequences from `sequence` the producer's feed can no longer // deliver, undefined while it may; assigned in its static block. let goneAt: (producer: Producer, sequence: bigint) => bigint | undefined; +// Whether the producer is fed by a wire subscription; assigned in its static block. +let received: (producer: Producer) => boolean; + +/** + * Whether `hole`, over a run from `run.start`, settles a reader judging under `budget` that + * covers everything from `coversFrom` up. Any hole but a received stale one does. A stale + * verdict holds for a reader no wider than the budget it was judged under, whose range covers + * the run and every group above it: its own successor and edge are then no older than those the + * verdict was judged against. Mirrors the Rust `Drift::settles`. + */ +function settles(run: DropRun, budget: number, coversFrom: bigint | undefined): boolean { + if (run.stale === undefined) return true; + return budget <= run.stale && coversFrom !== undefined && coversFrom <= run.start; +} // Actual object metadata outlives cached payloads, unlike empty group heads in model largest. let objectLargest: (producer: Producer) => Position | undefined; @@ -937,7 +953,15 @@ export class Producer { hooks.releaseTrack = (producer) => producer.#close(undefined); hooks.trackReleased = (producer) => producer.#state.closed; hooks.trackFinished = (producer) => producer.#state.end.peek() !== undefined; - hooks.dropTrackGroups = (producer, start, end, error) => producer.#drop(start, end, error, true); + hooks.dropTrackGroups = (producer, start, end, hole) => producer.#drop(start, end, hole, true); + hooks.supersedeStale = (producer) => { + for (const state of [producer.#state, ...producer.#sinks]) { + const drops = state.drops.peek(); + if (drops.some((run) => run.stale !== undefined)) + state.drops.set(drops.filter((run) => run.stale === undefined)); + } + }; + received = (producer) => producer.#subscribed; hooks.dropAnchors = (producer) => { for (const state of [producer.#state, ...producer.#sinks]) state.dropsAnchors.set(true); }; @@ -1021,18 +1045,18 @@ export class Producer { return undefined; } - // Record `[start, end)` given up with `error`. An upstream drop (`open`) also ends the - // groups still arriving in it with that error, so their readers settle now rather than - // once the stream's reset lands; one that finished here stays whole. - #drop(start: bigint, end: bigint, error: Error, open = false): void { + // Record `[start, end)` given up as `hole` says. An upstream drop (`open`) also ends the + // groups still arriving in it with its error, so their readers settle now rather than once + // the stream's reset lands; one that finished here stays whole. A skip names nothing produced. + #drop(start: bigint, end: bigint, hole: Hole, open = false): void { if (start >= end) return; for (const state of [this.#state, ...this.#sinks]) { - state.drops.update((drops) => insertDrop(drops, start, end, error)); + state.drops.update((drops) => insertDrop(drops, start, end, hole)); } - if (open) { + if (open && !hole.skip) { for (const [sequence, entry] of this.#cached) { if (sequence >= start && sequence < end && entry.group.closed.peek() === undefined) { - entry.group.abort(error); + entry.group.abort(hole.error); } } } @@ -1483,7 +1507,7 @@ export class Producer { // A re-created group already replaced this one. const current = this.#cached.get(group.sequence); if (current && current.group !== group) return; - this.#drop(group.sequence, group.sequence + 1n, closed); + this.#drop(group.sequence, group.sequence + 1n, { error: closed }); }); const sequence = group.sequence; if (this.#unchecked()) { @@ -2017,8 +2041,22 @@ export class Subscriber { // the track's drops last folded in, and every sequence already reported. Drops are // persistent track state, so they are tracked here rather than removed for everyone. #pending: DropRun[] = []; - #dropsSeen?: { drops: readonly DropRun[]; start: bigint; end?: bigint; revision: number }; + #dropsSeen?: { + drops: readonly DropRun[]; + start: bigint; + end?: bigint; + revision: number; + budget: number; + coversFrom?: bigint; + }; #reported: Run[] = []; + // The reported runs judged stale, with the budget each was judged under, for a wire whose + // widening re-offers them; see `hooks.reoffersReported`. + #reportedStale: DropRun[] = []; + // Whether a widening re-offers what this cursor reported stale, and the runs it took back + // since the owner last drained them. + #reoffersReported = false; + #unreported: Run[] = []; // The group the frame-level helpers are currently draining, acquired through the // sequence cursor so frame reads and {@link Ordered.nextGroup} share one floor. #frameGroup?: GroupConsumer; @@ -2297,6 +2335,17 @@ export class Subscriber { // skipped again, so a newer one handed out first never folds it into #delivered. Costs the // skipped groups, never the cache. #reoffer(budget: number): void { + // A reported stale run reversed is judged again, and owed again by the wire. + if (this.#reoffersReported && this.#reportedStale.some((run) => (run.stale ?? budget) < budget)) { + for (const run of this.#reportedStale) { + if ((run.stale ?? budget) >= budget) continue; + this.#reported = removeRun(this.#reported, run.start, run.end); + insertRun(this.#unreported, run.start, run.end); + } + this.#reportedStale = this.#reportedStale.filter((run) => (run.stale ?? budget) >= budget); + this.#pending = this.#pending.filter((run) => (run.stale ?? budget) >= budget); + this.#dropsSeen = undefined; + } if (this.#skipped.size === 0 || budget <= this.#narrowest) return; const sequences: bigint[] = []; let narrowest = Number.POSITIVE_INFINITY; @@ -2311,7 +2360,7 @@ export class Subscriber { // Judged again, so a stale drop not reported yet is withdrawn. if (dropAt(this.#pending, sequence) instanceof Error) { this.#pending = this.#pending.flatMap((run) => - run.start <= sequence && sequence < run.end && toStreamCode(run.error) === StreamCode.Old + run.start <= sequence && sequence < run.end && run.stale !== undefined ? removeRun([run], sequence, sequence + 1n) : [run], ); @@ -2542,6 +2591,21 @@ export class Subscriber { hooks.unreport = (subscriber, start, end) => { subscriber.#reported = removeRun(subscriber.#reported, start, end); }; + hooks.budget = (subscriber) => subscriber.#drift().budget; + hooks.reoffersReported = (subscriber) => { + subscriber.#reoffersReported = true; + return subscriber; + }; + hooks.takeUnreported = (subscriber) => { + const unreported = subscriber.#unreported; + subscriber.#unreported = []; + return unreported; + }; + hooks.isReceived = (subscriber) => { + if (subscriber.#resume) return true; + const producer = subscriber.#state.producer; + return producer !== undefined && received(producer); + }; hooks.tryRecvDatagram = (subscriber) => subscriber.#tryDatagram(); hooks.deliveredRuns = (subscriber) => subscriber.#delivered.runs; hooks.handedOut = (subscriber, sequence) => subscriber.#delivered.has(sequence); @@ -2942,7 +3006,6 @@ export class Subscriber { if (refused) return { kind: "error", error: refused }; const groups = this.#state.groups.peek(); const { start, end } = this.#cursor.peek(); - const old = () => new StreamError(StreamCode.Old, { message: "group is past the budget" }); if (dispositions) { const dropped = this.#tryDrop(); @@ -2968,7 +3031,12 @@ export class Subscriber { groups.splice(index, 1); const failed = group.closed.peek(); if (dispositions && failed instanceof Error) { - const dropped = this.#report(group.sequence, group.sequence + 1n, failed); + // A group expired under this subscription's budget is a stale verdict. + const hole = + toStreamCode(failed) === StreamCode.Old + ? { ...staleHole(drift.budget), error: failed } + : { error: failed }; + const dropped = this.#report(group.sequence, group.sequence + 1n, hole); if (dropped) return dropped; continue; } @@ -2982,11 +3050,16 @@ export class Subscriber { if (drift.bounded) this.#skipped.delete(group.sequence); else this.#skip(group.sequence, drift.budget); if (dispositions) { - const dropped = this.#report(group.sequence, group.sequence + 1n, old()); + const dropped = this.#report(group.sequence, group.sequence + 1n, staleHole(drift.budget)); if (dropped) return dropped; } else if (this.#dropsSeen) { // A cursor #tryDrop reads reports it there. - this.#pending = insertDrop(this.#pending, group.sequence, group.sequence + 1n, old()); + this.#pending = insertDrop( + this.#pending, + group.sequence, + group.sequence + 1n, + staleHole(drift.budget), + ); } continue; } @@ -3006,13 +3079,15 @@ export class Subscriber { } // The first part of `[from, to)` this cursor has not reported yet, now reported, with why. - #report(from: bigint, to: bigint, error: Error): (Recv & { kind: "drop" }) | undefined { + #report(from: bigint, to: bigint, hole: Hole): (Recv & { kind: "drop" }) | undefined { const { start, end } = this.#cursor.peek(); for (const run of this.#window.runs(max(from, start), min(to, end ?? to))) { const [gap] = runGaps(this.#reported, run.start, run.end); if (!gap) continue; insertRun(this.#reported, gap.start, gap.end); - return { kind: "drop", ...gap, error }; + if (hole.stale !== undefined) + this.#reportedStale = insertDrop(this.#reportedStale, gap.start, gap.end, hole); + return { kind: "drop", ...gap, error: hole.error, skip: hole.skip, stale: hole.stale }; } return undefined; } @@ -3030,20 +3105,42 @@ export class Subscriber { // Out-of-window runs were skipped, so a moved window rescans too. const drops = this.#state.drops.peek(); const revision = this.#revision; + const { budget } = this.#drift(); + const coversFrom = this.#coversFrom(); const seen = this.#dropsSeen; - if (!seen || seen.drops !== drops || seen.start !== start || seen.end !== end || seen.revision !== revision) { - this.#dropsSeen = { drops, start, end, revision }; - for (const run of drops) this.#pending = insertDrop(this.#pending, run.start, run.end, run.error); + if ( + !seen || + seen.drops !== drops || + seen.start !== start || + seen.end !== end || + seen.revision !== revision || + seen.budget !== budget || + seen.coversFrom !== coversFrom + ) { + this.#dropsSeen = { drops, start, end, revision, budget, coversFrom }; + for (const run of drops) { + // A received stale run that does not settle this reader is unknown to it. + if (!settles(run, budget, coversFrom)) continue; + this.#pending = insertDrop(this.#pending, run.start, run.end, run); + } } for (let run = this.#pending.shift(); run; run = this.#pending.shift()) { - const dropped = this.#report(run.start, run.end, run.error); + const dropped = this.#report(run.start, run.end, run); if (!dropped) continue; - if (dropped.end < run.end) this.#pending.unshift({ start: dropped.end, end: run.end, error: run.error }); + if (dropped.end < run.end) this.#pending.unshift({ ...run, start: dropped.end }); return dropped; } return undefined; } + // The lowest group from which this reader reads every group up past the live edge, undefined + // while a range end or a cap bounds it. + #coversFrom(): bigint | undefined { + if (this.#cursor.peek().end !== undefined || this.#window.pending) return undefined; + const last = this.#window.runs().at(-1); + return last?.end === UNBOUNDED ? last.start : undefined; + } + // A group from the routes' copies; see #resume. #tryResumed(ordered: boolean, dispositions = false): Recv { const resume = this.#resume as ResumeSource; diff --git a/quest/m1/drop-set.md b/quest/m1/drop-set.md index 370b6af8a33..98f36f54136 100644 --- a/quest/m1/drop-set.md +++ b/quest/m1/drop-set.md @@ -12,40 +12,47 @@ waits, and a widening re-offers what its new budget covers. On lite-07 this restores "growing means I want it": a group reported dropped under a narrower budget is served after a widening, as a cold relay would. -## Waiting - -- Maintainer: settle the JS run memory policy. The 2026-11-02 implementation - audit found that `js/net/src/group.ts` applies its 32 MiB limit to each - group's frame tail, not to track metadata or cursor runs. A mock-time - probe retained 64, 256, and 1,024 disjoint OLD runs with zero cached groups - and zero elapsed time. Unknown gaps prevent coalescing or folding these - into a cursor's final floor; idle expiry alone does not bound a flood - inside its window. Choose shared byte accounting for JS cache metadata - and runs, with eviction and the planned fold-or-`TooFarBehind` behavior - (recommended; settle its budget and ownership), or a different ownership - or backpressure bound that also covers drop-only tracks. The previous - assertion that every run sits between cached sequences is false when - drops name groups this relay never cached, a required case of this quest. -- Maintainer: settle how a received stale drop identifies its judged range - after SUBSCRIBE_UPDATE changes ranges without changing Max Age. A drop - sent under the old broad range can arrive after the relay requests a - narrower aggregate range; that reader now covers the aggregate but not - the range used for the verdict. Mock-time cursors over groups at 0, 1, - and 2 seconds confirm the difference: at a zero-millisecond budget - (wire Max Age 1) the open range serves group 2, whereas a range capped - before group 1 serves group 0. Max Age alone cannot distinguish an old - broad-range verdict from a new one. - Choose an explicit range-terms identifier echoed in stale drops and - treat superseded identifiers as unknown without keeping unbounded term - history (recommended; settle the lite-07 fields), or conservatively - qualify drops against all possibly judged ranges and refill whenever - that cannot prove applicability. The latter must specify how to bound - that state and when a reader is guaranteed a prompt verdict. This is - separate from the settled budget tag and from the deferred lite-ranges - design work. No wire or production change has been implemented. - ## Plan +Decided 2026-11-03 (maintainer, /quest-unblock): `js/net` bounds run memory +by porting Rust's per-broadcast `cache::Pool` to JS. One shared byte account +per broadcast bills frames and drop-set runs (a fixed cost per run) under +Rust's default budget, with eviction and the fold-into-floor or +`TooFarBehind` behavior below, so Rust and JS mirror each other with no new +setting. Reason: the 2026-11-02 audit found the existing 32 MiB limit in +`js/net/src/group.ts` bounds only one group's frame tail, and a mock-time +probe retained 64, 256, and 1,024 disjoint OLD runs with zero cached groups; +a shared pool bounds drop-only tracks the same way in both languages. Not +chosen: a per-track run cap; upstream backpressure. + +Decided 2026-11-03 (maintainer, /quest-unblock): range terms carry an +explicit update id. Each lite-07 SUBSCRIBE and SUBSCRIBE_UPDATE carries an +increasing range-terms id, and a stale drop echoes the id it was judged +under. A relay treats a drop whose id has been superseded as unknown and +refills it by range, keeping no term history. Reason: a drop sent under an +old broad range can arrive after SUBSCRIBE_UPDATE narrows the aggregate at +the same budget (mock-time cursors over groups at 0, 1, and 2 seconds serve +group 2 under the open range and group 0 under a range capped before group +1), and `Max Age` alone cannot tell the two verdicts apart. Not chosen: a +conservative refill with bounded remembered ranges; treating every range +update as invalidating in-flight drops. + +Decided 2026-11-03 (maintainer, /quest-unblock): lite-07 splits its hole +message by whether the group ever existed. `SUBSCRIBE_SKIP {Group Start, +Group End}` names sequences the publisher never produced, such as skipped +sequence numbers or ones past a cut: it has no error code, ends no reader, +and is final. `SUBSCRIBE_DROP {Group Start, Group End, Error Code, Max Age, +Terms}` names groups that existed but will not be delivered: an aborted or +reset group, eviction or expiry with no copy left, and stale (OLD). It ends +any held reader with its code; a stale drop carries the `Max Age` and the +range-terms id it was judged under, so a widening can re-offer it. lite-06 +and older keep today's SUBSCRIBE_DROP on the wire. The split runs through +moq-net's Rust and JS model and API, whose drop/skip naming follows it, and +through the draft. Reason: a hole that never existed has no reader to end +and no budget to qualify, so one message conflated a final, code-free +verdict with a reversible, reader-ending one. Not chosen: stale in SKIP; a +single message with a reason enum; renaming only. + Decided 2026-10-27 (maintainer): this runs after the range count cost work, which finished in [moq-dev/moq#7542](https://github.com/moq-dev/moq/pull/7542), since both edit `given_up` in `rs/moq-net/src/model/track/miss.rs`, and before diff --git a/rs/moq-net/src/fuzz.rs b/rs/moq-net/src/fuzz.rs index 3255184b462..9fbd0b1012d 100644 --- a/rs/moq-net/src/fuzz.rs +++ b/rs/moq-net/src/fuzz.rs @@ -653,6 +653,7 @@ impl LiteSample { max_delay: std::time::Duration::from_secs(10), min_delay: None, ranges: vec![lite::Range::OLDEST], + terms: 0, } .encode(w, version) .unwrap(), @@ -862,6 +863,7 @@ impl Default for Messages { start: Some(lite::Anchor::Absolute(crate::track::Position::group(1_000))), end: None, }], + terms: 0, }, lite_update: lite::SubscribeUpdate { wanted_group: None, @@ -873,6 +875,7 @@ impl Default for Messages { start: Some(lite::Anchor::Absolute(crate::track::Position::group(1_000))), end: Some(lite::Anchor::Absolute(crate::track::Position::group(2_001))), }], + terms: 0, }, lite_start: lite::SubscribeResponse::Start(lite::SubscribeStart { largest: None, @@ -1039,6 +1042,7 @@ pub fn encode_subscribe(broadcast: &Path, track: &str, ietf: bool, out: &mut Vec min_delay: None, ranges: vec![lite::Range::OLDEST], order: track::Order::Descending, + terms: 0, } .encode(w, BENCH_LITE) .unwrap(), diff --git a/rs/moq-net/src/ietf/subscriber.rs b/rs/moq-net/src/ietf/subscriber.rs index 8188b0f0bc1..7f4eef77adf 100644 --- a/rs/moq-net/src/ietf/subscriber.rs +++ b/rs/moq-net/src/ietf/subscriber.rs @@ -15318,7 +15318,7 @@ mod stitch_tests { run.track.broadcast().pool.poll(now + crate::ledger::GRACE, &waiter); assert_eq!( reader.poll_drop(&waiter), - Poll::Ready((1..3, crate::StreamError::NotFound)), + Poll::Ready((1..3, crate::track::Hole::drop(crate::StreamError::NotFound))), "{version}" ); assert!(reader.poll_drop(&waiter).is_pending(), "each drop is reported once"); diff --git a/rs/moq-net/src/lite/publisher.rs b/rs/moq-net/src/lite/publisher.rs index 865488d068f..e3a78e15bf7 100644 --- a/rs/moq-net/src/lite/publisher.rs +++ b/rs/moq-net/src/lite/publisher.rs @@ -3,6 +3,7 @@ use crate::transport::poll::Session as _; #[cfg(test)] use std::sync::atomic::AtomicU64; +use crate::model::track::Hole; use crate::{SessionError, announce, frame, group, origin, track}; use rangemap::{RangeMap, RangeSet}; use std::{ @@ -2262,6 +2263,7 @@ impl Request for SubscribeServe { }; let floor = cursor_floor(shared.version, bounds.groups().start(), latest); let mut run = TrackRun::new(sub, track, bounds, floor, steer_tx); + run.terms = msg.terms; if let Some(update) = update { run.update(update); } @@ -3963,7 +3965,8 @@ struct GroupOutcomes { #[cfg(test)] opened: std::sync::atomic::AtomicU64, unencodable: Mutex>, - dropped: Mutex>, + /// Groups whose stream failed, with why, for the control stream to name. + dropped: Mutex>, } impl Subscription { @@ -4113,7 +4116,9 @@ struct TrackRun { // The first servable group, held until the source resolves where its feed starts. first: Option, // Groups given up before the start resolved, which it must not name, with why. - skipped: RangeMap, + skipped: RangeMap, + // The range terms id of the SUBSCRIBE or latest SUBSCRIBE_UPDATE, which a drop echoes. + terms: u64, end_sent: bool, // Lite-07 with every range bounded: the subscription ends once nothing in its ranges is // left to deliver, rather than parking for an update to widen it. @@ -4152,8 +4157,12 @@ impl TrackRun { // flow through the SUBSCRIBE_UPDATE arm below. let groups = bounds.groups(); let cap = groups.end().map_or(Bound::Unbounded, Bound::Excluded); - // The FIN of a covered subscription is the wire's to send, below. - let track = track.with_range((Bound::Included(floor), cap)).parked(); + // The FIN of a covered subscription is the wire's to send, below. A lite-07 widening + // re-offers what was reported stale. + let mut track = track.with_range((Bound::Included(floor), cap)).parked(); + if ctx.version.has_skip() { + track = track.reoffers_reported(); + } let emit_range = ctx.version.has_track_stream(); let datagrams = ctx.version.has_datagrams() && ctx.session.max_datagram_size() > 0; @@ -4175,6 +4184,7 @@ impl TrackRun { start: None, first: None, skipped: RangeMap::new(), + terms: 0, end_sent: false, served, datagrams, @@ -4201,6 +4211,7 @@ impl TrackRun { // The update moves the cursor and the bounds the track side judges against, which // notifies nothing it parked on. Idle::disarm(&mut self.idle); + self.terms = upd.terms; // Every update restates the wanted group. Past our largest it is a join, which // repeats the pending ask; at or below it the group already exists. if upd @@ -4359,8 +4370,13 @@ impl TrackRun { let dropped = std::mem::take(&mut *self.ctx.outcomes.dropped.lock().unwrap()); if !dropped.is_empty() { - for dropped in dropped { - writer.buffer(&lite::SubscribeResponse::Drop(dropped))?; + for (sequence, err) in dropped { + // A stream expired under this subscription's budget is a stale verdict. + let hole = match err { + StreamError::Old => Hole::stale(self.track.budget()), + err => Hole::drop(err), + }; + writer.buffer(&self.hole(sequence..=sequence, &hole))?; } return Poll::Ready(Ok(ControlFlow::Continue(()))); } @@ -4449,9 +4465,12 @@ impl TrackRun { ) { match res? { Recv::Group(group) => { + self.owe_unreported(); // Never a second stream for a sequence this subscription already accounted // for (served, sent as a datagram, or dropped): a group re-created after its - // DROP stays dropped, since the subscriber may settle before it arrives. + // DROP stays dropped, since the subscriber may settle before it arrives. A + // stale drop a lite-07 widening reversed is owed again (see + // `Self::owe_unreported`). if self .served .as_ref() @@ -4468,13 +4487,13 @@ impl TrackRun { if self.emit_range && self.start.is_none() { // At u64::MAX no later group can resolve a start past this one. if let Some(end) = sequence.checked_add(1) { - self.skipped.insert(sequence..end, StreamError::NotFound); + self.skipped.insert(sequence..end, Hole::drop(StreamError::NotFound)); } } else if self.start.is_some() && let Some(end) = sequence.checked_add(1) { // Lite-05+ names it now; a no-op on other versions. - self.drop_range(sequence..end, &StreamError::NotFound, writer)?; + self.drop_range(sequence..end, &Hole::drop(StreamError::NotFound), writer)?; } return Poll::Ready(Ok(ControlFlow::Continue(()))); }; @@ -4541,12 +4560,15 @@ impl TrackRun { if !self.emit_range { return Ok(false); } - let Poll::Ready((range, err)) = self.track.poll_drop(waiter) else { + let polled = self.track.poll_drop(waiter); + // A widening judged inside that poll may owe a stale drop again, before it is redone. + self.owe_unreported(); + let Poll::Ready((range, hole)) = polled else { return Ok(false); }; match self.start { - None => self.skipped.insert(range, err), - Some(_) => self.drop_range(range, &err, writer)?, + None => self.skipped.insert(range, hole), + Some(_) => self.drop_range(range, &hole, writer)?, } Ok(true) } @@ -4593,6 +4615,18 @@ impl TrackRun { } } + /// Owe again what this subscription reported stale and a widening reversed: lite-07 + /// re-offers a group it still holds and drops again, under the new terms, one it cannot + /// serve. Earlier versions' drops are final, so their cursor takes nothing back. + fn owe_unreported(&mut self) { + let unreported = self.track.take_unreported(); + if let Some(served) = &mut self.served { + for run in unreported { + served.remove(run); + } + } + } + /// Record `sequence` as served, for [`Self::drop_unserved`]. fn mark_served(&mut self, sequence: u64) { let Some(served) = &mut self.served else { @@ -4603,12 +4637,12 @@ impl TrackRun { } } - /// Drop, with `err`, whatever part of `range` this subscription has not been served or - /// dropped yet. + /// Give up, as `hole` says, whatever part of `range` this subscription has not been + /// served or given up yet. fn drop_range( &mut self, range: Range, - err: &StreamError, + hole: &Hole, writer: &mut Writer, ) -> Result<(), Error> { let Some(served) = &mut self.served else { @@ -4618,14 +4652,33 @@ impl TrackRun { return Ok(()); } let gaps: Vec<_> = served.gaps(&range).collect(); + served.extend(gaps.iter().cloned()); for gap in gaps { - served.insert(gap.clone()); - let drop = lite::SubscribeDrop::new(gap.start..=gap.end - 1, err, self.ctx.version); - writer.buffer(&lite::SubscribeResponse::Drop(drop))?; + let msg = self.hole(gap.start..=gap.end - 1, hole); + writer.buffer(&msg)?; } Ok(()) } + /// The message naming `groups` given up as `hole`: lite-07 skips what was never + /// produced, and earlier versions drop it as not found. + fn hole(&self, groups: std::ops::RangeInclusive, hole: &Hole) -> lite::SubscribeResponse { + let version = self.ctx.version; + match hole { + Hole::Skip if version.has_skip() => lite::SubscribeResponse::Skip(lite::SubscribeSkip { + start: *groups.start(), + end: *groups.end(), + }), + hole => lite::SubscribeResponse::Drop(lite::SubscribeDrop::new( + groups, + &hole.error(), + hole.stale_budget(), + self.terms, + version, + )), + } + } + /// Name every sequence from the start to the end that this subscription never got. The /// track has ended, so none of them will be served now: groups it never produced (a /// skipped sequence), and groups the cursor passed over (stale, or missing their head). @@ -4646,7 +4699,13 @@ impl TrackRun { // A cap below the end bounds what the subscriber is owed, as on its side. let fin = final_sequence(fin); let end = self.track.subscription().end().map_or(fin, |end| end.min(fin)); - self.drop_range(start..end, &StreamError::NotFound, writer) + // The track's own publisher knows a group it never reported is one it never produced; + // a relay cannot tell what its source held. + let hole = match self.track.is_received() { + false => Hole::Skip, + true => Hole::drop(StreamError::NotFound), + }; + self.drop_range(start..end, &hole, writer) } } @@ -4699,8 +4758,8 @@ impl TrackRun { }))?; // The subscriber now owes itself everything from the start, so a range given up // above it is named as it would have been after the start. - for (range, err) in skipped.iter() { - self.drop_range(range.start.max(start)..range.end, err, writer)?; + for (range, hole) in skipped.iter() { + self.drop_range(range.start.max(start)..range.end, hole, writer)?; } // SUBSCRIBE_START names where delivery starts. Groups already skipped between // the requested floor and this group are not served. A later group at or above an @@ -5305,9 +5364,12 @@ impl GroupServe { /// that never got a stream has none, so the control stream names it with that error. fn settle_failed(&self, err: &Error) { if self.ctx.version.has_track_stream() { - let drop = - lite::SubscribeDrop::new(self.sequence..=self.sequence, &StreamError::from(err), self.ctx.version); - self.ctx.outcomes.dropped.lock().unwrap().push(drop); + self.ctx + .outcomes + .dropped + .lock() + .unwrap() + .push((self.sequence, StreamError::from(err))); } } } @@ -6187,6 +6249,7 @@ mod serve_group_test { min_delay: None, ranges: vec![lite::Range::span(None, 0, None, None)], order: Order::Descending, + terms: 0, }; run.update(update(Some(0))); assert_eq!(track.demand().wanted(), Some(0), "stale request is not a join"); @@ -6251,6 +6314,7 @@ mod serve_group_test { max_delay: Duration::ZERO, min_delay: None, ranges: vec![lite::Range::span(None, 0, None, None)], + terms: 0, }); { let mut poll = std::pin::pin!(kio::wait(|waiter| run.poll(&mut writer, waiter))); @@ -6366,6 +6430,7 @@ mod serve_group_test { max_delay: Duration::from_secs(3600), min_delay: None, ranges: vec![lite::Range::span(Some(5), 2, Some(7), Some(3))], + terms: 0, }); track @@ -6532,6 +6597,7 @@ mod serve_group_test { max_delay: Duration::from_secs(3600), min_delay: None, ranges: vec![lite::Range::span(None, 0, None, None)], + terms: 0, }); let second = track .append_datagram(Timestamp::from_millis(2).unwrap(), b"b".as_slice()) @@ -6659,9 +6725,10 @@ mod serve_group_test { } } - /// A skipped sequence is accounted for by DROP rather than a stream count. + /// A sequence the publisher never produced is accounted for by SKIP rather than a stream + /// count. #[kio::test] - async fn lite07_end_drops_skipped_sequences() { + async fn lite07_end_skips_sequences_never_produced() { let mut track = track::Producer::new( Arc::new(broadcast::Info::default()), "test", @@ -6677,12 +6744,9 @@ mod serve_group_test { track.finish().unwrap(); run.await.unwrap(); - // SUBSCRIBE_START with the range starting at 0, a DROP for the skipped group 1 - // with NOT_FOUND (0x33), which the end gives up on at once, and SUBSCRIBE_END at 3. - assert_eq!( - *log.writes.lock().unwrap(), - [0, 4, 1, 0, 1, 1, 2, 3, 1, 1, 0x33, 1, 2, 3, 0] - ); + // SUBSCRIBE_START with the range starting at 0, a SKIP for group 1, which the end + // gives up on at once, and SUBSCRIBE_END at 3. + assert_eq!(*log.writes.lock().unwrap(), [0, 4, 1, 0, 1, 1, 3, 2, 1, 1, 1, 2, 3, 0]); } /// A relay that holds only the first group of a backlog answers with the largest its @@ -6737,10 +6801,11 @@ mod serve_group_test { drops } - /// A stale group already named in SUBSCRIBE_DROP stays dropped once SUBSCRIBE_UPDATE - /// widens the max age past it: the subscriber may have settled it already. + /// A stale group already named in SUBSCRIBE_DROP stays dropped before lite-07 once + /// SUBSCRIBE_UPDATE widens the max age past it, since those subscribers settle a drop for + /// good. Lite-07 re-offers it, and only it: the groups served around it are never resent. #[kio::test] - async fn a_reported_stale_drop_survives_a_widened_max_age() { + async fn a_widened_max_age_reoffers_a_reported_stale_drop_on_lite07_alone() { for version in [Version::Lite05, Version::Lite06, Version::Lite07] { let mut track = track::Producer::new( Arc::new(broadcast::Info::default()), @@ -6769,12 +6834,22 @@ mod serve_group_test { min_delay: None, ranges: vec![lite::Range::span(None, 0, None, None)], order: Order::Descending, + terms: 1, }); while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + let (opened, why) = match version { + Version::Lite07 => (5, "the reported stale groups are re-offered, and nothing else"), + _ => (3, "no stream for a dropped group"), + }; assert_eq!( run.ctx.outcomes.opened.load(Ordering::Relaxed), - 3, - "{version:?}: no stream for a dropped group" + opened, + "{version:?}: {why}" + ); + assert_eq!( + drops(&mut writer, &log, version).await, + [(1, 2)], + "{version:?}: no drop again" ); } } @@ -6924,6 +6999,7 @@ mod serve_group_test { max_delay: Duration::from_secs(60), min_delay: None, ranges: vec![lite::Range::span(Some(0), 0, None, None)], + terms: 0, }); while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} assert_eq!(drops(&mut writer, &log, version).await, [(2, 2)]); @@ -7328,6 +7404,7 @@ mod serve_group_test { max_delay: Duration::from_secs(60), min_delay: None, ranges: vec![lite::Range::span(Some(0), 0, None, None)], + terms: 0, }); { let mut run = std::pin::pin!(kio::wait(|waiter| run.poll(&mut writer, waiter))); @@ -7369,6 +7446,7 @@ mod serve_group_test { max_delay: Duration::from_secs(1), min_delay: None, ranges: vec![lite::Range::span(Some(0), 0, None, None)], + terms: 0, }); { let mut run = std::pin::pin!(kio::wait(|waiter| run.poll(&mut writer, waiter))); @@ -7415,6 +7493,7 @@ mod serve_group_test { max_delay: Duration::from_secs(1), min_delay: None, ranges: vec![lite::Range::span(Some(start_group), 0, None, None)], + terms: 0, }; run.update(update(5)); { @@ -7460,6 +7539,7 @@ mod serve_group_test { max_delay: Duration::from_secs(10), min_delay: None, ranges: vec![lite::Range::span(Some(start_group), 0, None, None)], + terms: 0, }; run.update(update(5)); while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} @@ -7506,6 +7586,7 @@ mod serve_group_test { max_delay: Duration::from_secs(60), min_delay: None, ranges: vec![lite::Range::span(None, 0, None, None)], + terms: 0, }); write_group(&mut track, 5, 5); assert_eq!(settle(&mut run), 1, "served a group below the latest"); @@ -7536,6 +7617,7 @@ mod serve_group_test { max_delay: Duration::from_secs(60), min_delay: None, ranges: vec![lite::Range::span(None, 0, None, None)], + terms: 0, }); write_group(&mut track, 0, 0); @@ -7673,6 +7755,7 @@ mod serve_group_test { max_delay: Duration::from_secs(30), min_delay: None, ranges: vec![lite::Range::span(None, 0, None, None)], + terms: 0, }); relay.settle(); assert_eq!(relay.opened(), 1, "the finished group is the live edge"); @@ -7703,6 +7786,7 @@ mod serve_group_test { max_delay: Duration::from_secs(30), min_delay: None, ranges: vec![lite::Range::span(None, 0, None, None)], + terms: 0, }); relay.settle(); assert_eq!(relay.opened(), 3, "group 0 is cached and now in range"); @@ -7730,6 +7814,7 @@ mod serve_group_test { max_delay: Duration::ZERO, min_delay: None, ranges: vec![lite::Range::span(Some(7), 0, None, None)], + terms: 0, }; relay.run.update(update); relay.settle(); @@ -7861,8 +7946,9 @@ mod serve_group_test { assert!(futures::poll!(run.as_mut()).is_pending(), "group 1 is still opening"); assert_eq!( *log.writes.lock().unwrap(), - // The DROP carries OLD (0x34). - [0, 4, 1, 0, 1, 1, 2, 3, 0, 0, 0x34, 1, 2, 2, 0], + // The DROP carries OLD (0x34), judged under a zero budget (Max Age 1) and the + // SUBSCRIBE's terms. + [0, 4, 1, 0, 1, 1, 2, 5, 0, 0, 0x34, 1, 0, 1, 2, 2, 0], "the expired group is dropped and the end precedes stream credit" ); @@ -7874,7 +7960,7 @@ mod serve_group_test { run.await.unwrap(); assert_eq!( *log.writes.lock().unwrap(), - [0, 4, 1, 0, 1, 1, 2, 3, 0, 0, 0x34, 1, 2, 2, 0] + [0, 4, 1, 0, 1, 1, 2, 5, 0, 0, 0x34, 1, 0, 1, 2, 2, 0] ); } } @@ -8563,6 +8649,7 @@ mod tests { max_delay: Duration::ZERO, min_delay: None, ranges: vec![lite::Range::span(None, 0, None, None)], + terms: 0, }; leave_request::(version, &subscribe, wait, close).await; @@ -8623,6 +8710,7 @@ mod tests { max_delay: Duration::ZERO, min_delay: None, ranges: vec![lite::Range::span(None, 0, None, None)], + terms: 0, } .encode(&mut crate::coding::Encoder::new(&mut script, version.into()), version) .unwrap(); @@ -8840,6 +8928,7 @@ mod tests { min_delay: None, // A relay asks from its cached group, so the SUBSCRIBE is not a live-edge join. ranges: vec![lite::Range::span(Some(1), 0, None, None)], + terms: 0, } .encode(&mut crate::coding::Encoder::new(&mut script, version.into()), version) .unwrap(); @@ -8863,6 +8952,7 @@ mod tests { max_delay: Duration::ZERO, min_delay: None, ranges: vec![lite::Range::span(Some(1), 0, None, None)], + terms: 0, }; session.push(&update(Some(2), 3).encode_bytes(version).unwrap()); session.push(&update(Some(2), 7).encode_bytes(version).unwrap()); @@ -8922,6 +9012,7 @@ mod tests { max_delay: Duration::from_secs(3600), min_delay: None, ranges: vec![lite::Range::span(None, 0, None, None)], + terms: 0, } } @@ -8961,6 +9052,7 @@ mod tests { max_delay: Duration::from_secs(3600), min_delay: None, ranges: vec![lite::Range::span(None, 0, None, None)], + terms: 0, } .encode(&mut crate::coding::Encoder::new(&mut script, version.into()), version) .unwrap(); @@ -9383,6 +9475,7 @@ mod tests { max_delay: Duration::ZERO, min_delay: None, ranges: vec![lite::Range::span(None, 0, None, None)], + terms: 0, }; let session = ScriptedSession::new(request.encode_bytes(version).unwrap().to_vec()); let log = session.log.clone(); @@ -9495,6 +9588,7 @@ mod tests { max_delay: Duration::ZERO, min_delay: None, ranges: vec![lite::Range::span(None, 0, None, None)], + terms: 0, }; let session = ScriptedSession::new(request.encode_bytes(version).unwrap().to_vec()); let log = session.log.clone(); @@ -9633,6 +9727,7 @@ mod tests { max_delay: Duration::ZERO, min_delay: None, ranges: vec![lite::Range::span(None, 0, None, None)], + terms: 0, }; let mut script = Vec::new(); msg.encode(&mut crate::coding::Encoder::new(&mut script, version.into()), version) diff --git a/rs/moq-net/src/lite/subscribe.rs b/rs/moq-net/src/lite/subscribe.rs index efabfce87ae..c8233a266df 100644 --- a/rs/moq-net/src/lite/subscribe.rs +++ b/rs/moq-net/src/lite/subscribe.rs @@ -30,6 +30,10 @@ pub struct Subscribe<'a> { /// Which pending group to send first. Lite07+ only; earlier wires decode as /// [`Order::Descending`] and cannot ask for anything else. pub order: Order, + /// The range terms id: every SUBSCRIBE_UPDATE carries a larger one, and a stale + /// [`SubscribeDrop`] echoes the one it was judged under. Lite07+ only; earlier wires + /// decode as 0 and drop it on encode. + pub terms: u64, } impl Subscribe<'_> { @@ -45,6 +49,7 @@ impl Subscribe<'_> { min_delay: self.min_delay, ranges: self.ranges, order: self.order, + terms: self.terms, } } } @@ -142,6 +147,7 @@ impl<'a> Message<'a> for Subscribe<'a> { (order, max_delay, min_delay, decode_ranges(r, version)?) } }; + let terms = decode_terms(r, version)?; Ok(Self { id, @@ -153,6 +159,7 @@ impl<'a> Message<'a> for Subscribe<'a> { min_delay, ranges, order, + terms, }) } @@ -173,11 +180,28 @@ impl<'a> Message<'a> for Subscribe<'a> { encode_ranges(w, version, &self.ranges)?; } } + encode_terms(w, version, self.terms)?; Ok(()) } } +/// Decode the range `Terms` after the ranges. A wire without the field decodes as 0. +fn decode_terms(r: &mut Decoder<'_>, version: Version) -> Result { + match version.has_skip() { + true => r.varint(), + false => Ok(0), + } +} + +/// Encode the range `Terms`, dropping it on a wire without the field. +fn encode_terms(w: &mut Encoder<'_>, version: Version, terms: u64) -> Result<(), EncodeError> { + match version.has_skip() { + true => w.varint(terms), + false => Ok(()), + } +} + /// Decode the `Subscriber Min Age` after the max age: milliseconds plus one, zero meaning not /// stated. A wire without the field decodes as not stated. fn decode_min_delay(r: &mut Decoder<'_>, version: Version) -> Result, DecodeError> { @@ -649,6 +673,8 @@ pub struct SubscribeUpdate { pub ranges: Vec, /// See [`Subscribe::order`]. pub order: Order, + /// See [`Subscribe::terms`]: larger than the subscription's previous terms. + pub terms: u64, } impl<'a> Message<'a> for SubscribeUpdate { @@ -666,6 +692,7 @@ impl<'a> Message<'a> for SubscribeUpdate { let max_delay = std::time::Duration::from_millis(r.varint()?); let min_delay = decode_min_delay(r, version)?; let ranges = decode_ranges(r, version)?; + let terms = decode_terms(r, version)?; let wanted_group = match version.has_refresh() { true => r.varint()?.checked_sub(1), @@ -679,6 +706,7 @@ impl<'a> Message<'a> for SubscribeUpdate { min_delay, ranges, order, + terms, }) } @@ -695,6 +723,7 @@ impl<'a> Message<'a> for SubscribeUpdate { w.varint(u64::try_from(self.max_delay.as_millis()).map_err(|_| EncodeError::BoundsExceeded)?)?; encode_min_delay(w, version, self.min_delay)?; encode_ranges(w, version, &self.ranges)?; + encode_terms(w, version, self.terms)?; if version.has_refresh() { w.varint_opt(self.wanted_group)?; @@ -703,13 +732,15 @@ impl<'a> Message<'a> for SubscribeUpdate { } } -/// Indicates that one or more groups have been dropped. +/// Names groups that existed but will not be delivered: aborted or reset, evicted or +/// expired with no copy left, or stale. /// /// The range `[start, end]` is inclusive on both ends. For example, /// `start = 5, end = 7` means groups 5, 6, and 7 were dropped. /// -/// Lite03+; publishers send it on Lite05 and later. -#[derive(Clone, Debug)] +/// Lite03+; publishers send it on Lite05 and later. Before lite-07 it also names sequences +/// never produced, which lite-07 names in [`SubscribeSkip`]. +#[derive(Clone, Debug, PartialEq, Eq)] pub struct SubscribeDrop { /// The first absolute group sequence in the dropped range. pub start: u64, @@ -720,12 +751,27 @@ pub struct SubscribeDrop { /// Why the groups were given up: a stream error code on lite-07 and later, or an /// application-specific code before, where 0 means simply unavailable. pub error: u64, + + /// The budget a stale verdict was judged under, `None` for a drop that does not depend + /// on it. Lite07+ only, encoded as milliseconds plus one; earlier wires decode as `None`. + pub max_age: Option, + + /// The range terms id ([`Subscribe::terms`]) a stale verdict was judged under. Lite07+ + /// only; earlier wires decode as 0. + pub terms: u64, } #[cfg(feature = "async")] impl SubscribeDrop { - /// Name `groups` given up with `err`, as `version` carries it. - pub fn new(groups: std::ops::RangeInclusive, err: &StreamError, version: Version) -> Self { + /// Name `groups` given up with `err`, as `version` carries it, judged under `max_age` + /// and `terms` when the verdict depends on the budget. + pub fn new( + groups: std::ops::RangeInclusive, + err: &StreamError, + max_age: Option, + terms: u64, + version: Version, + ) -> Self { Self { start: *groups.start(), end: *groups.end(), @@ -734,6 +780,8 @@ impl SubscribeDrop { true => version.encode_stream_code(err).into(), false => 0, }, + max_age: max_age.filter(|_| version.has_skip()), + terms: if version.has_skip() { terms } else { 0 }, } } @@ -760,15 +808,22 @@ impl<'a> Message<'a> for SubscribeDrop { _ => {} } - let msg = Self { - start: r.varint()?, - end: r.varint()?, - error: r.varint()?, - }; - if msg.start.saturating_sub(msg.end) > 1 { + let (start, end) = (r.varint()?, r.varint()?); + if start.saturating_sub(end) > 1 { return Err(DecodeError::InvalidValue); } - Ok(msg) + let error = r.varint()?; + let (max_age, terms) = match version.has_skip() { + true => (r.varint_opt()?.map(std::time::Duration::from_millis), r.varint()?), + false => (None, 0), + }; + Ok(Self { + start, + end, + error, + max_age, + terms, + }) } fn encode_msg(&self, w: &mut Encoder<'_>, version: Version) -> Result<(), EncodeError> { @@ -786,11 +841,62 @@ impl<'a> Message<'a> for SubscribeDrop { w.varint(self.start)?; w.varint(self.end)?; w.varint(self.error)?; + if version.has_skip() { + let max_age = self + .max_age + .map(|age| u64::try_from(age.as_millis()).map_err(|_| EncodeError::BoundsExceeded)) + .transpose()?; + w.varint_opt(max_age)?; + w.varint(self.terms)?; + } Ok(()) } } +/// Names sequences the publisher never produced, such as skipped sequence numbers or ones +/// past a cut: final, with no error, ending no reader. +/// +/// The range `[start, end]` is inclusive on both ends. Lite07+ only; earlier versions name +/// them in a [`SubscribeDrop`]. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct SubscribeSkip { + /// The first absolute group sequence in the skipped range. + pub start: u64, + + /// The last absolute group sequence in the skipped range (inclusive). + pub end: u64, +} + +impl<'a> Message<'a> for SubscribeSkip { + const CONTROL: Option = Some(crate::stats::Message::Subscribe); + fn decode_msg(r: &mut Decoder<'a>, version: Version) -> Result { + if !version.has_skip() { + return Err(DecodeError::Version); + } + let msg = Self { + start: r.varint()?, + end: r.varint()?, + }; + if msg.start.saturating_sub(msg.end) > 1 { + return Err(DecodeError::InvalidValue); + } + Ok(msg) + } + + fn encode_msg(&self, w: &mut Encoder<'_>, version: Version) -> Result<(), EncodeError> { + if !version.has_skip() { + return Err(EncodeError::Version); + } + if self.start.saturating_sub(self.end) > 1 { + return Err(EncodeError::InvalidState); + } + w.varint(self.start)?; + w.varint(self.end)?; + Ok(()) + } +} + /// A response message on the subscribe stream, prefixed with a type discriminator /// on Lite03+. /// @@ -798,12 +904,14 @@ impl<'a> Message<'a> for SubscribeDrop { /// - Lite03/04: `0x0` SUBSCRIBE_OK, `0x1` SUBSCRIBE_DROP. /// - Lite05+: `0x0` SUBSCRIBE_START, `0x1` SUBSCRIBE_END, `0x2` SUBSCRIBE_DROP /// (SUBSCRIBE_OK was removed; acceptance is implicit). +/// - Lite07+: also `0x3` SUBSCRIBE_SKIP. #[derive(Clone, Debug)] pub enum SubscribeResponse { Ok(SubscribeOk), Start(SubscribeStart), End(SubscribeEnd), Drop(SubscribeDrop), + Skip(SubscribeSkip), } /// Write a `type` varint followed by the size-prefixed message body. @@ -834,6 +942,7 @@ impl Encode for SubscribeResponse { Self::Start(start) => encode_typed(w, 0, start, version)?, Self::End(end) => encode_typed(w, 1, end, version)?, Self::Drop(drop) => encode_typed(w, 2, drop, version)?, + Self::Skip(skip) => encode_typed(w, 3, skip, version)?, Self::Ok(_) => return Err(EncodeError::Version), }, } @@ -860,6 +969,7 @@ impl<'a> Decode<'a, Version> for SubscribeResponse { 0 => Ok(Self::Start(SubscribeStart::decode(buf, version)?)), 1 => Ok(Self::End(SubscribeEnd::decode(buf, version)?)), 2 => Ok(Self::Drop(SubscribeDrop::decode(buf, version)?)), + 3 if version.has_skip() => Ok(Self::Skip(SubscribeSkip::decode(buf, version)?)), _ => Err(DecodeError::InvalidMessage(typ)), } } @@ -882,7 +992,9 @@ mod test { SubscribeDrop { start: 3, end: 1, - error: 0 + error: 0, + max_age: None, + terms: 0, } .encode_bytes(version) .is_err() @@ -893,7 +1005,13 @@ mod test { (1 << 62) - 1 }; for (start, end) in [(2, 1), (max, max)] { - let msg = SubscribeDrop { start, end, error: 0 }; + let msg = SubscribeDrop { + start, + end, + error: 0, + max_age: None, + terms: 0, + }; let data = msg.encode_bytes(version).unwrap(); let (decoded, _) = SubscribeDrop::decode_slice(&data, version).unwrap(); assert_eq!((decoded.start, decoded.end), (start, end)); @@ -1013,11 +1131,14 @@ mod test { start: 1, end: 3, error: 7, + max_age: None, + terms: 0, }); let mut buf = Vec::new(); resp.encode(&mut Encoder::new(&mut buf, Version::Lite07.into()), Version::Lite07) .unwrap(); - assert_eq!(buf, [2, 3, 1, 3, 7]); + // A drop no budget reverses has a Max Age of 0. + assert_eq!(buf, [2, 5, 1, 3, 7, 0, 0]); match crate::coding::decode_buf(&mut buf.as_slice(), Version::Lite07, |r, v| { SubscribeResponse::decode(r, v) }) @@ -1028,12 +1149,76 @@ mod test { } } + /// A stale drop names the budget, as milliseconds plus one so a zero budget is 1, and the + /// terms it was judged under; earlier versions carry neither. + #[test] + fn a_stale_drop_names_its_max_age_and_terms_on_lite07() { + for max_age in [std::time::Duration::ZERO, std::time::Duration::from_millis(300)] { + let msg = SubscribeDrop { + start: 4, + end: 4, + error: 0x34, + max_age: Some(max_age), + terms: 2, + }; + let buf = msg.encode_bytes(Version::Lite07).unwrap(); + let (decoded, _) = SubscribeDrop::decode_slice(&buf, Version::Lite07).unwrap(); + assert_eq!(decoded, msg); + } + let msg = SubscribeDrop { + start: 4, + end: 4, + error: 0, + max_age: Some(std::time::Duration::from_millis(300)), + terms: 2, + }; + let buf = msg.encode_bytes(Version::Lite06).unwrap(); + assert_eq!(buf.as_ref(), [3, 4, 4, 0]); + let (decoded, _) = SubscribeDrop::decode_slice(&buf, Version::Lite06).unwrap(); + assert_eq!((decoded.max_age, decoded.terms), (None, 0)); + } + + /// SUBSCRIBE_SKIP is type 3 on lite-07 and unknown before it. + #[test] + fn subscribe_skip_is_lite07_only() { + let resp = SubscribeResponse::Skip(SubscribeSkip { start: 1, end: 3 }); + let buf = resp.encode_bytes(Version::Lite07).unwrap(); + assert_eq!(buf.as_ref(), [3, 2, 1, 3]); + match SubscribeResponse::decode_slice(&buf, Version::Lite07).unwrap().0 { + SubscribeResponse::Skip(skip) => assert_eq!(skip, SubscribeSkip { start: 1, end: 3 }), + other => panic!("expected Skip, got {other:?}"), + } + assert!(matches!( + SubscribeSkip::decode_slice(&[2, 3, 1], Version::Lite07), + Err(DecodeError::InvalidValue) + )); + assert!(resp.encode_bytes(Version::Lite06).is_err()); + assert!(matches!( + SubscribeResponse::decode_slice(&buf, Version::Lite06), + Err(DecodeError::InvalidMessage(3)) + )); + } + + /// SUBSCRIBE and SUBSCRIBE_UPDATE carry their range terms on lite-07 alone. + #[test] + fn range_terms_roundtrip_only_on_lite07() { + for version in [Version::Lite06, Version::Lite07] { + let mut msg = subscribe_sample(); + msg.terms = 5; + let buf = encode(&msg, version).unwrap(); + let decoded = crate::coding::decode_from(&mut buf.as_slice(), version, Subscribe::decode_msg).unwrap(); + assert_eq!(decoded.terms, if version.has_skip() { 5 } else { 0 }); + } + } + #[test] fn subscribe_drop_is_type_2_on_lite05() { let resp = SubscribeResponse::Drop(SubscribeDrop { start: 1, end: 3, error: 0, + max_age: None, + terms: 0, }); let mut buf = Vec::new(); resp.encode(&mut Encoder::new(&mut buf, Version::Lite05.into()), Version::Lite05) @@ -1054,6 +1239,8 @@ mod test { start: 1, end: 3, error: 0, + max_age: None, + terms: 0, }); let mut buf = Vec::new(); resp.encode(&mut Encoder::new(&mut buf, Version::Lite04.into()), Version::Lite04) @@ -1082,6 +1269,7 @@ mod test { start: absolute(7, 4), end: absolute(9, 3), }], + terms: 0, } } @@ -1260,6 +1448,7 @@ mod test { min_delay: None, ranges: msg.ranges.clone(), order: crate::track::Order::Ascending, + terms: 0, }; let buf = update.encode_bytes(Version::Lite07).unwrap(); let got = crate::coding::decode_buf(&mut buf.as_ref(), Version::Lite07, |r, v| SubscribeUpdate::decode(r, v)) @@ -1278,9 +1467,9 @@ mod test { msg.ranges = vec![Range::OLDEST]; let mut buf = encode(&msg, version).unwrap(); - // The list is last: a count of one, then two anchors of type 0. - let at = buf.len() - 3; - assert_eq!(&buf[at..], [1, 0, 0]); + // The list is last but for the terms: a count of one, then two anchors of type 0. + let at = buf.len() - 4; + assert_eq!(&buf[at..], [1, 0, 0, 0]); buf[at] = 0; buf.truncate(at + 1); assert!(matches!( @@ -1289,7 +1478,7 @@ mod test { )); let mut buf = encode(&msg, version).unwrap(); - let last = buf.len() - 1; + let last = buf.len() - 2; buf[last] = 4; assert!(matches!( crate::coding::decode_from(&mut buf.as_slice(), version, Subscribe::decode_msg), @@ -1355,6 +1544,7 @@ mod test { min_delay: None, ranges: vec![Range::OLDEST], order: crate::track::Order::Descending, + terms: 0, }; let buf = update.encode_bytes(version).unwrap(); let got = crate::coding::decode_buf(&mut buf.as_ref(), version, |r, v| SubscribeUpdate::decode(r, v)) @@ -1388,6 +1578,7 @@ mod test { min_delay, ranges: vec![Range::OLDEST], order: crate::track::Order::Descending, + terms: 0, }; let buf = update.encode_bytes(version).unwrap(); let got = crate::coding::decode_buf(&mut buf.as_ref(), version, |r, v| SubscribeUpdate::decode(r, v)) @@ -1416,6 +1607,7 @@ mod test { min_delay: None, ranges: vec![Range::OLDEST], order, + terms: 0, }; let mut buf = Vec::new(); update diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index 787ac6fc9c1..5848efa53dd 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -2465,6 +2465,7 @@ mod tests { bounded: false, updated: false, _working: crate::session::Work::default().start(), + terms: 0, }) } @@ -2670,6 +2671,7 @@ mod tests { bounded: false, updated: false, _working: crate::session::Work::default().start(), + terms: 0, }); if update { h.serve @@ -3127,6 +3129,7 @@ mod tests { bounded: false, updated: false, _working: crate::session::Work::default().start(), + terms: 0, }; assert_eq!( sub.owed(None), @@ -4156,6 +4159,8 @@ mod tests { start: 2, end: 2, error: code.into(), + max_age: None, + terms: 0, }), ] { resp.encode( @@ -7585,6 +7590,9 @@ struct SubStream { /// Whether an update changed the ranges since the SUBSCRIBE, which a publisher may /// FIN before reading. updated: bool, + /// The range terms id of the latest SUBSCRIBE or SUBSCRIBE_UPDATE sent: a stale drop + /// judged under an earlier one is no verdict on the current terms. + terms: u64, /// Holds the session busy for as long as the subscription is open. _working: crate::session::Working, } @@ -8093,7 +8101,7 @@ impl TrackServe { if start_moved { let _ = producer.start_at(active.start); } - buffer_update(active)?; + buffer_update(active, producer)?; } Ok(Begin::None) } @@ -8345,9 +8353,15 @@ fn wire_max_delay(max_delay: Option) -> std::time::Duration max_delay.unwrap_or(std::time::Duration::from_millis(super::MAX_SAFE_DELAY_MS)) } -/// Buffer the current SUBSCRIBE parameters. The caller flushes. -fn buffer_update(active: &mut SubStream) -> Result<(), Error> { +/// Buffer the current SUBSCRIBE parameters under new range terms. The caller flushes. +/// +/// A stale drop judged under earlier terms may still be in flight, and the update may have +/// moved the range it was judged against, so the runs `serving` holds as stale become +/// unknown again: a reader fills them by range rather than trust an old verdict. +fn buffer_update(active: &mut SubStream, serving: &track::Producer) -> Result<(), Error> { active.bounded |= bounded(&active.ranges); + active.terms += 1; + serving.supersede_stale(); active.stream.writer.buffer(&lite::SubscribeUpdate { wanted_group: active.wanted, priority: active.priority, @@ -8355,6 +8369,7 @@ fn buffer_update(active: &mut SubStream) -> Result<(), Error> { min_delay: active.min_delay, ranges: active.ranges.clone(), order: active.order, + terms: active.terms, }) } @@ -8398,6 +8413,7 @@ impl Establish { min_delay: self.subscription.delay.min, ranges: self.ranges.clone(), order: self.subscription.order, + terms: 0, }; stream = stream.with_stats(self.serve.subscriber.origin.stats()); stream.writer.buffer(&lite::ControlType::Subscribe)?; @@ -8452,6 +8468,7 @@ impl Establish { wanted: None, bounded: bounded(&self.ranges), updated: false, + terms: 0, _working: self.session.work().start(), } } @@ -9535,7 +9552,7 @@ impl ServeLoop { && active.wanted != Some(wanted) { active.wanted = Some(wanted); - if let Err(err) = buffer_update(active) { + if let Err(err) = buffer_update(active, &self.serving) { return Poll::Ready(ServeEnd::GiveBack(err)); } continue; @@ -9670,16 +9687,32 @@ impl ServeLoop { // are accounted for without a stream, and a reader of // one still arriving ends with the drop's error now. lite::SubscribeResponse::Drop(dropped) => { + let groups = dropped.start..dropped.end.saturating_add(1); + if let Ok(mut tail) = active.tail.write() { + tail.account(groups.clone(), serve.subscriber.runtime.now()); + } + // A stale verdict under terms an update has since replaced + // says nothing of the current ones: the groups are unknown, + // and a reader that wants one fills it by range. + let superseded = dropped.terms < active.terms; + match dropped.max_age { + Some(_) if superseded => {} + stale => self.serving.give_up( + groups, + track::Hole::Drop { + error: dropped.error(serve.subscriber.version), + stale, + }, + ), + } + } + // Never produced, so nothing is waited for and no reader ends. + lite::SubscribeResponse::Skip(skipped) => { + let groups = skipped.start..skipped.end.saturating_add(1); if let Ok(mut tail) = active.tail.write() { - tail.account( - dropped.start..dropped.end.saturating_add(1), - serve.subscriber.runtime.now(), - ); + tail.account(groups.clone(), serve.subscriber.runtime.now()); } - self.serving.drop_groups( - dropped.start..dropped.end.saturating_add(1), - dropped.error(serve.subscriber.version), - ); + self.serving.skip_groups(groups); } // OK just resolves the range (the producer already orders // groups). diff --git a/rs/moq-net/src/lite/version.rs b/rs/moq-net/src/lite/version.rs index 4ee7c9687c6..aa79d0f9085 100644 --- a/rs/moq-net/src/lite/version.rs +++ b/rs/moq-net/src/lite/version.rs @@ -289,6 +289,17 @@ impl Version { ) } + /// Whether holes split into SUBSCRIBE_SKIP, for sequences never produced, and + /// SUBSCRIBE_DROP, for groups that existed, whose drop names the `Max Age` and range + /// `Terms` a stale verdict was judged under; SUBSCRIBE and SUBSCRIBE_UPDATE carry those + /// `Terms`. Added in lite-07; earlier versions name every hole in SUBSCRIBE_DROP. + pub(crate) fn has_skip(self) -> bool { + !matches!( + self, + Self::Lite01 | Self::Lite02 | Self::Lite03 | Self::Lite04 | Self::Lite05 | Self::Lite06 + ) + } + /// Whether SUBSCRIBE_END carries the error the track ended with, so an errored end /// keeps its history across a hop. Added in lite-07; earlier versions can only reset. pub(crate) fn has_end_error(self) -> bool { diff --git a/rs/moq-net/src/model/resume.rs b/rs/moq-net/src/model/resume.rs index 10e31cc604c..c075845032b 100644 --- a/rs/moq-net/src/model/resume.rs +++ b/rs/moq-net/src/model/resume.rs @@ -895,7 +895,7 @@ impl Subscriber { &mut self, delivered: &Delivered, waiter: &kio::Waiter, - ) -> Poll<(std::ops::Range, StreamError)> { + ) -> Poll<(std::ops::Range, track::Hole)> { self.reader().poll_drop(delivered, waiter) } @@ -1095,7 +1095,7 @@ struct Reader { out: Vec<(u64, Weak<()>)>, /// Runs the serving copy gave up that the reader has yet to report; see /// [`Reader::poll_drop`]. - drops: RangeMap, + drops: RangeMap, } impl Reader { @@ -1849,16 +1849,16 @@ impl Reader { /// through its [`Recover`]. A copy it replaced may still deliver what is under its bound, /// so a run there waits until that copy no longer feeds. #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] - fn poll_drop(&mut self, delivered: &Delivered, waiter: &kio::Waiter) -> Poll<(std::ops::Range, StreamError)> { + fn poll_drop(&mut self, delivered: &Delivered, waiter: &kio::Waiter) -> Poll<(std::ops::Range, track::Hole)> { let mut given_up = Vec::new(); if let Poll::Ready(Some(sub)) = self.poll_serving(waiter) { while let Poll::Ready(run) = sub.poll_drop(waiter) { given_up.push(run); } } - for (run, err) in given_up { - if !route_failed(&Error::Stream(err.clone())) { - self.drops.insert(run, err); + for (run, hole) in given_up { + if !route_failed(&Error::Stream(hole.error())) { + self.drops.insert(run, hole); } } let covered: Vec<_> = self @@ -3631,8 +3631,8 @@ mod test { /// Every run the reader gives up right now. fn dropped(sub: &mut track::Subscriber) -> Vec<(std::ops::Range, StreamError)> { let mut runs = Vec::new(); - while let Poll::Ready(run) = sub.poll_drop(&kio::Waiter::noop()) { - runs.push(run); + while let Poll::Ready((run, hole)) = sub.poll_drop(&kio::Waiter::noop()) { + runs.push((run, hole.error())); } runs } diff --git a/rs/moq-net/src/model/resume_model.rs b/rs/moq-net/src/model/resume_model.rs index 1f2f2128155..884bec66566 100644 --- a/rs/moq-net/src/model/resume_model.rs +++ b/rs/moq-net/src/model/resume_model.rs @@ -719,7 +719,7 @@ impl Model { if reader.reading.contains_key(&group) { return Err(fail( index, - format!("dropped {range:?} ({err}) after handing out {group}"), + format!("dropped {range:?} ({err:?}) after handing out {group}"), )); } } diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index d0db688ce30..438b1e82399 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -32,9 +32,11 @@ pub(super) use demand::Preferences; use demand::Subscriptions; mod fill; +mod hole; mod miss; pub use fill::Fill; use fill::FillState; +pub(crate) use hole::Hole; #[cfg_attr(not(feature = "async"), allow(unused_imports, reason = "async-only"))] pub(crate) use fill::{GroupFill, Readers as FillReaders}; @@ -494,11 +496,11 @@ pub(crate) struct TrackState { // budget, but neither this copy's timestamps nor a fill can judge an older group against // a reader's budget. A widened budget reaches lower groups through the feed's re-offer. fills_from_start: bool, - // Sequences the upstream publisher gave up, with the error each was given up with: an - // incoming SUBSCRIBE_DROP or a producer cut. Kept apart from the cache so reclaiming a - // slot cannot erase a disposition a relay must forward, and as ranges so a wide DROP - // costs one entry. - drops: RangeMap, + // Sequences the upstream publisher gave up, with why: an incoming SUBSCRIBE_SKIP or + // SUBSCRIBE_DROP, or a producer cut. Kept apart from the cache so reclaiming a slot + // cannot erase a disposition a relay must forward, and as ranges so a wide one costs + // one entry. + drops: RangeMap, // Bumped on every insert into `drops`, so a cursor rescans it only after a change. drops_revision: u64, drops_changes: ledger::Changes, @@ -1122,18 +1124,46 @@ impl TrackState { } } - /// Record sequences the upstream publisher gave up, with `err`. - fn drop_groups(&mut self, groups: ops::Range, err: StreamError) { + /// Record sequences the upstream publisher gave up, with why. + fn drop_groups(&mut self, groups: ops::Range, hole: Hole) { if groups.is_empty() { return; } self.cache.wakes().arrived(groups.clone()); - self.change_drops(groups, Some(err)); + self.change_drops(groups, Some(hole)); + } + + /// Why a run a reader's misses gave up will not come: never produced when this track's + /// own publisher ended without it ever arriving, whatever else judged it; otherwise stale + /// under `stale` when that is the verdict, or unknown, so simply not found. + fn missed(&self, run: &ops::Range, stale: Option) -> Hole { + let never = !self.received + && self.final_sequence().is_some_and(|fin| run.end <= fin) + && self.arrivals.accounted(run.clone()).next().is_none() + && self.lookup.range(run.clone()).next().is_none(); + match (never, stale) { + (true, _) => Hole::Skip, + (false, Some(budget)) => Hole::stale(budget), + (false, None) => Hole::drop(StreamError::NotFound), + } + } + + /// Forget every run held as stale; see [`Producer::supersede_stale`]. + fn supersede_stale(&mut self) { + let stale: Vec<_> = self + .drops + .iter() + .filter(|(_, hole)| hole.stale_budget().is_some()) + .map(|(run, _)| run.clone()) + .collect(); + for run in stale { + self.change_drops(run, None); + } } /// Keep the change index aligned with the range map, including the neighbours an /// insertion coalesces and the two pieces a removal can leave. - fn change_drops(&mut self, groups: ops::Range, err: Option) { + fn change_drops(&mut self, groups: ops::Range, hole: Option) { let mut affected = groups.start.saturating_sub(1)..groups.end.saturating_add(1); let old: Vec<_> = self.drops.overlapping(&affected).map(|(run, _)| run.clone()).collect(); for run in old { @@ -1141,8 +1171,8 @@ impl TrackState { affected.start = affected.start.min(run.start); affected.end = affected.end.max(run.end); } - match err { - Some(err) => self.drops.insert(groups, err), + match hole { + Some(hole) => self.drops.insert(groups, hole), None => self.drops.remove(groups), } for (run, _) in self.drops.overlapping(&affected) { @@ -1159,7 +1189,7 @@ impl TrackState { && let Some(err) = slot.group.cut_error() && let Some(end) = sequence.checked_add(1) { - self.drop_groups(sequence..end, StreamError::from(err)); + self.drop_groups(sequence..end, Hole::drop(StreamError::from(err))); } } @@ -1168,7 +1198,7 @@ impl TrackState { if let Some(err) = self.lookup.get(&sequence).and_then(Slot::cut) { return Some(err.clone()); } - self.drops.get(&sequence).cloned().map(Error::Stream) + self.drops.get(&sequence).map(|hole| Error::Stream(hole.error())) } /// Whether a Producer was minted, committed or not. @@ -2106,7 +2136,7 @@ impl TrackState { .filter_map(|(&sequence, _)| Some(sequence..sequence.checked_add(1)?)) .collect(); for groups in open { - self.drop_groups(groups, dropped.clone()); + self.drop_groups(groups, Hole::drop(dropped.clone())); } self.lookup .retain(|_, slot| slot.copies().any(group::Handle::is_finished)); @@ -3745,29 +3775,60 @@ impl Producer { /// waiting on one moves past it, a group still arriving in them ends with `err`, the /// declared end does not wait for them, and a relay serving this track forwards the drop. pub fn drop_groups(&self, groups: ops::Range, err: StreamError) { + self.give_up(groups, Hole::drop(err)); + } + + /// Record that the publisher never produced `groups`, such as skipped sequence numbers: + /// none will arrive, so a reader waiting on one moves past it, the declared end does not + /// wait for them, and a relay serving this track forwards the skip. Unlike + /// [`Self::drop_groups`], it ends no reader. + pub fn skip_groups(&self, groups: ops::Range) { + self.give_up(groups, Hole::Skip); + } + + /// Forget every run held as stale: the terms a received copy's upstream judged them + /// under were replaced, so they are unknown again, and a reader that wants one fills it. + #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] + pub(crate) fn supersede_stale(&self) { + if let Ok(mut state) = self.modify() { + state.supersede_stale(); + } + } + + /// Record `groups` given up as `hole` describes. + pub(crate) fn give_up(&self, groups: ops::Range, hole: Hole) { if groups.is_empty() { return; } + let error = match &hole { + Hole::Drop { error, .. } => Some(error.clone()), + Hole::Skip => None, + }; let open: Vec = match self.modify() { Ok(mut state) => { - state.drop_groups(groups.clone(), err.clone()); + state.drop_groups(groups.clone(), hole); // Only the live feed's copy: a drop judges the subscription, so a fetched head - // or fetch-only backfill may still complete. - state - .lookup - .range(groups) - .filter(|(_, slot)| slot.visible) - .map(|(_, slot)| &slot.group) - .filter(|group| !group.is_aborted()) - .cloned() - .collect() + // or fetch-only backfill may still complete. A skip names nothing produced. + match error { + None => Vec::new(), + Some(_) => state + .lookup + .range(groups) + .filter(|(_, slot)| slot.visible) + .map(|(_, slot)| &slot.group) + .filter(|group| !group.is_aborted()) + .cloned() + .collect(), + } } Err(_) => return, }; // Settle the readers of a group still arriving now, rather than once its stream's // reset lands, with the track guard released. - for group in open { - group.abort_open(Error::Stream(err.clone())); + if let Some(error) = error { + for group in open { + group.abort_open(Error::Stream(error.clone())); + } } } @@ -6047,6 +6108,23 @@ struct Drift { /// The budget is the track's max age bound, which no widening passes: its verdicts are final. bounded: bool, edge: Edge, + /// The lowest group from which this reader reads every group up past the live edge, + /// `None` while a range end or a cap bounds it. + covers_from: Option, +} + +impl Drift { + /// Whether `hole`, over a run from `start`, settles this reader. Any hole but a received + /// stale one does. A stale verdict holds for a reader no wider than the budget it was + /// judged under, whose range covers the run and every group above it: its own successor + /// and edge are then no older than those the verdict was judged against, wherever the + /// upstream's range ended. Any other reader treats the run as unknown, and fills it. + fn settles(&self, start: u64, hole: &Hole) -> bool { + match hole.stale_budget() { + None => true, + Some(budget) => self.budget <= budget && self.covers_from.is_some_and(|from| from <= start), + } + } } /// Keeps one handed-out group tied to the subscription whose cursor selected it. @@ -6691,6 +6769,11 @@ struct Cursor { skipped: Skipped, /// The drops a wire publisher forwards, once it first polls for one. drops: Option>, + /// Whether a widening re-offers what this cursor reported stale: a wire whose drops a + /// widening reverses, unlike one whose drops are final. + reoffers_reported: bool, + /// The reported stale runs a widening took back since the owner last drained them. + unreported: Vec>, /// What this cursor misses and asks fills for, once it first reads. misses: Option>, /// The heads a bounded window asked for, once it first finds a group missing one. @@ -6741,11 +6824,13 @@ struct Heads { filled: BTreeMap, } -/// The sequences a [`DropCursor`] reports from: a cursor's span, narrowed to its window. +/// The sequences a [`DropCursor`] reports from: a cursor's span, narrowed to its window, and +/// the drift that says which received stale runs settle it. struct Within<'a> { span: ops::Range, window: &'a Window, revision: u64, + drift: Option<&'a Drift>, } /// What a wire publisher's cursor gave up, found as it changes rather than by rescanning @@ -6755,15 +6840,19 @@ struct DropCursor { #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] cuts: super::aborts::Watch, /// Sequences given up and not reported yet, with why: stale skips, cuts, and upstream - /// drops. - pending: RangeMap, - /// The [`TrackState::drops_revision`], span, and window revision last folded into - /// `pending`. + /// skips and drops. + pending: RangeMap, + /// The [`TrackState::drops_revision`], span, window revision, and what settles received + /// stale runs, last folded into `pending`. #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] - scanned: Option<(u64, ops::Range, u64)>, + scanned: Option<(u64, ops::Range, u64, Option<(Duration, Option)>)>, /// Every sequence this cursor already reported. #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] reported: RangeSet, + /// The reported runs judged stale, with the budget each was judged under, for a wire + /// whose widening re-offers them; see [`Cursor::reoffers_reported`]. + #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] + reported_stale: RangeMap, } #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] @@ -6775,15 +6864,44 @@ impl DropCursor { pending: RangeMap::new(), scanned: None, reported: RangeSet::new(), + reported_stale: RangeMap::new(), + } + } + + /// Take back the reported stale runs judged under a budget narrower than `budget`, and + /// withdraw those not reported yet: a widening reverses their verdict, so they are + /// judged again, and reported again if still given up. + fn widened(&mut self, budget: Duration) -> Vec> { + let pending: Vec<_> = self + .pending + .iter() + .filter(|(_, hole)| hole.stale_budget().is_some_and(|judged| judged < budget)) + .map(|(run, _)| run.clone()) + .collect(); + for run in pending { + self.pending.remove(run); + } + let widened: Vec<_> = self + .reported_stale + .iter() + .filter(|(_, judged)| **judged < budget) + .map(|(run, _)| run.clone()) + .collect(); + for run in &widened { + self.reported_stale.remove(run.clone()); + self.reported.remove(run.clone()); } + // Rescanned, so a received stale run a wider budget no longer settles stays unknown. + self.scanned = None; + widened } - /// Record a run this cursor's fills gave up on as a miss: why it is gone is unknown, so - /// a reason already pending, such as the track's own drop, is kept. - fn give_up(&mut self, run: ops::Range) { + /// Record a run this cursor's fills gave up on as a miss, as `hole` says; a reason + /// already pending, such as the track's own drop, is kept. + fn give_up(&mut self, run: ops::Range, hole: Hole) { let gaps: Vec<_> = self.pending.gaps(&run).collect(); for gap in gaps { - self.pending.insert(gap, StreamError::NotFound); + self.pending.insert(gap, hole.clone()); } } @@ -6793,17 +6911,12 @@ impl DropCursor { && let Some(err) = slot.group.cut_error() && let Some(end) = sequence.checked_add(1) { - self.pending.insert(sequence..end, StreamError::from(err)); + self.pending.insert(sequence..end, Hole::drop(StreamError::from(err))); } } /// The lowest unreported run of `within` given up, now marked reported, with why. - fn next( - &mut self, - state: &TrackState, - cut: BTreeSet, - within: &Within<'_>, - ) -> Option<(ops::Range, StreamError)> { + fn next(&mut self, state: &TrackState, cut: BTreeSet, within: &Within<'_>) -> Option<(ops::Range, Hole)> { let (span, window) = (&within.span, within.window); // A cut already reclaimed from the cache is in the track's drops instead. for sequence in cut { @@ -6811,24 +6924,32 @@ impl DropCursor { self.cut(sequence, slot); } } - // Rescan only after the drops change or the window moves; out-of-window runs were - // skipped and may be owed now. - let scan = (state.drops_revision, span.clone(), within.revision); + // Rescan only after the drops change, the window moves, or what settles a stale run + // does; out-of-window and unsettling runs were skipped and may be owed now. + let settles = within.drift.map(|drift| (drift.budget, drift.covers_from)); + let scan = (state.drops_revision, span.clone(), within.revision, settles); if self.scanned.as_ref() != Some(&scan) { self.scanned = Some(scan); - for (dropped, err) in state.drops.overlapping(span.clone()) { - self.pending - .insert(dropped.start.max(span.start)..dropped.end.min(span.end), err.clone()); + for (dropped, hole) in state.drops.overlapping(span.clone()) { + let run = dropped.start.max(span.start)..dropped.end.min(span.end); + // A received stale run that does not settle this reader is unknown to it. + let settled = match within.drift { + Some(drift) => drift.settles(run.start, hole), + None => hole.stale_budget().is_none(), + }; + if settled { + self.pending.insert(run, hole.clone()); + } } for (&sequence, slot) in state.lookup.range(span.clone()) { self.cut(sequence, slot); } } - while let Some((range, err)) = self + while let Some((range, hole)) = self .pending .iter() .next() - .map(|(range, err)| (range.clone(), err.clone())) + .map(|(range, hole)| (range.clone(), hole.clone())) { self.pending.remove(range.clone()); // Only what this subscriber asked for is owed: the first run of its ranges in it, @@ -6837,14 +6958,17 @@ impl DropCursor { continue; }; if clipped.end < range.end { - self.pending.insert(clipped.end..range.end, err.clone()); + self.pending.insert(clipped.end..range.end, hole.clone()); } if let Some(gap) = self.reported.gaps(&clipped).next() { if gap.end < range.end { - self.pending.insert(gap.end..range.end, err.clone()); + self.pending.insert(gap.end..range.end, hole.clone()); } self.reported.insert(gap.clone()); - return Some((gap, err)); + if let Some(budget) = hole.stale_budget() { + self.reported_stale.insert(gap.clone(), budget); + } + return Some((gap, hole)); } } None @@ -6941,6 +7065,8 @@ impl Cursor { seek_pending: BTreeMap::new(), skipped: Skipped::default(), drops: None, + reoffers_reported: false, + unreported: Vec::new(), misses: None, heads: None, idle: None, @@ -6949,23 +7075,26 @@ impl Cursor { } #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] - fn poll_drop(&mut self, taken: &Delivered, waiter: &kio::Waiter) -> Poll<(ops::Range, StreamError)> { + fn poll_drop(&mut self, taken: &Delivered, waiter: &kio::Waiter) -> Poll<(ops::Range, Hole)> { self.sync(); if self.drops.is_none() { let mut drops = Box::new(DropCursor::new(&self.state.read())); if let Some(misses) = &self.misses { + let state = self.state.read(); for run in misses.given_up().iter() { - drops.give_up(run.clone()); + drops.give_up(run.clone(), state.missed(run, None)); } } self.drops = Some(drops); } + let mut judged = None; if self.filter.refused.is_none() && let Poll::Ready(Ok(drift)) = self.poll_drift(&self.servable, waiter) { // From where the reader reads next: an ordered one asks nothing below what it took. let from = self.next_sequence.unwrap_or(u64::MAX).max(self.floor()); self.probe(from, &drift, taken, waiter); + judged = Some(drift); } // A cap at or below the floor leaves nothing to report, not a reversed span. let floor = self.floor(); @@ -6973,6 +7102,7 @@ impl Cursor { span: floor..self.end_sequence.unwrap_or(u64::MAX).max(floor), window: &self.filter.window, revision: self.filter.revision, + drift: judged.as_ref(), }; let drops = match &mut self.drops { Some(drops) => drops, @@ -7318,6 +7448,18 @@ impl Cursor { /// `budget`: only a widening reverses a stale verdict. Costs the skipped runs, never /// the cache, and nothing when the budget did not widen. fn reoffer(&mut self, budget: Duration) { + // A reported stale run reversed is judged again: a group still cached is owed below + // as a skipped one is, and a missing one is missed again. + if self.reoffers_reported + && let Some(drops) = &mut self.drops + { + for run in drops.widened(budget) { + if let Some(misses) = &mut self.misses { + misses.retry(run.clone()); + } + self.unreported.push(run); + } + } for run in self.skipped.widened(budget) { self.filter .written_off @@ -7328,7 +7470,7 @@ impl Cursor { let old: Vec<_> = drops .pending .overlapping(span.clone()) - .filter(|(_, err)| **err == StreamError::Old) + .filter(|(_, hole)| hole.stale_budget().is_some()) .map(|(range, _)| range.start.max(span.start)..range.end.min(span.end)) .collect(); for range in old { @@ -7402,12 +7544,24 @@ impl Cursor { max_delay = subscription.budget(); Poll::<()>::Pending }); + // Uncapped, an open last range reads everything above its start. + let covers_from = match self.end_sequence { + Some(_) => None, + None => self + .filter + .window + .groups() + .and_then(|groups| groups.last(0, u64::MAX)) + .filter(|run| *run.end() == u64::MAX) + .map(|run| *run.start()), + }; self.poll(waiter, |state| { let bound = state.max_age_bound(); Poll::Ready(Ok(Drift { budget: clamp_max_delay(max_delay, bound), bounded: bound.is_some_and(|bound| max_delay >= bound), edge: state.drift_edge(servable), + covers_from, })) }) } @@ -7642,8 +7796,9 @@ impl Cursor { misses.expire(&self.state.read(), drift, &mut given_up, waiter); } if let Some(drops) = &mut self.drops { - for run in given_up { - drops.give_up(run); + let state = self.state.read(); + for (run, stale) in given_up { + drops.give_up(run.clone(), state.missed(&run, stale)); } } // A refusal taken in may move a time end walking history past it, widening what @@ -7661,8 +7816,9 @@ impl Cursor { let mut given_up = Vec::new(); let group = misses.next_filled(&self.state.read(), order, &mut given_up); if let Some(drops) = &mut self.drops { - for run in given_up { - drops.give_up(run); + let state = self.state.read(); + for (run, stale) in given_up { + drops.give_up(run.clone(), state.missed(&run, stale)); } } group @@ -7811,7 +7967,7 @@ impl Cursor { if let Some(drops) = &mut self.drops && let Some(end) = sequence.checked_add(1) { - drops.pending.insert(sequence..end, StreamError::Old); + drops.pending.insert(sequence..end, Hole::stale(drift.budget)); } match drift.bounded { // A re-offered group stale at the bound is final. @@ -7958,9 +8114,10 @@ impl Cursor { for run in given_up { misses.give_up(run, &mut runs); } + // A head the cursor found missing belonged to a group that existed. if let Some(drops) = &mut self.drops { - for run in runs { - drops.give_up(run); + for (run, _) in runs { + drops.give_up(run, Hole::drop(StreamError::NotFound)); } } } @@ -8900,7 +9057,7 @@ impl Subscriber { /// Stale skips are recorded only once this has been polled, so a reader that never /// forwards drops keeps none. #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] - pub(crate) fn poll_drop(&mut self, waiter: &kio::Waiter) -> Poll<(ops::Range, StreamError)> { + pub(crate) fn poll_drop(&mut self, waiter: &kio::Waiter) -> Poll<(ops::Range, Hole)> { match &mut self.inner { Inner::Plain(cursor) => cursor.poll_drop(&self.delivered, waiter), Inner::Resume(resume, _) => resume.poll_drop(&self.delivered, waiter), @@ -8994,6 +9151,28 @@ impl Subscriber { self } + /// Have a widening re-offer what this reader reported stale through + /// [`Self::poll_drop`], as a wire whose drops a widening reverses does: each run it + /// takes back is judged again, and [`Self::take_unreported`] hands it to the owner. A + /// front's reader reports only what its copies give up, so it keeps its reports. + #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] + pub(crate) fn reoffers_reported(mut self) -> Self { + if let Inner::Plain(cursor) = &mut self.inner { + cursor.reoffers_reported = true; + } + self + } + + /// The runs this reader reported stale that a widening took back since the last call, + /// which are owed again; see [`Self::reoffers_reported`]. + #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] + pub(crate) fn take_unreported(&mut self) -> Vec> { + match &mut self.inner { + Inner::Plain(cursor) => std::mem::take(&mut cursor.unreported), + Inner::Resume(..) => Vec::new(), + } + } + pub(crate) fn set_range(&mut self, groups: impl RangeBounds) { let (start, end) = super::subscription::sequence_bounds(groups); match &mut self.inner { @@ -9119,6 +9298,28 @@ impl Subscriber { } } + /// The budget this subscriber judges staleness under: its max delay, clamped to the + /// track's max age bound. + #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] + pub(crate) fn budget(&self) -> Duration { + let budget = self.subscription().budget(); + match &self.inner { + Inner::Plain(cursor) => clamp_max_delay(budget, cursor.state.read().max_age_bound()), + Inner::Resume(..) => budget, + } + } + + /// Whether this subscriber reads a copy received from a remote publisher, which cannot + /// tell a group its source never produced from one it never received. A front's reader + /// always reads its routes' copies. + #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] + pub(crate) fn is_received(&self) -> bool { + match &self.inner { + Inner::Plain(cursor) => cursor.state.read().received, + Inner::Resume(..) => true, + } + } + #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] pub(crate) fn latest_sequence(&self) -> Option { match &self.inner { @@ -12430,7 +12631,7 @@ mod test { assert_eq!(read_now(&mut reader, true), None, "the fresh suffix is still owed"); assert_eq!( reader.poll_drop(&waiter), - Poll::Ready((0..3, StreamError::NotFound)), + Poll::Ready((0..3, Hole::drop(StreamError::NotFound))), "the unjudgeable prefix is given up" ); assert!(fill.wanted().is_some(), "the fresh suffix stays wanted"); @@ -12538,7 +12739,10 @@ mod test { "a late live arrival may still answer" ); elapse(&producer, ledger::GRACE); - assert_eq!(reader.poll_drop(&waiter), Poll::Ready((1..3, StreamError::NotFound))); + assert_eq!( + reader.poll_drop(&waiter), + Poll::Ready((1..3, Hole::drop(StreamError::NotFound))) + ); assert!(reader.poll_drop(&waiter).is_pending(), "each drop is reported once"); if bounded { assert_eq!(read_now(&mut reader, ordered), Some(None), "nothing remains owed"); @@ -14264,6 +14468,7 @@ mod test { budget: Duration::ZERO, bounded: false, edge: state.drift_edge(&Servable::ALL), + covers_from: None, }; assert!( state.is_stale(0, &drift.edge, drift.budget), @@ -22938,10 +23143,83 @@ mod test { } assert_eq!( dropped, - vec![(2..4, StreamError::NotFound), (6..8, StreamError::NotFound)] + vec![ + (2..4, Hole::drop(StreamError::NotFound)), + (6..8, Hole::drop(StreamError::NotFound)) + ] ); } + /// A received stale run settles at once only a reader no wider than the budget it was + /// judged under, whose range covers it and everything above it; any other reader treats + /// it as unknown. A skip settles every reader. + #[test] + #[cfg(feature = "async")] + fn a_received_stale_drop_settles_only_the_readers_it_covers() { + let producer = track_producer("test", Info::timed()); + let budget = Duration::from_millis(500); + let reader = |max: Duration, groups: Range| { + producer.subscribe( + Subscription::default() + .with_delay(Delay::default().with_max(max)) + .with_ranges([groups]), + ) + }; + let mut covering = reader(budget, Range::groups(0..)); + let mut narrower = reader(Duration::from_millis(100), Range::groups(0..)); + let mut wider = reader(Duration::from_secs(5), Range::groups(0..)); + let mut capped = reader(budget, Range::groups(0..4)); + let mut above = reader(budget, Range::groups(3..)); + producer.give_up(1..3, Hole::stale(budget)); + producer.skip_groups(3..4); + let waiter = kio::Waiter::noop(); + let mut dropped = |subscriber: &mut Subscriber| { + let mut runs = Vec::new(); + while let Poll::Ready(run) = subscriber.poll_drop(&waiter) { + runs.push(run); + } + runs + }; + let settled = vec![(1..3, Hole::stale(budget)), (3..4, Hole::Skip)]; + assert_eq!(dropped(&mut covering), settled); + assert_eq!(dropped(&mut narrower), settled, "a narrower budget is settled too"); + assert_eq!(dropped(&mut wider), [(3..4, Hole::Skip)], "a wider budget waits"); + assert_eq!( + dropped(&mut capped), + [(3..4, Hole::Skip)], + "a capped range may judge otherwise" + ); + assert_eq!( + dropped(&mut above), + [(3..4, Hole::Skip)], + "the stale run is outside its range" + ); + } + + /// A received stale run superseded by new range terms is unknown again, and a group a + /// widening re-offers wins over its drop. + #[test] + #[cfg(feature = "async")] + fn superseded_stale_runs_are_unknown_again() { + let producer = track_producer("test", Info::timed()); + let budget = Duration::from_millis(500); + let mut subscriber = producer.subscribe( + Subscription::default() + .with_delay(Delay::default().with_max(budget)) + .with_ranges([Range::groups(0..)]), + ); + let waiter = kio::Waiter::noop(); + producer.give_up(1..2, Hole::stale(budget)); + producer.supersede_stale(); + assert!( + subscriber.poll_drop(&waiter).is_pending(), + "no verdict under the new terms" + ); + assert!(producer.state.read().drops.is_empty()); + write_group(&producer, 1); + assert_eq!(drain_groups(&mut subscriber), [1]); + } + /// A report describes a miss, not a delivery: a late group is still handed out once. #[test] #[cfg(feature = "async")] @@ -22952,7 +23230,7 @@ mod test { producer.drop_groups(2..4, StreamError::NotFound); assert_eq!( subscriber.poll_drop(&waiter), - Poll::Ready((2..4, StreamError::NotFound)) + Poll::Ready((2..4, Hole::drop(StreamError::NotFound))) ); assert!(subscriber.poll_drop(&waiter).is_pending()); write_group(&producer, 3); @@ -23121,7 +23399,7 @@ mod test { let waiter = kio::Waiter::noop(); assert!(matches!( relay.poll_drop(&waiter), - Poll::Ready((range, StreamError::App(5))) if range == (0..1) + Poll::Ready((range, Hole::Drop { error: StreamError::App(5), .. })) if range == (0..1) )); } diff --git a/rs/moq-net/src/model/track/fill.rs b/rs/moq-net/src/model/track/fill.rs index 66ee59971cc..ea7dae84701 100644 --- a/rs/moq-net/src/model/track/fill.rs +++ b/rs/moq-net/src/model/track/fill.rs @@ -1324,7 +1324,7 @@ impl TrackState { } } for groups in dropped.iter() { - self.drop_groups(groups.clone(), err.clone()); + self.drop_groups(groups.clone(), super::Hole::drop(err.clone())); } } } diff --git a/rs/moq-net/src/model/track/hole.rs b/rs/moq-net/src/model/track/hole.rs new file mode 100644 index 00000000000..bd010292144 --- /dev/null +++ b/rs/moq-net/src/model/track/hole.rs @@ -0,0 +1,52 @@ +use std::time::Duration; + +use crate::StreamError; + +/// Why a group sequence will not be delivered. +/// +/// lite-07 carries the two kinds in their own messages, SUBSCRIBE_SKIP and SUBSCRIBE_DROP; +/// earlier versions name both in SUBSCRIBE_DROP. +#[derive(Clone, Debug, PartialEq, Eq)] +pub(crate) enum Hole { + /// Never produced, such as a skipped sequence number: final, and it ends no reader. + Skip, + /// Produced but not delivered, ending any reader of it with `error`. + Drop { + error: StreamError, + /// The budget a staleness verdict was judged under, `None` for a drop that does not + /// depend on it. Only a widening past it reverses the verdict. + stale: Option, + }, +} + +impl Hole { + /// A drop with `error` that no budget reverses. + pub fn drop(error: StreamError) -> Self { + Self::Drop { error, stale: None } + } + + /// A group judged stale under `budget`. + pub fn stale(budget: Duration) -> Self { + Self::Drop { + error: StreamError::Old, + stale: Some(budget), + } + } + + /// The error a reader waiting on the sequence ends with: a sequence never produced is + /// simply not found. + pub fn error(&self) -> StreamError { + match self { + Self::Skip => StreamError::NotFound, + Self::Drop { error, .. } => error.clone(), + } + } + + /// The budget a staleness verdict was judged under, if this is one. + pub fn stale_budget(&self) -> Option { + match self { + Self::Drop { stale, .. } => *stale, + Self::Skip => None, + } + } +} diff --git a/rs/moq-net/src/model/track/miss.rs b/rs/moq-net/src/model/track/miss.rs index 035e0c8e4b8..5771b36e009 100644 --- a/rs/moq-net/src/model/track/miss.rs +++ b/rs/moq-net/src/model/track/miss.rs @@ -93,8 +93,9 @@ pub(super) struct Misses { refilled: RangeSet, } -/// A run of misses the reader gives up, for the drop cursor to name. -pub(super) type GivenUp = Vec>; +/// A run of misses the reader gives up, for the drop cursor to name, with the budget it was +/// judged stale under when that is why. +pub(super) type GivenUp = Vec<(ops::Range, Option)>; /// The joins whose fills changed their answer, recorded by each join's own waiter before it /// wakes the reader's task. @@ -872,7 +873,7 @@ impl Misses { self.filled.extend(held.clone()); } for hole in holes(state, run.clone()) { - for hole in self.unclaimed(state, hole) { + for hole in self.unclaimed(state, hole, drift) { let (unfed, awaited) = state.unfed(hole, now); for run in unfed.iter() { asks.insert(run.clone()); @@ -934,7 +935,7 @@ impl Misses { .collect(); for run in stale { asks.remove(run.clone()); - self.give_up(run, given_up); + self.give_up_stale(run, drift.budget, given_up); } asks } @@ -961,8 +962,12 @@ impl Misses { released.insert(run.clone()); } self.release(&released); + // Below an old cache's floor is a boundary, not a drift verdict; above it, a stretch + // went stale under this reader's budget. + let floor = state.cache_floor(drift.budget); for run in stale.iter() { - self.give_up(run.clone(), given_up); + self.give_up(run.start..run.end.min(floor), given_up); + self.give_up_stale(run.start.max(floor)..run.end, drift.budget, given_up); } } @@ -1089,13 +1094,16 @@ impl Misses { } } - /// The part of `hole` nobody has a verdict on yet. - fn unclaimed(&self, state: &TrackState, hole: ops::Range) -> Vec> { + /// The part of `hole` nobody has a verdict on yet: a received stale run that does not + /// settle this reader is no verdict for it (see `Drift::settles`). + fn unclaimed(&self, state: &TrackState, hole: ops::Range, drift: &Drift) -> Vec> { let mut left = RangeSet::new(); left.insert(hole.clone()); // Only what overlaps the hole, so judging every run costs each one the same. - for (run, _) in state.drops.overlapping(&hole) { - left.remove(run.clone()); + for (run, gone) in state.drops.overlapping(&hole) { + if drift.settles(run.start.max(hole.start), gone) { + left.remove(run.clone()); + } } for claimed in [&self.given_up, &self.wanted] { for run in claimed.overlapping(&hole) { @@ -1129,13 +1137,27 @@ impl Misses { } } + /// Judge `groups` again: a widening reversed the stale verdict they were given up with. + pub(super) fn retry(&mut self, groups: ops::Range) { + self.given_up.remove(groups); + } + pub(super) fn give_up(&mut self, groups: ops::Range, given_up: &mut GivenUp) { + self.give_up_judged(groups, None, given_up); + } + + /// Give up `groups` as stale under `budget`, a verdict a wider budget reverses. + fn give_up_stale(&mut self, groups: ops::Range, budget: Duration, given_up: &mut GivenUp) { + self.give_up_judged(groups, Some(budget), given_up); + } + + fn give_up_judged(&mut self, groups: ops::Range, stale: Option, given_up: &mut GivenUp) { if groups.is_empty() { return; } self.given_up.insert(groups.clone()); self.newly_gone.insert(groups.clone()); - given_up.push(groups); + given_up.push((groups, stale)); } } @@ -1438,6 +1460,7 @@ mod tests { budget, bounded: false, edge: state.drift_edge(&Servable::ALL), + covers_from: None, }; let asks = misses.judge( &state, @@ -1580,6 +1603,7 @@ mod tests { budget, bounded: false, edge: state.drift_edge(&Servable::ALL), + covers_from: None, }; let asks = misses.judge( &state, @@ -1630,7 +1654,7 @@ mod tests { &waiter, ); assert!(!asks.contains(&2), "a refused hole is not asked again"); - assert_eq!(given_up, vec![2..3]); + assert_eq!(given_up, vec![(2..3, None)], "a refusal is no stale verdict"); } #[kio::test] async fn unchanged_probes_and_late_arrivals_keep_other_gap_deadlines() { @@ -1653,6 +1677,7 @@ mod tests { budget, bounded: false, edge: state.drift_edge(&Servable::ALL), + covers_from: None, }; assert!( misses @@ -1729,7 +1754,11 @@ mod tests { &waiter, ); assert!(asks.is_empty(), "the successor has aged past this reader's budget too"); - assert_eq!(given_up, vec![3..4]); + assert_eq!( + given_up, + vec![(3..4, Some(drift.budget))], + "judged stale under the reader's budget" + ); assert!(misses.deferred.runs.is_empty()); } @@ -1756,6 +1785,7 @@ mod tests { budget, bounded: false, edge: state.drift_edge(&Servable::ALL), + covers_from: None, }; let full = |span| [span].into_iter().collect(); assert!( @@ -1936,6 +1966,7 @@ mod tests { budget, bounded: false, edge: state.drift_edge(&servable), + covers_from: None, }; let expected = each(&misses, &state, &drift); let found = misses.expired(&state, &drift, &kio::Waiter::noop()); @@ -2049,6 +2080,7 @@ mod tests { budget, bounded: false, edge: state.drift_edge(&Servable::ALL), + covers_from: None, }; let asks = misses.judge( &state, From a1c55f7b5d885a832de42072305bfaf179d561c2 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 13:51:28 -0800 Subject: [PATCH 05/68] fix(net): settle a declared end over given-up groups and spare covered cuts A local producer's end settles once the gap below it is wholly skipped or dropped, so moq-archive drops what it holds nothing for instead of faking an empty group. A cut a fetched copy still covers is no longer reported dropped. Split the JS cache pool into its own quest and rewrite the drop set to the run memory work that remains. Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 3 +- quest/m1/drop-set.md | 409 +++++------------- quest/m1/js-cache-pool.md | 52 +++ quest/m1/lite07-finalize.md | 9 +- quest/m1/read-backfill.md | 5 +- quest/m1/subscribe-ranges/README.md | 9 +- quest/m1/subscribe-ranges/lite.md | 2 +- .../m1/subscribe-ranges/start-bookkeeping.md | 3 +- quest/m1/time-addressing/ietf.md | 7 +- quest/m1/time-addressing/lite07.md | 4 +- quest/m1/time-addressing/model.md | 5 +- quest/m2/README.md | 2 +- quest/m2/ietf-hole-marker.md | 16 +- quest/m2/lite-tail-drop-count.md | 2 +- rs/moq-archive/src/consumer/tests.rs | 3 +- rs/moq-archive/src/consumer/track.rs | 12 +- rs/moq-net/src/model/track.rs | 26 +- 17 files changed, 222 insertions(+), 347 deletions(-) create mode 100644 quest/m1/js-cache-pool.md diff --git a/quest/m1/README.md b/quest/m1/README.md index 2a4ca041d2c..0bcb859088d 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -149,7 +149,8 @@ per-frame CPU growth since #5926. - [Byte-stream renditions form a ladder](/quest/m1/stream-ladder.md) - moq-ffi stream producers link as `lower` and disable unfundable rungs like frame producers - [A job's exact route continues its claim's readers](/quest/m1/announce-takeover.md) - a job's route inherits its claim's instance, replacing the ANNOUNCE_TAKEOVER predecessor design - [Subscribe ranges](/quest/m1/subscribe-ranges/README.md) - a lite-07 SUBSCRIBE asks for past and live ranges with absolute or relative ends, in either order, and replaces FETCH; merged ranges never starve a subscriber; relays fill misses by range, including over moq-transport -- [Drop set](/quest/m1/drop-set.md) - each cursor judges missing groups against its own budget, so a narrow reader drops promptly and a widening re-offers groups already reported dropped on lite-07 +- [JS cache pool](/quest/m1/js-cache-pool.md) - `js/net` bills cached frames and drop-set runs to a per-broadcast byte pool ported from Rust +- [Drop set](/quest/m1/drop-set.md) - the drop set's track and cursor runs are billed to the cache pool, so fragmenting drops costs what the pool holds; eviction folds final runs or ends the cursor with `TooFarBehind` - [Datagram tracks](/quest/m1/datagram-tracks/README.md) - a track is datagrams or streams; datagram tracks have their own request and sequence, no ranges, no cache, and map to object ID 0 on moq-transport - [A step back smaller than the delay keeps the TS export](/quest/m1/ts-small-step-back.md) - audio restarts instead of regressing; video stays within its declared depth - [release carries the fanout-threads bench](/quest/m1/release-fanout-bench.md) - a bench-only port of session_fanout_threads for the pre-cut pairing diff --git a/quest/m1/drop-set.md b/quest/m1/drop-set.md index 98f36f54136..11928b9bf42 100644 --- a/quest/m1/drop-set.md +++ b/quest/m1/drop-set.md @@ -2,18 +2,29 @@ ## Goal -Every cursor judges a missing group against its own budget. Today a dropped -group is final for every reader of a track: a narrow reader can wait -indefinitely for an upstream OLD naming a group the relay never cached, -and a wider reader never gets a group an upstream already reported stale. -The track state instead keeps the dropped and expired runs with their -reason and budget, so a narrow reader gets a prompt drop, a wide reader -waits, and a widening re-offers what its new budget covers. On lite-07 this -restores "growing means I want it": a group reported dropped under a -narrower budget is served after a widening, as a cold relay would. +A hostile upstream fragmenting drops costs what the cache's byte pool would +hold, never more: the drop set's per-track runs and each cursor's +disposition record are billed to the pool at a fixed cost per run, in Rust +and `js/net`, and a cursor's CPU per drop tracks the runs it touches, never +their span or the set. Eviction keeps every promise a reader was made: a +final run folds into the cursor's floor, and losing any other ends the +subscription with `TooFarBehind`. ## Plan +[moq-dev/moq#7670](https://github.com/moq-dev/moq/pull/7670) landed the +behavior: lite-07 SUBSCRIBE_SKIP and SUBSCRIBE_DROP with `Max Age` and +`Terms`, a per-run reason (`Hole` in `rs/moq-net/src/model/track/hole.rs`, +`Hole` in `js/net/src/internal.ts`), relays settling a received stale drop +only for readers it covers (`Drift::settles`, `settles` in +`js/net/src/track.ts`) and forgetting stale runs when they send new terms +(`supersede_stale`), and lite-07 widenings re-offering what was reported +stale (`reoffers_reported`). It kept today's storage: `TrackState::drops`, +`DropCursor` with its `reported` and `reported_stale` sets, `Skipped`, +`Delivered`, `TrackRun::served`, and the JS `drops` signal, `#pending`, +`#reported`, and `#reportedStale`, none billed to anything. This quest is +what remains: bounding them. + Decided 2026-11-03 (maintainer, /quest-unblock): `js/net` bounds run memory by porting Rust's per-broadcast `cache::Pool` to JS. One shared byte account per broadcast bills frames and drop-set runs (a fixed cost per run) under @@ -23,7 +34,9 @@ setting. Reason: the 2026-11-02 audit found the existing 32 MiB limit in `js/net/src/group.ts` bounds only one group's frame tail, and a mock-time probe retained 64, 256, and 1,024 disjoint OLD runs with zero cached groups; a shared pool bounds drop-only tracks the same way in both languages. Not -chosen: a per-track run cap; upstream backpressure. +chosen: a per-track run cap; upstream backpressure. The pool port itself is +[JS cache pool](/quest/m1/js-cache-pool.md), split out on 2026-11-03 since it +lands on its own; this quest bills the runs to it. Decided 2026-11-03 (maintainer, /quest-unblock): range terms carry an explicit update id. Each lite-07 SUBSCRIBE and SUBSCRIBE_UPDATE carries an @@ -55,302 +68,96 @@ single message with a reason enum; renaming only. Decided 2026-10-27 (maintainer): this runs after the range count cost work, which finished in [moq-dev/moq#7542](https://github.com/moq-dev/moq/pull/7542), -since both edit `given_up` in `rs/moq-net/src/model/track/miss.rs`, and before -[Rust groups by start](/quest/m1/time-addressing/model.md), which requires it -because runs keyed by group sequence convert mechanically to starts. - -Decided by the maintainer on 2026-10-19, splitting this from the widened -max age re-offer -([moq-dev/moq#6717](https://github.com/moq-dev/moq/pull/6717)), which keeps -drops final for now. That PR re-offers cached groups a cursor skipped as -stale, in the subscription's order (newest first unless ascending; `js/net` -lowest first, since its buffer is ascending), through the -[moq-dev/moq#5929](https://github.com/moq-dev/moq/pull/5929) owed path. Its -skipped record (`Skipped` in `rs/moq-net/src/model/skipped.rs`, `#skipped` -in `js/net/src/track.ts`) keeps a run only while the cache holds one of its -groups. On 2026-10-25 the maintainer deferred the full pool-billed -disposition record to this quest. - -- **One run-length set per track.** The track state keeps the dropped and - expired group runs within the subscription range. Each run carries its - reason (stale, cut, never produced, or evicted or expired where no copy - remains: received as such from the upstream, or at an origin with no - upstream) and, for a stale run, the - budget it was judged under. A group that arrives removes its sequence; a - drop never removes a group that already arrived. A relay's own eviction - is never a run: the group is unknown again, and any cursor that wants it, - new, widened, or already owed it, fills it from upstream by range, the - existing miss fill. That needs no new state, since the cursor's record - already holds the span as unknown, a run at a time. -- **Each cursor judges for itself, a run at a time.** A cursor walks the - set's runs in its range, never their sequences, and settles each whole run - in O(1): - - A run whose reason does not depend on the budget (a cut, a group never - produced, an upstream or origin eviction or expiry) is dropped for every reader - as one range, as today. - - A received stale run is dropped promptly as one range for a reader - whose budget is no wider than the one it was judged under and whose - range covers this relay's aggregate range for the track, which the - upstream judged under. A verdict depends on the range as well as the - budget, since `drifted` caps the edge to the reader's range - (`Edge::cap`), so a narrower reader treats the run as unknown and fills - it by range, where the upstream judges it under that narrower range. - The cost is an extra range request, only for narrower readers, when a - stale drop arrives. This qualification is incomplete when an update - changes the aggregate range; settle the range-terms item under `Waiting`. - - A missing run this relay judges itself is dropped for a reader under - whose own budget and range `TrackState::drifted` convicts it. `drifted` - judges a missing sequence by its successor here, and every sequence in - a missing run shares one successor, so one call settles the run. - - A wider reader skips the run and waits for a group, a re-offer, or its - own verdict, each of which splits the run at most once. - - An unknown span (a gap between runs, or an evicted run) is judged the - same way, as one unit bounded by its neighbours: by one `drifted` call - against its shared successor, or by one range miss fill upstream. - - A hostile upstream's huge run costs one Charge and O(1) per cursor, and - splitting it costs per-run Charges the pool bounds, so CPU tracks pool - bytes, never span. - - This fixes the gap Codex found on - [moq-dev/moq#5986](https://github.com/moq-dev/moq/pull/5986): today - `TrackRun::poll_forward_drop` emits only recorded drops, cuts, or stale - cached groups. A sequence the relay never cached is not judged until the - track finishes. -- **Each cursor keeps a disposition record,** next to the per-track set: a - group that arrives leaves the shared set, so only the cursor's record - remembers that it reported cached G stale. The record is one run-length - map per cursor, each run marked delivered, skipped as stale, or reported - in SUBSCRIBE_DROP, with its reason and, for a stale run, the budget and - range it was judged under. It replaces the skipped record, `Delivered` - (`rs/moq-net/src/model/delivered.rs`), whose floor folds stale gaps in - with delivered groups, `TrackRun::served` in - `rs/moq-net/src/lite/publisher.rs`, and the JS wire cursor's `#reported`. - Delivery is recorded, not inferred from the cursor position, since an - arrival-order or descending cursor delivers out of sequence order. On - lite-07 a reported stale run is no longer final: like a skipped one, it - is re-offered when a widening covers it. -- **The record is bounded by the pool, never by guessing.** Each run is - billed to the cache's `Pool` through a `Charge` at a fixed per-run cost - and is settled a run at a time, so a cursor's state and CPU track its - runs, never their span. The floor only rises over a final prefix: the - runs from the floor up to the lowest run that is not final. An unknown - hole is never final: a descending cursor can deliver H while a lower G - is still unknown. When the pool evicts a run inside the final prefix, it - folds into the floor; any other run cannot go without guessing, so the - subscription ends with `TooFarBehind` and the subscriber re-subscribes. -- **A widening re-offers the dropped runs now inside its budget,** through - the same re-offer path as skipped runs, in the subscription's order. - Upstream, the relay's own widening asks for them again. -- **Only a widening reverses a stale verdict,** local or received alike: no - run tracks the group that convicted it, and an abort or eviction of that - group leaves the run standing. So warm/cold equivalence after an edge - loss is not promised: when the newer group that made G stale is aborted - or evicted, a cold cursor may serve G while a warm one stays past it. - Watching for it would need both groups a verdict depends on (`drifted` - uses G's successor and a possibly later presentation edge), and a run - skipped at the max age bound would no longer be final. A received run is - also reversed by the upstream re-offering its group after a widening, - since a later group wins over its drop. Cuts and never-produced runs are - unaffected. -- **A stale drop names its budget.** A SUBSCRIBE_UPDATE has no - acknowledgement, so after a widening from B1 to B2, an in-flight B1 drop - and the B2 re-drop look identical, and no guess settles a B2 reader. On - lite-07 SUBSCRIBE_DROP carries `Max Age`: the budget a staleness verdict - was judged under, encoded like `Publisher Max Age` (milliseconds plus one), - with `0` for a drop that does not depend on the budget: a cut, a group - never produced, or a group the sender evicted or expired with no copy it - could fetch (an origin, or a relay forwarding its upstream's eviction - drop), none of which depends on the reader's budget. A relay never sends - an eviction drop for its own eviction; it fills the group by range - instead. A widening never reverses a `0` drop. The relay records each run with that - exact budget. A B1 - drop then settles only readers no wider than B1, and the B2 re-drop - settles a B2 reader. The cost is one varint per drop, a few bytes on an - event that is already per group, which is worth an unambiguous term. - Rejected: an implicit generation counted by both sides per - SUBSCRIBE_UPDATE. It needs a generation-to-budget history at the relay - and agreement on every update counted, while the budget itself is - self-describing, order-free, and unaffected by updates that change - nothing. -- **lite-07 only for re-offering reported groups.** Released lite-05 and - lite-06 peers treat a flushed SUBSCRIBE_DROP as final, so a drop received - over those versions is final for every reader, and a lite-07 hop never - sends a group onto a lite-05 or lite-06 hop after reporting it. On - lite-07, a widening obliges the publisher to serve the newly covered groups - it holds and never delivered on this subscription, including ones it - reported dropped, and to drop again, under the new terms, the ones it - cannot serve. SUBSCRIBE_UPDATE stays incremental and never resets, since a - reset races in-flight, out-of-order data. A shrink sends no drops. -- **Draft wording.** `drafts/draft-lcurley-moq-lite.md` states, for lite-07's - restored SUBSCRIBE_DROP, that a drop holds only under the terms in effect - when it was sent, that a widening re-offers held groups and re-drops the - rest, and that a later group, when sent, wins over its drop. This quest - writes it, including the `Max Age` field above, which is free to add - because lite-07 is unpublished. - -Code it replaces, in Rust and `js/net` alike: - -- `TrackState::drop_groups`, the `drops` range set and `DropCursor` in - `rs/moq-net/src/model/track.rs`, which record a whole range without its - reason or budget, even over groups already cached. -- In `js/net/src/track.ts`: `#drop`, `#undrop`, the `drops` signal, and - the wire cursor's `#pending` and `#dropsSeen`, whose `#tryRecvGroup` - closes any group already reported. - -Untrusted input: a hostile upstream can fragment the set by alternating -drops and arrivals, and an open live range is no bound, since its groups are -evicted while their runs would stay. Bill each run to the cache's existing -byte accounting like a group: the Rust `Pool` through a `Charge` at a fixed -per-run cost. Under pressure the pool evicts the oldest runs as it evicts -groups, and a run older than the cache's retention window is forgotten, so -the set never outgrows what the pool would hold, with no new constant. An -evicted run's sequences become unknown again. `js/net` has no byte pool; -its run memory policy is open under `Waiting`. Neither a cached-sequence -count nor its existing per-group frame-tail limit bounds drop-only runs. -Expiry and eviction must preserve the same fold-or-`TooFarBehind` behavior -as Rust. - -Only a final run folds into a cursor's floor: a delivered run, a run -skipped at the track's max age bound (no widening reaches past it), or a -budget-independent drop (a cut, a group never produced, an upstream or -origin eviction or expiry, or any drop reported over lite-05 or lite-06). A stale run reported on lite-07 within reach of the -max age bound is budget-qualified, so it is treated like an unknown hole: -the floor never rises over it, and if the pool must evict it or a run above -it, the cursor fails with `TooFarBehind`. Folding it would forgo the -re-offer this plan promises after a widening. A later group wins over its -drop only while the cursor's record still holds that drop: a drop folded -into the floor under pool pressure is final for that cursor, so its group -arriving late is not delivered there. - -A cursor judges an unknown missing sequence (never a stale run its record -still holds as reported, which only a widening settles) as it would a -group never received: locally against its own budget (`TrackState::drifted`), -and otherwise as a cache miss, which the relay fills upstream by range -([Subscribe ranges](/quest/m1/subscribe-ranges/README.md)), so no reader -waits indefinitely on forgotten state. Extend `track_subscriber_churn` in -`rs/moq-net/benches/track.rs` with a fragmented set up to pool capacity, -swept over subscriber count, shared runs, runs per cursor record, and run -span, so enumeration cost tracks the touched runs, never the set, the -record, or a run's span. - -Test on mock time through a relay: -- On a live track, a reader no wider than the upstream budget gets a prompt - drop for an upstream OLD naming a group the relay never cached. -- On lite-07, a widened reader gets that group re-offered. A drop already - flushed before the widening SUBSCRIBE_UPDATE was received, so the reader - then gets the re-offer and no further final drop; a drop still in flight - does not settle the widened reader. Check what each reader receives, not - only tail accounting. -- On lite-07, after a widening from B1 to B2 for a group the upstream no - longer holds, an in-flight drop tagged B1 settles only the B1 reader, and - the re-drop tagged B2 settles the B2 reader. -- A group arriving after its drop, while the cursor's record still holds - the drop, is served, and a drop arriving after its group leaves it - served. -- Through a lite-06 hop, a reported group stays dropped for every reader, - with no session error. -- A shrink sends no drops. -- Rust and `js/net` each encode and decode an origin's eviction drop and - retention-expiry drop with `Max Age` `0`, across languages; a relay - forwards them unchanged, and a widening re-offers neither. -- A group a relay evicted while the upstream still holds it is never - reported dropped: a new reader, a widened reader, and a reader already - owed it each fill it by range and receive it. -- A reader whose range is narrower than the relay's aggregate range is not - settled by a received stale drop: it fills G by range and gets G or a - drop judged under its own range, while a reader covering the aggregate - range is settled at once. Add the in-flight broad-range drop after a - narrowing from the range-terms item under `Waiting`; its qualification - remains open. -- On lite-07, a cursor that delivered F, reported its neighbour G stale - while G stayed cached, and delivered H, gets only G after a widening that - covers all three: G is re-offered and F and H are never resent. -- Under pool pressure, a cursor's delivered runs and budget-independent - drops fold into the floor, while evicting a lite-07 stale-reported run - within reach, or a run above it, ends the cursor with `TooFarBehind` - instead of losing the re-offer. Through a lite-06 hop the reported run - folds. -- A flood of fragmented runs stays within the pool in Rust, evicting the - oldest shared runs first, and a sequence whose shared run was evicted is - judged locally or filled by range, never waited on indefinitely. In - `js/net`, include a drop-only flood with no cached groups and check the - bound chosen under `Waiting`, including eviction or expiry of a - nonfinal cursor run. That memory-policy decision blocks this assertion. -- A single run spanning billions of sequences is settled by each cursor in - one step, whether dropped, skipped by a wider reader, or evicted and - judged again. -- Per-reader misses are bounded the same way. [Dynamic range - requests](https://github.com/moq-dev/moq/pull/6171) adds a third unbounded - record: each reader's given-up misses, one per gap, which a reader whose - floor never moves cannot forget. Fold them into the cursor's runs, billed - to the pool (from +and before [Rust groups by start](/quest/m1/time-addressing/model.md), which +requires it because runs keyed by group sequence convert mechanically to +starts. + +- **One disposition record per cursor,** a run-length map whose runs are + delivered, skipped as stale, or reported dropped or skipped, with the + reason and, for a stale run, the budget it was judged under. It replaces + `Skipped` (`rs/moq-net/src/model/skipped.rs`), `Delivered` + (`rs/moq-net/src/model/delivered.rs`, whose floor folds stale gaps in with + delivered groups), `DropCursor`'s `reported` and `reported_stale`, + `TrackRun::served` in `rs/moq-net/src/lite/publisher.rs`, and the JS + wire cursor's `#reported` and `#reportedStale`, `#skipped`, and the + publisher's `served`. Delivery is recorded, not inferred from the cursor + position, since an arrival-order or descending cursor delivers out of + sequence order. +- **Billed to the pool, never guessed.** Each track run and each cursor run + is billed through a `Charge` at a fixed per-run cost (in JS, through the + [JS cache pool](/quest/m1/js-cache-pool.md)). Under pressure the pool + evicts the oldest track runs as it evicts groups, and a run older than the + cache's retention window is forgotten. An evicted track run's sequences are + unknown again: a reader judges them by `drifted` or fills them by range. + A relay's own eviction is never a run. +- **The floor rises only over a final prefix:** the runs from the floor up to + the lowest run that is not final. A final run is a delivered one, one + skipped at the track's max age bound, a skip, a budget-independent drop, + or any drop reported over lite-05 or lite-06. A lite-07 stale report within + reach of the max age bound, or an unknown hole, is not: when the pool must + evict it or a run above it, the cursor fails with `TooFarBehind` rather + than forgo the re-offer. A drop folded into the floor is final for that + cursor, so its group arriving late is not delivered there. +- **Per-reader misses** fold into the cursor's runs too: the given-up misses + `rs/moq-net/src/model/track/miss.rs` keeps per gap (from [moq-dev/moq#6171](https://github.com/moq-dev/moq/pull/6171), decided 2026-10-23). +- **CPU per drop.** Every SUBSCRIBE_DROP bumps `drops_revision`, and each + reader's `DropCursor::next` rescans the overlapping runs, so N fragmented + runs and R readers cost O(R*N) per 6-byte drop; JS `insertDrop` + (`js/net/src/internal.ts`) copies the whole run array once for the track + and once per sink. Keep the per-cursor walk incremental in both languages. + Extend `track_subscriber_churn` in `rs/moq-net/benches/track.rs` with a + fragmented set up to pool capacity, swept over subscriber count, shared + runs, runs per cursor record, and run span. + +Still to cover, beyond the memory work: + +- **Idle START unreport test.** [moq-dev/moq#6717](https://github.com/moq-dev/moq/pull/6717) + made the JS lite publisher unreport stale skips below an idle + SUBSCRIBE_START, without a test: the case needs an update between START + and the first group. Cover it deterministically through the track hooks. +- **Cancelled fetches hold the end.** Found 2026-10-28 by + [moq-dev/moq#7149](https://github.com/moq-dev/moq/pull/7149): a cancelled + fetch below a track's end holds the end open until the cache expiry scan + removes it. Settle it with a run reason. +- **Untimed fills below a resolved start.** Noted 2026-10-31 from + [moq-dev/moq#7235](https://github.com/moq-dev/moq/pull/7235): a lite-07 + copy with no media time asks a fill below the start even for a reader whose + budget the publisher already exceeded. If the extra FETCH shows up in relay + traffic, bound it with a run below the start carrying its budget. +- **A fill's stale verdict is final for everyone.** `TrackState::drop_unfed` + (`rs/moq-net/src/model/track/fill.rs`) records what a fill's source gave up + as a budget-independent drop, including OLD judged under that fill's own + terms. Record it as a stale run, so it settles only the readers it covers. + +Tests, on mock time through a relay, in Rust and mirrored in `js/net`: + +- After a widening from B1 to B2 for a group the upstream no longer holds, an + in-flight drop tagged B1 settles only the B1 reader, and the re-drop tagged + B2 settles the B2 reader, over the wire. +- A group a relay evicted while the upstream still holds it is never reported + dropped: a new, a widened, and an owed reader each fill it by range. +- Rust and `js/net` encode and decode an origin's eviction and retention + expiry drop with `Max Age` 0 across languages; a relay forwards them + unchanged, and a widening re-offers neither. +- Through a lite-06 hop, a reported group stays dropped for every reader, + with no session error; a shrink sends no drops. +- Under pool pressure, delivered runs and budget-independent drops fold into + the floor, while evicting a lite-07 stale report within reach, or a run + above it, ends the cursor with `TooFarBehind`. +- A flood of fragmented runs, including a JS drop-only flood with no cached + groups, stays within the pool, evicting the oldest shared runs first, and + a sequence whose run was evicted is judged locally or filled by range. +- A single run spanning billions of sequences is settled by each cursor in + one step, whether dropped, skipped by a wider reader, or evicted and judged + again. -- A group the publisher cut while a fetched copy still serves its end is - delivered and never also reported dropped. Today `DropCursor::cut` - (`rs/moq-net/src/model/track.rs`) reports any visible slot's - `cut_error()`, so the arrival walk can hand out the fetched copy and report - the group dropped too; `Slot::cut()` already returns nothing while a copy - serves the end. Found by - [moq-dev/moq#7053](https://github.com/moq-dev/moq/pull/7053) (added - 2026-10-28). - -Mirror each case in `js/net`. - -Public API: none. Wire: lite-07 SUBSCRIBE_DROP (unpublished) gains `Max Age` -and the widening and ordering rules above; published versions are -unchanged. -- A local producer's declared end must settle over dropped groups, as the - doc on `TrackState::drop_groups` already claims (added 2026-10-25): let - `reaches(fin)` count a live edge whose gap `edge..fin` is wholly dropped, - then delete moq-archive's fake empty group at `fin - 1` - (`rs/moq-archive/src/consumer/track.rs`). - -### CPU per drop (2026-10-24 pre-cut audit) - -Besides memory, run fragmentation costs CPU today, new on main: every -SUBSCRIBE_DROP bumps `drops_revision`, and each reader's `DropCursor::next` -rescans the overlapping runs (`rs/moq-net/src/model/track.rs` about lines -5802-5870), so N fragmented runs and R downstream readers cost O(R*N) per -6-byte DROP. JS `insertDrop` (`js/net/src/internal.ts:136`) copies the whole -run array once for the track and once per sink (`track.ts:914`). The -replacement keeps the per-cursor walk incremental in both languages, with a -benchmark swept over runs and readers. - -### Idle START unreport test - -[moq-dev/moq#6717](https://github.com/moq-dev/moq/pull/6717) made the JS lite -publisher unreport stale skips below an idle SUBSCRIBE_START, so a later -widening can re-offer them as Rust does, but landed it without a test: the -case needs an update to arrive between START and the first group. Cover it -deterministically, through the track hooks rather than wire timing, when the -drop runs are reworked here. - -### A covered cut is still reported - -`DropCursor::cut` reports a group as cut even when a fetched copy covers it, -noted in [moq-dev/moq#7053](https://github.com/moq-dev/moq/pull/7053) and again -by [moq-dev/moq#7149](https://github.com/moq-dev/moq/pull/7149), which made a -fetched copy settle such a group. Decided 2026-10-29 (maintainer): settle it -here, where a cut run gets a reason and a covering copy can clear it. - -### Cancelled fetches hold the end - -Found 2026-10-28 by [moq-dev/moq#7149](https://github.com/moq-dev/moq/pull/7149): -a cancelled fetch below a track's end still holds the end open until the cache -expiry scan removes it, so how fast a subscription ends depends on expiry -timing. Settle it here, where cancelled and expired runs get a reason. +Public API: none. Wire: none. -### Untimed fills below a resolved start +## Required -Noted 2026-10-31 from [moq-dev/moq#7235](https://github.com/moq-dev/moq/pull/7235), -which made everything below a resolved SUBSCRIBE_START fillable from upstream: -a lite-07 copy with no media time asks a fill below the start even for a reader -whose budget the publisher already exceeded, and that fill ends empty. If the -extra FETCH shows up in relay traffic, bound it here, where a run below the -start can carry the budget it was resolved under. +- [JS cache pool](/quest/m1/js-cache-pool.md) - the byte account the JS runs bill ## Related diff --git a/quest/m1/js-cache-pool.md b/quest/m1/js-cache-pool.md new file mode 100644 index 00000000000..b9bab0d69c1 --- /dev/null +++ b/quest/m1/js-cache-pool.md @@ -0,0 +1,52 @@ +# [L] JS cache pool + +## Goal + +`js/net` bounds what a broadcast caches in bytes, as Rust does: a port of +Rust's per-broadcast `cache::Pool` (`rs/moq-net/src/model/cache.rs`) bills +every cached group's frames and fixed bookkeeping, and the drop set's track +and cursor runs at a fixed cost per run, and evicts under it the way Rust +does. Rust and JS mirror each other under Rust's default budget, with no new +setting. + +## Plan + +Decided 2026-11-03 (maintainer, /quest-unblock, recorded in +[Drop set](/quest/m1/drop-set.md)): port Rust's per-broadcast `cache::Pool` +to JS. Shared byte accounting per broadcast bills frames and drop-set runs (a +fixed cost per run), under Rust's default budget, with eviction and the +fold-into-floor or `TooFarBehind` behavior the drop set plans. Not chosen: a +per-track run cap; upstream backpressure. Split out of the drop set on +2026-11-03, since the pool lands on its own: its first consumer is the group +cache, and the drop set's run billing follows on top of it. + +What JS has today: `js/net/src/group.ts` caps each group's frame tail at +`TAIL_BYTES` (32 MiB) and evicts that group's oldest frames past it, while +the track cache (`#prune` in `js/net/src/track.ts`) is purely time-based on +`CACHE_EXPIRY` (30 s). Nothing bills bytes across groups or tracks, and +`TrackState.drops` and each cursor's runs grow without bound under a peer +that fragments them. + +Port, mirroring Rust's names and shape: + +- A pool per broadcast, defaulting as `broadcast::Info::default` does in + Rust: no byte capacity and `DEFAULT_EXPIRY` (30 s) idle expiry. +- A per-track account and a per-group charge billing payload plus a fixed + per-group overhead, and a fixed per-run cost for drop-set runs. +- Write-time eviction debt: a track over capacity pays by evicting its own + oldest groups with EVICTED, the latest live group protected, and the + access-time mean ordering eviction across tracks. +- The idle-expiry sweep on the pool's clock, replacing `#prune`'s own timer. + +Test on mock time: a capacity bounds cached groups across tracks of one +broadcast, evicting the oldest first and sparing a fresher track; idle groups +expire; a drop-only flood with no cached groups stays within the pool once +[Drop set](/quest/m1/drop-set.md) bills its runs; the per-group frame-tail +limit still applies. + +Public API: none beyond what Rust already exposes, mirrored if a JS consumer +needs it. Wire: none. + +## Related + +- [Drop set](/quest/m1/drop-set.md) - bills its track and cursor runs to this pool diff --git a/quest/m1/lite07-finalize.md b/quest/m1/lite07-finalize.md index ed639960eb8..9276f8f48fc 100644 --- a/quest/m1/lite07-finalize.md +++ b/quest/m1/lite07-finalize.md @@ -62,8 +62,12 @@ The cut: treat a longer SETUP as a protocol violation" stays. The lite-07 changelog names that rejection, not only the cap. -Decided 2026-10-20 (maintainer): [Drop set](/quest/m1/drop-set.md) joins -Required, because it adds `Max Age` to lite-07's SUBSCRIBE_DROP. Nothing +Decided 2026-10-20 (maintainer): the drop set joined Required, because it +adds `Max Age` to lite-07's SUBSCRIBE_DROP. That wire work, with +SUBSCRIBE_SKIP and range `Terms`, landed in +[moq-dev/moq#7670](https://github.com/moq-dev/moq/pull/7670); what remains of +the drop set is memory accounting with no wire change, so it no longer gates +the cut. Nothing else from the auth line gates the cut: its lite-07 wire work, the SETUP Token parameter, has landed, and the rest is API work that does not block the rename. @@ -107,7 +111,6 @@ timestamp bound in [#6168](https://github.com/moq-dev/moq/pull/6168). ## Required - [Routes and announces](/quest/m1/cluster-routing/routes.md) - lite-07 carries the route layer: ROUTE per origin node and path-less ANNOUNCE, with the hop list gone -- [Drop set](/quest/m1/drop-set.md) - lite-07's SUBSCRIBE_DROP carries `Max Age`, so a widening re-offers groups reported dropped - [Lite-07 ranges](/quest/m1/subscribe-ranges/lite.md) - a relay fills a lite-07 upstream through the subscription - [JS ranges](/quest/m1/subscribe-ranges/js.md) - JS speaks lite-07 ranges - [Remove FETCH from lite-07](/quest/m1/subscribe-ranges/lite07-fetch.md) - lite FETCH is gone from lite-07, in Rust and JS diff --git a/quest/m1/read-backfill.md b/quest/m1/read-backfill.md index 4611998630d..ea468967a09 100644 --- a/quest/m1/read-backfill.md +++ b/quest/m1/read-backfill.md @@ -30,7 +30,8 @@ delivers a group that arrives below the cursor as a late backfill; Rust ([Subscribe ranges](/quest/m1/subscribe-ranges/README.md)): given for an absolute start, and for a relative start the start the publisher resolved for this request. Every sequence in range arrives or is named by - SUBSCRIBE_DROP, so the read never waits on a group that won't come. + SUBSCRIBE_SKIP or SUBSCRIBE_DROP, so the read never waits on a group that + won't come. - Don't infer a relative start from the reported largest location: a frameless newest group counts when a relative end resolves but not in `largest()`, sparse moq-transport groups break the arithmetic, and a later @@ -42,7 +43,7 @@ delivers a group that arrives below the cursor as a late backfill; Rust cursor there. - The cursor is range-aware, not just floored: with disjoint ranges it skips straight across an unrequested gap (group 1, then 10 onward) instead of - waiting for the latency budget, since no SUBSCRIBE_DROP is owed there. + waiting for the latency budget, since no SUBSCRIBE_SKIP or SUBSCRIBE_DROP is owed there. - Once both land: set the ordered read's floor from those bounds in Rust and JS, then make JS discard groups below the cursor. Add a test mirroring Rust's: a reordered group below the floor is never yielded and no extra diff --git a/quest/m1/subscribe-ranges/README.md b/quest/m1/subscribe-ranges/README.md index b8bd2fcc77d..80815ac6347 100644 --- a/quest/m1/subscribe-ranges/README.md +++ b/quest/m1/subscribe-ranges/README.md @@ -6,7 +6,8 @@ A moq-lite-07 SUBSCRIBE replaces FETCH. One subscription asks for any number of group ranges, past or live, bounded by group or by time, in ascending or descending order, and the publisher serves their union from one cursor without sending a group twice. -Every sequence in range either arrives or is named by a SUBSCRIBE_DROP, so +Every sequence in range either arrives or is named by a SUBSCRIBE_SKIP or +SUBSCRIBE_DROP, so nothing blocks on a cache miss. Relays fill misses upstream with ranges, not one request per group, including over moq-transport where group numbers may be sparse. Lite FETCH is removed from lite-07. @@ -28,9 +29,9 @@ relay bounds a sparse FETCH only if the model can request ranges: not accidentally give it the live SUBSCRIBE zero-delay default. SUBSCRIBE_UPDATE replaces the list. Lite FETCH is removed from lite-07; published versions keep it, answered through the same model. -- **Holes.** SUBSCRIBE_DROP (restored in lite-07) names every sequence in the - requested ranges the publisher won't deliver: never existed, expired, or - refused upstream. On lite-01..06 datagram groups are always dropped in a past range (#4982); on lite-07 ranges never cover datagrams ([Datagram tracks](/quest/m1/datagram-tracks/README.md)). The +- **Holes.** Lite-07 names every sequence in the requested ranges the + publisher won't deliver: SUBSCRIBE_SKIP one that never existed, and + SUBSCRIBE_DROP (restored in lite-07) one expired or refused upstream. On lite-01..06 datagram groups are always dropped in a past range (#4982); on lite-07 ranges never cover datagrams ([Datagram tracks](/quest/m1/datagram-tracks/README.md)). The publisher delivers what it has and fills the rest as upstream answers. - **Model.** `Subscription` gains `ranges` beside its `order`, mirroring the wire; `request_groups` as a separate API was rejected. A publisher's diff --git a/quest/m1/subscribe-ranges/lite.md b/quest/m1/subscribe-ranges/lite.md index fcce6cb6211..69a30dec567 100644 --- a/quest/m1/subscribe-ranges/lite.md +++ b/quest/m1/subscribe-ranges/lite.md @@ -143,7 +143,7 @@ were using QUIC FIN to signal that but that prevents updates in the future, so we should have a signal instead". Settled the same day: - No SUBSCRIBE_COVERED message: a subscriber judges a bounded range complete - from the groups it received, SUBSCRIBE_DROP, the Range Starts in + from the groups it received, SUBSCRIBE_SKIP and SUBSCRIBE_DROP, the Range Starts in SUBSCRIBE_OK, and SUBSCRIBE_FINISH. Drops and groups name absolute sequences, so the subscriber judges against its latest ranges and an update in flight changes nothing it has counted; the FIN's race with an unread diff --git a/quest/m1/subscribe-ranges/start-bookkeeping.md b/quest/m1/subscribe-ranges/start-bookkeeping.md index 1d7f173b5ce..4b8249aa316 100644 --- a/quest/m1/subscribe-ranges/start-bookkeeping.md +++ b/quest/m1/subscribe-ranges/start-bookkeeping.md @@ -3,7 +3,8 @@ ## Goal What each lite cursor is owed and what was dropped lives in the model, and -lite `TrackRun` only translates it into SUBSCRIBE_START, SUBSCRIBE_DROP, and +lite `TrackRun` only translates it into SUBSCRIBE_START, SUBSCRIBE_SKIP, +SUBSCRIBE_DROP, and SUBSCRIBE_END per version. `skipped`, `served`, `first`, and the pre-06 branches are deleted rather than layering lite-07 ranges on top, and on lite-07 a subscriber's end is "owed range covered and no stream active". diff --git a/quest/m1/time-addressing/ietf.md b/quest/m1/time-addressing/ietf.md index 572cfec285e..0caec5857cb 100644 --- a/quest/m1/time-addressing/ietf.md +++ b/quest/m1/time-addressing/ietf.md @@ -26,9 +26,9 @@ TIMESCALE (0x08): moq-lite viewer therefore arrives without a frame PTS, which is accepted. Today's TIMESTAMP code (`ietf/group.rs`, `ietf/object.rs`, `ietf/max_age.rs`) is deleted or reduced to that pass-through. -- **Prior Group ID Gap stays** as a best-effort SUBSCRIBE_DROP signal: +- **Prior Group ID Gap stays** as a best-effort SUBSCRIBE_SKIP signal: egress sends the gap to the previous key, and ingest reads it as the - skipped IDs being dropped. A peer that sends none still works. + skipped IDs never produced. A peer that sends none still works. - **GROUP_END maps to END_OF_GROUP.** The supported drafts refuse properties on a status object (`check_status` in `rs/moq-net/src/ietf/object.rs`), so END_OF_GROUP stays bare and the end time is lost on this wire. @@ -39,7 +39,8 @@ Decision prompts: - Timestamp property: ✅ stop, with LOC proxied only between moq-transport peers; keep reading but stop sending; keep the 2026-10-25 plan. - Prior Group ID Gap: drop it; ✅ keep it as a best-effort SUBSCRIBE_DROP - signal. + signal (a SUBSCRIBE_SKIP signal since lite-07 split never-produced holes + out, 2026-11-03). Test both directions over drafts 18, 20, and 22: IDs round-trip exactly through a relay, a gap reaches the subscriber as a drop, and LOC properties diff --git a/quest/m1/time-addressing/lite07.md b/quest/m1/time-addressing/lite07.md index 42861d0cfe2..f20618c9f53 100644 --- a/quest/m1/time-addressing/lite07.md +++ b/quest/m1/time-addressing/lite07.md @@ -20,7 +20,7 @@ Decided 2026-10-25 (maintainer), from the spike: instead of resetting it, which changes the draft's reset rule for those groups (around line 760). - SUBSCRIBE and SUBSCRIBE_UPDATE bounds, SUBSCRIBE_OK and its largest, - SUBSCRIBE_END, and SUBSCRIBE_DROP address by start. Datagrams keep their own sequence (maintainer, 2026-11-01). Drops name + SUBSCRIBE_END, SUBSCRIBE_SKIP, and SUBSCRIBE_DROP address by start. Datagrams keep their own sequence (maintainer, 2026-11-01). Drops name time spans with no group to deliver, not sequence ranges. `Relative(n)` stays a group count. - Positions stay (start, frame index), preserving transport frame offsets. @@ -42,7 +42,7 @@ and `just test interop --all`. Public API: none beyond [the model](/quest/m1/time-addressing/model.md). Wire: lite-07-wip GROUP, GROUP_END, SUBSCRIBE, SUBSCRIBE_UPDATE, -SUBSCRIBE_OK, SUBSCRIBE_END, and SUBSCRIBE_DROP change. +SUBSCRIBE_OK, SUBSCRIBE_END, SUBSCRIBE_SKIP, and SUBSCRIBE_DROP change. ## Required diff --git a/quest/m1/time-addressing/model.md b/quest/m1/time-addressing/model.md index cfec36cb232..cdad5d1bfe9 100644 --- a/quest/m1/time-addressing/model.md +++ b/quest/m1/time-addressing/model.md @@ -69,7 +69,7 @@ it: - The cache key, the duplicate check, the hand-out-once record (`delivered.rs`), and slot stamps. - Successor, reach, expiry, the arrival ledger, tail accounting, fills and - misses, and SUBSCRIBE_DROP accounting. Successor and reach already scan the + misses, and SUBSCRIBE_SKIP and SUBSCRIBE_DROP accounting. Successor and reach already scan the cache by range, so they tolerate sparse keys; `delivered.rs` runs, fills, misses, and `subscription/window.rs` treat `sequence + 1` as the next group and must follow previous-start links instead. @@ -78,7 +78,8 @@ it: refresh. - Not datagrams: they become their own track kind with their own sequence ([Datagram tracks](/quest/m1/datagram-tracks/README.md)). - The legacy codecs: lite GROUP, SUBSCRIBE and SUBSCRIBE_UPDATE bounds, - SUBSCRIBE_START and its largest, SUBSCRIBE_END, SUBSCRIBE_DROP, FETCH, and + SUBSCRIBE_START and its largest, SUBSCRIBE_END, SUBSCRIBE_SKIP, + SUBSCRIBE_DROP, FETCH, and every IETF Location, Filter, FETCH, Largest, and NEW_GROUP_REQUEST. ### Timestamps are required and legacy IDs pass through diff --git a/quest/m2/README.md b/quest/m2/README.md index 4c04ff593e6..6abc1d22279 100644 --- a/quest/m2/README.md +++ b/quest/m2/README.md @@ -215,7 +215,7 @@ lite-07; bitrate enforcement is not part of this release. - [moq-bench pins the instances it measures](/quest/m2/bench-epoch-pin.md) - pooled requests pin their announced epoch, so a takeover is not counted as the same broadcast - [The memory bench finishes at 1 MiB frames](/quest/m2/memory-bench-hang.md) - the allocation mode's park at `size=1048576` is root-caused and fixed - [Audio's Latest start reaches back](/quest/m2/audio-latest-reach-back.md) - moq_audio's start covers min plus warmup like other consumers -- [moq-lite tails count drops](/quest/m2/lite-tail-drop-count.md) - SUBSCRIBE_DROP accounting replaces the session-tail grace from lite-05 +- [moq-lite tails count drops](/quest/m2/lite-tail-drop-count.md) - SUBSCRIBE_SKIP and SUBSCRIBE_DROP accounting replaces the session-tail grace from lite-05 - [Re-run the SPS fuzzer on h264-reader](/quest/m2/sps-fuzz-h264-reader.md) - the H.264 SPS overflow is rechecked on the new parser - [Audio track changes are gapless](/quest/m2/gapless-audio-switch.md) - no silence beyond a frame on audio format or rendition changes, enforced by media QA - [PCM encodes in one pass](/quest/m2/pcm-bulk-encode.md) - the PCM encoder converts a frame in bulk instead of one sample at a time diff --git a/quest/m2/ietf-hole-marker.md b/quest/m2/ietf-hole-marker.md index 32267578599..77cf6ed36f9 100644 --- a/quest/m2/ietf-hole-marker.md +++ b/quest/m2/ietf-hole-marker.md @@ -4,7 +4,8 @@ A negotiated moq-transport extension (a new `drafts/draft-lcurley-moq-*` document) lets a publisher name each group it won't deliver on a SUBSCRIBE, -like moq-lite's SUBSCRIBE_DROP: never produced, or dropped with its reason. +like moq-lite's SUBSCRIBE_SKIP and SUBSCRIBE_DROP: never produced, or dropped +with its reason. A relay's copy ingested over moq-transport then knows its holes ("null") instead of leaving them unknown ("undefined"). Our moq-transport publishers then answer a multi-group FETCH, and a multi-group draft-22 fill, whenever @@ -12,8 +13,8 @@ the cache knows every group in the range: a local, moq-lite, or extension-carrying copy. Groups go out in order. End of Non-Existent Range (`0x8C`) marks only a proven hole: a group never produced, or a budget-independent drop (a cut, an origin's eviction or expiry). A stale -drop judged under a narrower budget or range, as in -[Drop set](/quest/m1/drop-set.md), proves nothing for a wider FETCH. That +drop judged under a narrower budget or range, as lite-07's `Max Age` and +`Terms` qualify it, proves nothing for a wider FETCH. That group is unknown, so the publisher fills it or the range is refused. Any other range stays refused, as [moq-dev/moq#7204](https://github.com/moq-dev/moq/pull/7204) @@ -59,9 +60,8 @@ Open, for this quest's interview: - The carrier: a message on the SUBSCRIBE's request stream (drafts 18 and 22 give each request its own stream), or a status on a group stream. -- Whether the drop names its budget, as lite-07's SUBSCRIBE_DROP `Max Age` - does in [Drop set](/quest/m1/drop-set.md), so a stale drop doesn't settle - a wider reader. +- Whether the drop names its budget and terms, as lite-07's SUBSCRIBE_DROP + `Max Age` and `Terms` do, so a stale drop doesn't settle a wider reader. - How a publisher answering a range walks the drop set's runs in group order, and where FETCH_OK's End Location falls. @@ -75,10 +75,6 @@ the extension is unchanged. Public API: none. Wire: a new negotiated SETUP option and its drop message. Our publishers answer multi-group FETCH and fills over known caches. -## Required - -- [Drop set](/quest/m1/drop-set.md) - the per-track runs that say which groups are known holes - ## Related - [Ordered reads wait only on owed groups](https://github.com/moq-dev/moq/pull/7403) - the ordered-reader wait a signalled hole ends diff --git a/quest/m2/lite-tail-drop-count.md b/quest/m2/lite-tail-drop-count.md index 0ec7946a4d7..bd3819dc000 100644 --- a/quest/m2/lite-tail-drop-count.md +++ b/quest/m2/lite-tail-drop-count.md @@ -2,7 +2,7 @@ ## Goal -From lite-05 on, a moq-lite session's tail is complete once every group up to the end has arrived or been reported in SUBSCRIBE_DROP. The 1 s `ledger::GRACE` (`rs/moq-net/src/ledger.rs:19`) then no longer decides it, in Rust and JS. +From lite-05 on, a moq-lite session's tail is complete once every group up to the end has arrived or been reported in SUBSCRIBE_SKIP or SUBSCRIBE_DROP. The 1 s `ledger::GRACE` (`rs/moq-net/src/ledger.rs:19`) then no longer decides it, in Rust and JS. ## Plan diff --git a/rs/moq-archive/src/consumer/tests.rs b/rs/moq-archive/src/consumer/tests.rs index 8a1b6b98f54..e71fffe90df 100644 --- a/rs/moq-archive/src/consumer/tests.rs +++ b/rs/moq-archive/src/consumer/tests.rs @@ -580,10 +580,9 @@ async fn a_completed_archive_ends_as_recorded() { ); assert_eq!(fetch(&broadcast, "errored", 0).await.unwrap(), expected(true, 0, 0..1)); - // The sequence below the end replays as an empty group, and those before it as dropped. + // The sequences the archive holds nothing for are dropped, and the end settles over them. let mut reader = subscribe(&broadcast, "tail").await; assert_eq!(next(&mut reader).await.unwrap(), (0, expected(true, 0, 0..1))); - assert_eq!(next(&mut reader).await.unwrap(), (3, Vec::new())); assert!(next(&mut reader).await.is_none()); assert_eq!(reader.finished().await.unwrap(), Position::group(4)); } diff --git a/rs/moq-archive/src/consumer/track.rs b/rs/moq-archive/src/consumer/track.rs index 5ca84109ef0..2ec3d1f2b97 100644 --- a/rs/moq-archive/src/consumer/track.rs +++ b/rs/moq-archive/src/consumer/track.rs @@ -141,17 +141,11 @@ impl Live { }, None, )?; - // An empty group stores no object, yet the end only settles once the live edge - // reaches it. The sequence just below the end replays as the empty group a - // source most often ended with, and any before it as dropped. + // An empty group stores no object, so what the archive holds nothing for up to the + // end is not found here, and the end settles over it. let fin = at.group + u64::from(at.frame != 0); let from = self.next.unwrap_or(0); - if from < fin { - self.producer.drop_groups(from..fin - 1, moq_net::StreamError::NotFound); - self.producer - .create_group(group::Info { sequence: fin - 1 })? - .finish()?; - } + self.producer.drop_groups(from..fin, moq_net::StreamError::NotFound); } End::Error(code) => { // After the newest stored sequence, keeping the history an abort would refuse. diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index 438b1e82399..30a9bb57eb1 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -2795,10 +2795,12 @@ impl TrackState { self.readable() || self.unfed && window.end().is_some() } - /// Whether the live edge reached `fin`, or a received copy's tail decides it instead: - /// what it still misses up to `fin` is waited for only as long as the grace. + /// Whether the live edge reached `fin`, counting a gap up to it that was wholly given up + /// (skipped or dropped), or a received copy's tail decides it instead: what it still + /// misses up to `fin` is waited for only as long as the grace. fn reaches(&self, fin: u64) -> bool { - self.edge() >= fin || self.tail.is_some() + let edge = self.edge(); + edge >= fin || self.tail.is_some() || self.drops.gaps(&(edge..fin)).next().is_none() } /// Whether the largest can no longer advance while a producer is still held: the track @@ -6905,9 +6907,11 @@ impl DropCursor { } } - /// Record `slot`, cached at `sequence`, if it holds a cut. + /// Record `slot`, cached at `sequence`, if it holds a cut no fetched copy covers: a + /// copy still serving the group's end delivers it, so it is never also dropped. fn cut(&mut self, sequence: u64, slot: &Slot) { if slot.visible + && !slot.serves_end() && let Some(err) = slot.group.cut_error() && let Some(end) = sequence.checked_add(1) { @@ -18960,6 +18964,20 @@ mod test { ); } + /// A group the publisher cut while a fetched copy still serves its end is delivered, + /// and never also reported dropped. + #[kio::test] + #[cfg(feature = "async")] + async fn a_cut_a_fetched_copy_covers_is_not_reported() { + let producer = track_producer("test", Info::timed()); + let mut subscriber = producer.subscribe(replay()); + let (live, _head) = live_group_with_head(&producer).await; + live.abort(Error::Stream(StreamError::App(5))).unwrap(); + let waiter = kio::Waiter::noop(); + assert!(subscriber.poll_drop(&waiter).is_pending(), "the copy covers the cut"); + assert_eq!(drain_groups(&mut subscriber), [0]); + } + /// Reads land on the fetched head, so they keep the slot from expiring even though /// the live copy beside it sits idle. #[kio::test] From 160a5ca32547376612ca4215e80346ebc37b183b Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 13:56:39 -0800 Subject: [PATCH 06/68] fix(net): supersede stale runs on a new upstream SUBSCRIBE and refuse a re-created dropped group in JS A re-established upstream subscription starts new terms, so stale runs the last one reported become unknown. The JS lite publisher no longer opens a second stream for a sequence it already accounted for, as Rust does. Co-Authored-By: Claude Opus 5.5 --- js/net/src/lite/publisher.test.ts | 6 +++--- js/net/src/lite/subscriber.ts | 2 ++ rs/moq-net/src/lite/subscriber.rs | 8 ++++++-- rs/moq-net/src/model/track.rs | 11 +++++++---- 4 files changed, 18 insertions(+), 9 deletions(-) diff --git a/js/net/src/lite/publisher.test.ts b/js/net/src/lite/publisher.test.ts index 6a2c192fbb5..3a40a450e5e 100644 --- a/js/net/src/lite/publisher.test.ts +++ b/js/net/src/lite/publisher.test.ts @@ -1147,8 +1147,9 @@ test("lite draft-06: the end sweep does not drop a sent datagram", async () => { } }); -// A drop describes what was missing then; a late copy of the same group is still served. -test("lite draft-06: a reported cut still allows a late group", async () => { +// A cut this subscription already dropped stays dropped, as in Rust: the subscriber accounted +// for it, so a group the publisher re-creates under it afterwards is not served. +test("lite draft-06: a reported cut stays dropped when re-created", async () => { const sub = await servedSubscription({ version: Version.DRAFT_06, startGroup: 0n }); try { sub.serve(0n); @@ -1163,7 +1164,6 @@ test("lite draft-06: a reported cut still allows a late group", async () => { sub.serve(1n); sub.serve(2n); - expect(await sub.servedSequence()).toBe(1n); expect(await sub.servedSequence()).toBe(2n); sub.track.finish(); sub.track.close(); diff --git a/js/net/src/lite/subscriber.ts b/js/net/src/lite/subscriber.ts index 5b004e3386e..1a486349b61 100644 --- a/js/net/src/lite/subscriber.ts +++ b/js/net/src/lite/subscriber.ts @@ -1071,6 +1071,8 @@ export class Subscriber { } const id = this.#subscribeNext++; const msg = this.#subscribeMessage(id, broadcast, consumed, producer.name, subscription, ranges); + // The new SUBSCRIBE's terms are new, so a stale run the last one reported is no verdict on them. + hooks.supersedeStale(producer); console.debug(`subscribe reopen: id=${id} broadcast=${broadcast} track=${producer.name}`); // Bounded like the first setup, and given up once nobody wants the track, so a peer // that stops reading holds neither the stream nor its entry. diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index 5848efa53dd..f5889db9658 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -8063,8 +8063,12 @@ impl TrackServe { match request { Some((subscription, ranges)) => { match sub { - // Open an upstream SUBSCRIBE for the first subscriber. - Sub::None => Ok(Begin::Establish(self.prepare_establish(producer, subscription, ranges))), + // Open an upstream SUBSCRIBE for the first subscriber. Its terms are new, so a + // stale run an earlier subscription reported is no verdict on them. + Sub::None => { + producer.supersede_stale(); + Ok(Begin::Establish(self.prepare_establish(producer, subscription, ranges))) + } Sub::Active(active) => { // Downstream preferences changed: forward them upstream as a // SUBSCRIBE_UPDATE (Lite03+ only; older peers can't carry one). diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index 30a9bb57eb1..45875279c06 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -6835,6 +6835,10 @@ struct Within<'a> { drift: Option<&'a Drift>, } +/// The [`TrackState::drops_revision`], span, window revision, and the budget and coverage +/// that settle received stale runs (see [`Drift::settles`]) a [`DropCursor`] folded in. +type Scan = (u64, ops::Range, u64, Option<(Duration, Option)>); + /// What a wire publisher's cursor gave up, found as it changes rather than by rescanning /// the cache on every poll. struct DropCursor { @@ -6844,10 +6848,9 @@ struct DropCursor { /// Sequences given up and not reported yet, with why: stale skips, cuts, and upstream /// skips and drops. pending: RangeMap, - /// The [`TrackState::drops_revision`], span, window revision, and what settles received - /// stale runs, last folded into `pending`. + /// What `pending` last folded in; see [`Scan`]. #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] - scanned: Option<(u64, ops::Range, u64, Option<(Duration, Option)>)>, + scanned: Option, /// Every sequence this cursor already reported. #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] reported: RangeSet, @@ -23191,7 +23194,7 @@ mod test { producer.give_up(1..3, Hole::stale(budget)); producer.skip_groups(3..4); let waiter = kio::Waiter::noop(); - let mut dropped = |subscriber: &mut Subscriber| { + let dropped = |subscriber: &mut Subscriber| { let mut runs = Vec::new(); while let Poll::Ready(run) = subscriber.poll_drop(&waiter) { runs.push(run); From ceb2b68e9cae627cd9af121263bbad7c0d42a6f6 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 14:18:54 -0800 Subject: [PATCH 07/68] fix(net): keep received stale runs from settling bounded readers or outliving their terms A bounded window never covers everything above a received stale run, so its completion no longer counts one. A cursor folds the track's runs in afresh on each rescan, so a superseded stale run leaves nothing pending, and the widening check runs only when the budget changes. Co-Authored-By: Claude Opus 5.5 --- js/net/src/internal.ts | 25 +++++-- js/net/src/track.test.ts | 34 ++++++++++ js/net/src/track.ts | 15 ++++- rs/moq-net/src/model/track.rs | 119 ++++++++++++++++++++++++++++++---- 4 files changed, 170 insertions(+), 23 deletions(-) diff --git a/js/net/src/internal.ts b/js/net/src/internal.ts index adceca6bbbc..17bd23a4a54 100644 --- a/js/net/src/internal.ts +++ b/js/net/src/internal.ts @@ -204,11 +204,19 @@ export function staleHole(budget: number): Hole { } /** A run of sequences given up, with why. */ -export interface DropRun extends Run, Hole {} +export interface DropRun extends Run, Hole { + /** Copied into a cursor from the track's runs, which it folds in afresh on each change. */ + received?: boolean; +} -/** Whether two holes say the same thing, so their runs may merge. */ -function sameHole(a: Hole, b: Hole): boolean { - return toStreamCode(a.error) === toStreamCode(b.error) && !a.skip === !b.skip && a.stale === b.stale; +/** Whether two runs say the same thing, so they may merge. */ +function sameHole(a: DropRun, b: DropRun): boolean { + return ( + toStreamCode(a.error) === toStreamCode(b.error) && + !a.skip === !b.skip && + a.stale === b.stale && + !a.received === !b.received + ); } /** @@ -216,12 +224,17 @@ function sameHole(a: Hole, b: Hole): boolean { * it overlaps. Neighbours given up alike merge, so a wide DROP stays one run and a peer naming * one sequence per DROP cannot grow the runs. */ -export function insertDrop(drops: readonly DropRun[], start: bigint, end: bigint, hole: Hole): DropRun[] { +export function insertDrop( + drops: readonly DropRun[], + start: bigint, + end: bigint, + hole: Hole & { received?: boolean }, +): DropRun[] { if (start >= end) return [...drops]; const next = removeRun(drops, start, end); let at = 0; while (at < next.length && next[at].start < start) at++; - next.splice(at, 0, { start, end, error: hole.error, skip: hole.skip, stale: hole.stale }); + next.splice(at, 0, { start, end, error: hole.error, skip: hole.skip, stale: hole.stale, received: hole.received }); const merged: DropRun[] = []; for (const run of next) { const last = merged[merged.length - 1]; diff --git a/js/net/src/track.test.ts b/js/net/src/track.test.ts index 770dfd05c8d..124edcb90ce 100644 --- a/js/net/src/track.test.ts +++ b/js/net/src/track.test.ts @@ -3878,6 +3878,40 @@ test("a superseded stale run is unknown again, and its group is handed out", () producer.close(); }); +test("superseding withdraws a partly reported stale run", () => { + const producer = new TrackProducer("video").accept({}); + const reader = producer.subscribe({ delay: { max: Milli(500) }, ranges: [{ start: { absolute: 0n } }] }); + hooks.dropTrackGroups(producer, 4n, 5n, { error: DROPPED }); + expect(hooks.tryDrop(reader)).toMatchObject({ start: 4n, end: 5n }); + hooks.dropTrackGroups(producer, 3n, 7n, staleHole(500)); + expect(hooks.tryDrop(reader)).toMatchObject({ start: 3n, end: 4n, stale: 500 }); + hooks.supersedeStale(producer); + // 5 and 6 are unknown again. + expect(hooks.tryDrop(reader)).toBeUndefined(); + reader.close(); + producer.finish(); + producer.close(); +}); + +test("a received stale run does not end a bounded reader", () => { + const producer = new TrackProducer("video").accept({}); + hooks.receiveTrack(producer); + const reader = producer.subscribe({ + delay: { max: Milli(5000) }, + ranges: [{ start: { absolute: 0n }, end: { absolute: 2n } }], + }); + hooks.dropTrackGroups(producer, 0n, 2n, staleHole(500)); + // The groups are still wanted. + expect(hooks.tryRecvGroup(reader, false)).toEqual({ kind: "idle" }); + producer.createGroup(0n).finish(); + const recv = hooks.tryRecvGroup(reader, false); + expect(recv.kind === "group" ? recv.group.sequence : recv).toBe(0n); + if (recv.kind === "group") recv.group.close(); + reader.close(); + producer.finish(); + producer.close(); +}); + test("a wide drop is held and reported as one run", () => { const producer = new TrackProducer("video").accept({}); const reader = producer.subscribe({ delay: { max: Milli(60_000) } }); diff --git a/js/net/src/track.ts b/js/net/src/track.ts index ac2f4fe0f2f..31a4c0dcbc6 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -2057,6 +2057,8 @@ export class Subscriber { // since the owner last drained them. #reoffersReported = false; #unreported: Run[] = []; + // The budget the reported stale runs were last checked against: only a change can reverse one. + #reofferedAt?: number; // The group the frame-level helpers are currently draining, acquired through the // sequence cursor so frame reads and {@link Ordered.nextGroup} share one floor. #frameGroup?: GroupConsumer; @@ -2336,7 +2338,9 @@ export class Subscriber { // skipped groups, never the cache. #reoffer(budget: number): void { // A reported stale run reversed is judged again, and owed again by the wire. - if (this.#reoffersReported && this.#reportedStale.some((run) => (run.stale ?? budget) < budget)) { + const changed = this.#reofferedAt !== budget; + this.#reofferedAt = budget; + if (this.#reoffersReported && changed && this.#reportedStale.some((run) => (run.stale ?? budget) < budget)) { for (const run of this.#reportedStale) { if ((run.stale ?? budget) >= budget) continue; this.#reported = removeRun(this.#reported, run.start, run.end); @@ -2507,7 +2511,9 @@ export class Subscriber { if (end === undefined) return false; const held = this.#state.groups.peek(); const received = this.#state.received.peek(); - const drops = this.#state.drops.peek(); + // A bounded window never covers everything above a received stale run, so such a run + // does not settle it (see `settles`): the reader still wants those groups. + const drops = this.#state.drops.peek().filter((run) => run.stale === undefined); const producer = this.#state.producer; for (const run of this.#window.runs(this.#window.floor() ?? 0n, end)) { let gaps = [run]; @@ -3118,10 +3124,13 @@ export class Subscriber { seen.coversFrom !== coversFrom ) { this.#dropsSeen = { drops, start, end, revision, budget, coversFrom }; + // The track's runs are folded in afresh, so one it forgot, such as a stale run new terms + // superseded, or one that stopped settling this reader, leaves nothing behind. + this.#pending = this.#pending.filter((run) => !run.received); for (const run of drops) { // A received stale run that does not settle this reader is unknown to it. if (!settles(run, budget, coversFrom)) continue; - this.#pending = insertDrop(this.#pending, run.start, run.end, run); + this.#pending = insertDrop(this.#pending, run.start, run.end, { ...run, received: true }); } } for (let run = this.#pending.shift(); run; run = this.#pending.shift()) { diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index 45875279c06..089a8d5bf98 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -6776,6 +6776,9 @@ struct Cursor { reoffers_reported: bool, /// The reported stale runs a widening took back since the owner last drained them. unreported: Vec>, + /// The budget the reported stale runs were last checked against: only a change can + /// reverse one, so an unchanged budget costs nothing per poll. + reoffered_at: Option, /// What this cursor misses and asks fills for, once it first reads. misses: Option>, /// The heads a bounded window asked for, once it first finds a group missing one. @@ -6835,6 +6838,22 @@ struct Within<'a> { drift: Option<&'a Drift>, } +/// A run a [`DropCursor`] has yet to report, and whether it was copied from the track's +/// runs, which a rescan folds in afresh: one the track forgot, such as a stale run new terms +/// superseded, leaves nothing behind. +#[derive(Clone, Debug, PartialEq, Eq)] +struct Pending { + hole: Hole, + received: bool, +} + +impl Pending { + /// A verdict this cursor reached itself. + fn own(hole: Hole) -> Self { + Self { hole, received: false } + } +} + /// The [`TrackState::drops_revision`], span, window revision, and the budget and coverage /// that settle received stale runs (see [`Drift::settles`]) a [`DropCursor`] folded in. type Scan = (u64, ops::Range, u64, Option<(Duration, Option)>); @@ -6847,7 +6866,7 @@ struct DropCursor { cuts: super::aborts::Watch, /// Sequences given up and not reported yet, with why: stale skips, cuts, and upstream /// skips and drops. - pending: RangeMap, + pending: RangeMap, /// What `pending` last folded in; see [`Scan`]. #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] scanned: Option, @@ -6880,7 +6899,7 @@ impl DropCursor { let pending: Vec<_> = self .pending .iter() - .filter(|(_, hole)| hole.stale_budget().is_some_and(|judged| judged < budget)) + .filter(|(_, pending)| pending.hole.stale_budget().is_some_and(|judged| judged < budget)) .map(|(run, _)| run.clone()) .collect(); for run in pending { @@ -6892,12 +6911,12 @@ impl DropCursor { .filter(|(_, judged)| **judged < budget) .map(|(run, _)| run.clone()) .collect(); + // A wider budget changes the scan's key, so a received stale run it no longer settles is + // left out of the next scan. for run in &widened { self.reported_stale.remove(run.clone()); self.reported.remove(run.clone()); } - // Rescanned, so a received stale run a wider budget no longer settles stays unknown. - self.scanned = None; widened } @@ -6906,7 +6925,7 @@ impl DropCursor { fn give_up(&mut self, run: ops::Range, hole: Hole) { let gaps: Vec<_> = self.pending.gaps(&run).collect(); for gap in gaps { - self.pending.insert(gap, hole.clone()); + self.pending.insert(gap, Pending::own(hole.clone())); } } @@ -6918,7 +6937,8 @@ impl DropCursor { && let Some(err) = slot.group.cut_error() && let Some(end) = sequence.checked_add(1) { - self.pending.insert(sequence..end, Hole::drop(StreamError::from(err))); + self.pending + .insert(sequence..end, Pending::own(Hole::drop(StreamError::from(err)))); } } @@ -6937,6 +6957,15 @@ impl DropCursor { let scan = (state.drops_revision, span.clone(), within.revision, settles); if self.scanned.as_ref() != Some(&scan) { self.scanned = Some(scan); + let stale: Vec<_> = self + .pending + .iter() + .filter(|(_, pending)| pending.received) + .map(|(run, _)| run.clone()) + .collect(); + for run in stale { + self.pending.remove(run); + } for (dropped, hole) in state.drops.overlapping(span.clone()) { let run = dropped.start.max(span.start)..dropped.end.min(span.end); // A received stale run that does not settle this reader is unknown to it. @@ -6945,18 +6974,24 @@ impl DropCursor { None => hole.stale_budget().is_none(), }; if settled { - self.pending.insert(run, hole.clone()); + self.pending.insert( + run, + Pending { + hole: hole.clone(), + received: true, + }, + ); } } for (&sequence, slot) in state.lookup.range(span.clone()) { self.cut(sequence, slot); } } - while let Some((range, hole)) = self + while let Some((range, pending)) = self .pending .iter() .next() - .map(|(range, hole)| (range.clone(), hole.clone())) + .map(|(range, pending)| (range.clone(), pending.clone())) { self.pending.remove(range.clone()); // Only what this subscriber asked for is owed: the first run of its ranges in it, @@ -6965,12 +7000,13 @@ impl DropCursor { continue; }; if clipped.end < range.end { - self.pending.insert(clipped.end..range.end, hole.clone()); + self.pending.insert(clipped.end..range.end, pending.clone()); } if let Some(gap) = self.reported.gaps(&clipped).next() { if gap.end < range.end { - self.pending.insert(gap.end..range.end, hole.clone()); + self.pending.insert(gap.end..range.end, pending.clone()); } + let hole = pending.hole; self.reported.insert(gap.clone()); if let Some(budget) = hole.stale_budget() { self.reported_stale.insert(gap.clone(), budget); @@ -7074,6 +7110,7 @@ impl Cursor { drops: None, reoffers_reported: false, unreported: Vec::new(), + reoffered_at: None, misses: None, heads: None, idle: None, @@ -7265,7 +7302,13 @@ impl Cursor { left.remove(run.clone()); } } - for (run, _) in state.drops.overlapping(&gap) { + // A bounded window never covers everything above a received stale run, so such a + // run does not settle it (see `Drift::settles`): the reader still wants those groups. + for (run, _) in state + .drops + .overlapping(&gap) + .filter(|(_, hole)| hole.stale_budget().is_none()) + { left.remove(run.clone()); } for sequence in self.seek_pending.range(gap).map(|(sequence, _)| *sequence) { @@ -7458,6 +7501,7 @@ impl Cursor { // A reported stale run reversed is judged again: a group still cached is owed below // as a skipped one is, and a missing one is missed again. if self.reoffers_reported + && self.reoffered_at.replace(budget) != Some(budget) && let Some(drops) = &mut self.drops { for run in drops.widened(budget) { @@ -7477,7 +7521,7 @@ impl Cursor { let old: Vec<_> = drops .pending .overlapping(span.clone()) - .filter(|(_, hole)| hole.stale_budget().is_some()) + .filter(|(_, pending)| pending.hole.stale_budget().is_some()) .map(|(range, _)| range.start.max(span.start)..range.end.min(span.end)) .collect(); for range in old { @@ -7974,7 +8018,9 @@ impl Cursor { if let Some(drops) = &mut self.drops && let Some(end) = sequence.checked_add(1) { - drops.pending.insert(sequence..end, Hole::stale(drift.budget)); + drops + .pending + .insert(sequence..end, Pending::own(Hole::stale(drift.budget))); } match drift.bounded { // A re-offered group stale at the bound is final. @@ -23241,6 +23287,51 @@ mod test { assert_eq!(drain_groups(&mut subscriber), [1]); } + /// A superseded stale run leaves nothing a cursor still holds to report, even the part + /// of it queued behind an earlier report. + #[test] + #[cfg(feature = "async")] + fn superseding_withdraws_a_partly_reported_stale_run() { + let producer = track_producer("test", Info::timed()); + let budget = Duration::from_millis(500); + let mut subscriber = producer.subscribe( + Subscription::default() + .with_delay(Delay::default().with_max(budget)) + .with_ranges([Range::groups(0..)]), + ); + let waiter = kio::Waiter::noop(); + producer.drop_groups(4..5, StreamError::NotFound); + assert_eq!( + subscriber.poll_drop(&waiter), + Poll::Ready((4..5, Hole::drop(StreamError::NotFound))) + ); + producer.give_up(3..7, Hole::stale(budget)); + assert_eq!(subscriber.poll_drop(&waiter), Poll::Ready((3..4, Hole::stale(budget)))); + producer.supersede_stale(); + assert!(subscriber.poll_drop(&waiter).is_pending(), "5 and 6 are unknown again"); + } + + /// A bounded reader is never settled by a received stale run, so it still waits for the + /// groups the run names rather than end. + #[test] + #[cfg(feature = "async")] + fn a_received_stale_run_does_not_end_a_bounded_reader() { + let producer = track_producer("test", Info::timed()); + producer.received(); + let mut subscriber = producer.subscribe( + Subscription::default() + .with_delay(Delay::default().with_max(Duration::from_secs(5))) + .with_ranges([Range::groups(0..2)]), + ); + producer.give_up(0..2, Hole::stale(Duration::from_millis(500))); + assert!( + subscriber.poll_recv_group(&kio::Waiter::noop()).is_pending(), + "the groups are still wanted" + ); + write_group(&producer, 0); + assert_eq!(drain_groups(&mut subscriber), [0]); + } + /// A report describes a miss, not a delivery: a late group is still handed out once. #[test] #[cfg(feature = "async")] From 35971a4ec98c403e367a20f96fea04eae6af330e Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 14:25:39 -0800 Subject: [PATCH 08/68] fix(net): settle a received stale run up to where the upstream's range ended A relay records where the ranges of its current terms end, and a stale run settles a reader whose range covers it and everything above it up to there, so a bounded subscription is settled by its own upstream's stale drop. A JS bounded read no longer ends over a stale run that does not settle it. Co-Authored-By: Claude Opus 5.5 --- js/net/src/internal.ts | 5 +- js/net/src/lite/subscriber.ts | 19 +++- js/net/src/track.test.ts | 34 ++++--- js/net/src/track.ts | 63 ++++++++----- rs/moq-net/src/lite/subscriber.rs | 19 +++- rs/moq-net/src/model/track.rs | 145 ++++++++++++++++++++--------- rs/moq-net/src/model/track/miss.rs | 14 +-- 7 files changed, 208 insertions(+), 91 deletions(-) diff --git a/js/net/src/internal.ts b/js/net/src/internal.ts index 17bd23a4a54..d93eea15630 100644 --- a/js/net/src/internal.ts +++ b/js/net/src/internal.ts @@ -503,9 +503,10 @@ export const hooks: { dropTrackGroups: (producer: Producer, start: bigint, end: bigint, hole: Hole) => void; /** * Forget every run a received copy holds as stale: the terms its upstream judged them under - * were replaced, so they are unknown again. Mirrors the Rust `supersede_stale`. + * were replaced, so they are unknown again. The new terms' ranges end at `judgedEnd`, + * undefined past the live edge or unknown. Mirrors the Rust `supersede_stale`. */ - supersedeStale: (producer: Producer) => void; + supersedeStale: (producer: Producer, judgedEnd: bigint | undefined) => void; /** Mark a track as fed by a wire subscription, which owes missing groups. Assigned by `track.ts`. */ receiveTrack: (producer: Producer) => void; /** Create a received group at its wire offset before publishing it to readers. Assigned by `track.ts`. */ diff --git a/js/net/src/lite/subscriber.ts b/js/net/src/lite/subscriber.ts index 1a486349b61..b71301c338e 100644 --- a/js/net/src/lite/subscriber.ts +++ b/js/net/src/lite/subscriber.ts @@ -213,6 +213,19 @@ export class Minted { } } +/** + * Where a stale drop judged under `ranges` stops looking: the exclusive end past every group they + * name, undefined when one stays open or ends at an anchor only the publisher resolves. + */ +function judgedEnd(ranges: readonly track.Range[]): bigint | undefined { + let end: bigint | undefined; + for (const range of ranges) { + if (range.end?.absolute === undefined) return undefined; + if (end === undefined || range.end.absolute > end) end = range.end.absolute; + } + return end; +} + /** Subscribe to the peer and own its receive-side protocol tasks. * @internal */ @@ -1072,7 +1085,7 @@ export class Subscriber { const id = this.#subscribeNext++; const msg = this.#subscribeMessage(id, broadcast, consumed, producer.name, subscription, ranges); // The new SUBSCRIBE's terms are new, so a stale run the last one reported is no verdict on them. - hooks.supersedeStale(producer); + hooks.supersedeStale(producer, judgedEnd(ranges)); console.debug(`subscribe reopen: id=${id} broadcast=${broadcast} track=${producer.name}`); // Bounded like the first setup, and given up once nobody wants the track, so a peer // that stops reading holds neither the stream nor its entry. @@ -1397,6 +1410,8 @@ export class Subscriber { state: SubscribeSetup, ): SubscribeEntry { hooks.receiveTrack(producer); + // The SUBSCRIBE's terms: where a stale drop judged under them stops looking. + hooks.supersedeStale(producer, judgedEnd(ranges)); if (!hasRanges(this.version)) hooks.dropAnchors(producer); // Lite-06+ resolves where the feed starts only with SUBSCRIBE_START, so a relay serving // this copy holds its first group until the answer arrives. @@ -1993,7 +2008,7 @@ export class Subscriber { // may move the range it was judged against, so the runs held as stale become unknown // again: a reader fills them rather than trust an old verdict. entry.terms = update.terms; - hooks.supersedeStale(entry.track); + hooks.supersedeStale(entry.track, judgedEnd(ranges)); // Encoded apart from the write, so a bound the wire can't carry still fails the track. const encoded = await Writer.collect(this.version, (w) => update.encode(w, this.version)); // A failed write means the publisher stopped reading, as it does once it finishes the diff --git a/js/net/src/track.test.ts b/js/net/src/track.test.ts index 124edcb90ce..79197d08c2d 100644 --- a/js/net/src/track.test.ts +++ b/js/net/src/track.test.ts @@ -3867,7 +3867,7 @@ test("a superseded stale run is unknown again, and its group is handed out", () const producer = new TrackProducer("video").accept({}); const reader = producer.subscribe({ delay: { max: Milli(500) }, ranges: [{ start: { absolute: 0n } }] }); hooks.dropTrackGroups(producer, 1n, 2n, staleHole(500)); - hooks.supersedeStale(producer); + hooks.supersedeStale(producer, undefined); expect(hooks.tryDrop(reader)).toBeUndefined(); producer.createGroup(1n).finish(); const recv = hooks.tryRecvGroup(reader, true); @@ -3885,7 +3885,7 @@ test("superseding withdraws a partly reported stale run", () => { expect(hooks.tryDrop(reader)).toMatchObject({ start: 4n, end: 5n }); hooks.dropTrackGroups(producer, 3n, 7n, staleHole(500)); expect(hooks.tryDrop(reader)).toMatchObject({ start: 3n, end: 4n, stale: 500 }); - hooks.supersedeStale(producer); + hooks.supersedeStale(producer, undefined); // 5 and 6 are unknown again. expect(hooks.tryDrop(reader)).toBeUndefined(); reader.close(); @@ -3893,21 +3893,33 @@ test("superseding withdraws a partly reported stale run", () => { producer.close(); }); -test("a received stale run does not end a bounded reader", () => { +test("a received stale run settles a bounded reader only to the judged end", () => { const producer = new TrackProducer("video").accept({}); hooks.receiveTrack(producer); - const reader = producer.subscribe({ - delay: { max: Milli(5000) }, - ranges: [{ start: { absolute: 0n }, end: { absolute: 2n } }], - }); + const bounded = (max: number) => + producer.subscribe({ + delay: { max: Milli(max) }, + ranges: [{ start: { absolute: 0n }, end: { absolute: 2n } }], + }); + const wider = bounded(5000); + const matching = bounded(500); + // The upstream judged its own range of 0..2 under 500ms. + hooks.supersedeStale(producer, 2n); hooks.dropTrackGroups(producer, 0n, 2n, staleHole(500)); - // The groups are still wanted. - expect(hooks.tryRecvGroup(reader, false)).toEqual({ kind: "idle" }); + // A wider budget still wants the groups, however the feed accounted for them. + expect(hooks.tryRecvGroup(wider, false)).toEqual({ kind: "idle" }); + expect(hooks.tryDrop(matching)).toMatchObject({ start: 0n, end: 2n, stale: 500 }); + expect(hooks.tryRecvGroup(matching, false)).toEqual({ kind: "done" }); + // Judged against an open range, a capped reader may see an older edge. + hooks.supersedeStale(producer, undefined); + hooks.dropTrackGroups(producer, 0n, 2n, staleHole(500)); + const capped = bounded(500); + expect(hooks.tryDrop(capped)).toBeUndefined(); producer.createGroup(0n).finish(); - const recv = hooks.tryRecvGroup(reader, false); + const recv = hooks.tryRecvGroup(wider, false); expect(recv.kind === "group" ? recv.group.sequence : recv).toBe(0n); if (recv.kind === "group") recv.group.close(); - reader.close(); + for (const reader of [wider, matching, capped]) reader.close(); producer.finish(); producer.close(); }); diff --git a/js/net/src/track.ts b/js/net/src/track.ts index 31a4c0dcbc6..c99fe823ec5 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -629,6 +629,9 @@ class TrackState { // DROP costs one entry. Replaced rather than mutated, so a cursor can tell it changed by // identity. drops = new Signal([]); + // Where the ranges a received copy's upstream judges stale runs against end: the exclusive end + // of its current terms, undefined past the live edge or unknown. + judgedEnd?: bigint; // Every group still in the producer's replay cache, including groups this // subscriber already consumed, sorted by sequence. Drift anchors have the same // lifetime as content. Sorted so the latency guard, evaluated per group and per @@ -759,16 +762,22 @@ let goneAt: (producer: Producer, sequence: bigint) => bigint | undefined; // Whether the producer is fed by a wire subscription; assigned in its static block. let received: (producer: Producer) => boolean; +/** A reader's last contiguous run of groups: its start, and its exclusive end unless it reads past the live edge. */ +type Coverage = { start: bigint; end?: bigint } | undefined; + /** - * Whether `hole`, over a run from `run.start`, settles a reader judging under `budget` that - * covers everything from `coversFrom` up. Any hole but a received stale one does. A stale - * verdict holds for a reader no wider than the budget it was judged under, whose range covers - * the run and every group above it: its own successor and edge are then no older than those the - * verdict was judged against. Mirrors the Rust `Drift::settles`. + * Whether `run` settles a reader judging under `budget` with `coverage`, on a track whose upstream + * judged stale runs against ranges ending at `judgedEnd` (undefined past the live edge, or + * unknown). Any run but a received stale one does. A stale verdict holds for a reader no wider + * than the budget it was judged under, whose range covers the run and every group above it up to + * where the upstream's range ended: its own successor and edge are then no older than those the + * verdict was judged against. Mirrors the Rust `settles`. */ -function settles(run: DropRun, budget: number, coversFrom: bigint | undefined): boolean { +function settles(run: DropRun, budget: number, coverage: Coverage, judgedEnd: bigint | undefined): boolean { if (run.stale === undefined) return true; - return budget <= run.stale && coversFrom !== undefined && coversFrom <= run.start; + if (budget > run.stale || coverage === undefined || coverage.start > run.start) return false; + if (coverage.end === undefined) return true; + return judgedEnd !== undefined && coverage.end >= judgedEnd; } // Actual object metadata outlives cached payloads, unlike empty group heads in model largest. let objectLargest: (producer: Producer) => Position | undefined; @@ -954,8 +963,9 @@ export class Producer { hooks.trackReleased = (producer) => producer.#state.closed; hooks.trackFinished = (producer) => producer.#state.end.peek() !== undefined; hooks.dropTrackGroups = (producer, start, end, hole) => producer.#drop(start, end, hole, true); - hooks.supersedeStale = (producer) => { + hooks.supersedeStale = (producer, judgedEnd) => { for (const state of [producer.#state, ...producer.#sinks]) { + state.judgedEnd = judgedEnd; const drops = state.drops.peek(); if (drops.some((run) => run.stale !== undefined)) state.drops.set(drops.filter((run) => run.stale === undefined)); @@ -2047,7 +2057,8 @@ export class Subscriber { end?: bigint; revision: number; budget: number; - coversFrom?: bigint; + coverage?: Coverage; + judgedEnd?: bigint; }; #reported: Run[] = []; // The reported runs judged stale, with the budget each was judged under, for a wire whose @@ -2511,15 +2522,20 @@ export class Subscriber { if (end === undefined) return false; const held = this.#state.groups.peek(); const received = this.#state.received.peek(); - // A bounded window never covers everything above a received stale run, so such a run - // does not settle it (see `settles`): the reader still wants those groups. - const drops = this.#state.drops.peek().filter((run) => run.stale === undefined); + // A received stale run that does not settle this reader is unknown to it (see `settles`): + // the reader still wants those groups, however the feed accounted for them. + const budget = this.#drift().budget; + const coverage = this.#coverage(); + const all = this.#state.drops.peek(); + const drops = all.filter((run) => settles(run, budget, coverage, this.#state.judgedEnd)); + const unknown = all.filter((run) => !settles(run, budget, coverage, this.#state.judgedEnd)); const producer = this.#state.producer; for (const run of this.#window.runs(this.#window.floor() ?? 0n, end)) { let gaps = [run]; for (const delivered of taken) gaps = gaps.flatMap((gap) => delivered.gaps(gap.start, gap.end)); for (const gap of gaps) { for (const open of runGaps(drops, gap.start, gap.end)) { + if (unknown.some((run) => run.start < open.end && open.start < run.end)) return false; // Not received yet, so it may still arrive. if (open.end > received) return false; // Held, so a read still hands it out, skips it, or waits for the cap to rise. @@ -3112,7 +3128,8 @@ export class Subscriber { const drops = this.#state.drops.peek(); const revision = this.#revision; const { budget } = this.#drift(); - const coversFrom = this.#coversFrom(); + const coverage = this.#coverage(); + const judgedEnd = this.#state.judgedEnd; const seen = this.#dropsSeen; if ( !seen || @@ -3121,15 +3138,17 @@ export class Subscriber { seen.end !== end || seen.revision !== revision || seen.budget !== budget || - seen.coversFrom !== coversFrom + seen.coverage?.start !== coverage?.start || + seen.coverage?.end !== coverage?.end || + seen.judgedEnd !== judgedEnd ) { - this.#dropsSeen = { drops, start, end, revision, budget, coversFrom }; + this.#dropsSeen = { drops, start, end, revision, budget, coverage, judgedEnd }; // The track's runs are folded in afresh, so one it forgot, such as a stale run new terms // superseded, or one that stopped settling this reader, leaves nothing behind. this.#pending = this.#pending.filter((run) => !run.received); for (const run of drops) { // A received stale run that does not settle this reader is unknown to it. - if (!settles(run, budget, coversFrom)) continue; + if (!settles(run, budget, coverage, judgedEnd)) continue; this.#pending = insertDrop(this.#pending, run.start, run.end, { ...run, received: true }); } } @@ -3142,12 +3161,14 @@ export class Subscriber { return undefined; } - // The lowest group from which this reader reads every group up past the live edge, undefined - // while a range end or a cap bounds it. - #coversFrom(): bigint | undefined { - if (this.#cursor.peek().end !== undefined || this.#window.pending) return undefined; + // This reader's last contiguous run of groups, capped; see `Coverage`. + #coverage(): Coverage { + if (this.#window.pending) return undefined; const last = this.#window.runs().at(-1); - return last?.end === UNBOUNDED ? last.start : undefined; + if (!last) return undefined; + const cap = this.#cursor.peek().end; + const end = last.end === UNBOUNDED ? cap : cap === undefined ? last.end : min(last.end, cap); + return { start: last.start, end }; } // A group from the routes' copies; see #resume. diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index f5889db9658..57006eb0a17 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -8066,7 +8066,7 @@ impl TrackServe { // Open an upstream SUBSCRIBE for the first subscriber. Its terms are new, so a // stale run an earlier subscription reported is no verdict on them. Sub::None => { - producer.supersede_stale(); + producer.supersede_stale(judged_end(&ranges)); Ok(Begin::Establish(self.prepare_establish(producer, subscription, ranges))) } Sub::Active(active) => { @@ -8357,6 +8357,21 @@ fn wire_max_delay(max_delay: Option) -> std::time::Duration max_delay.unwrap_or(std::time::Duration::from_millis(super::MAX_SAFE_DELAY_MS)) } +/// Where a stale drop judged under `ranges` stops looking: the exclusive end past every +/// group they name, `None` when one stays open or ends at an anchor only the publisher +/// resolves. +fn judged_end(ranges: &[lite::Range]) -> Option { + ranges + .iter() + .map(|range| match range.end { + Some(lite::Anchor::Absolute(end)) => Some(end.group.saturating_add(u64::from(end.frame > 0))), + _ => None, + }) + .collect::>>()? + .into_iter() + .max() +} + /// Buffer the current SUBSCRIBE parameters under new range terms. The caller flushes. /// /// A stale drop judged under earlier terms may still be in flight, and the update may have @@ -8365,7 +8380,7 @@ fn wire_max_delay(max_delay: Option) -> std::time::Duration fn buffer_update(active: &mut SubStream, serving: &track::Producer) -> Result<(), Error> { active.bounded |= bounded(&active.ranges); active.terms += 1; - serving.supersede_stale(); + serving.supersede_stale(judged_end(&active.ranges)); active.stream.writer.buffer(&lite::SubscribeUpdate { wanted_group: active.wanted, priority: active.priority, diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index 089a8d5bf98..340ec516e3b 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -501,6 +501,9 @@ pub(crate) struct TrackState { // cannot erase a disposition a relay must forward, and as ranges so a wide one costs // one entry. drops: RangeMap, + // Where the ranges a received copy's upstream judges stale runs against end: the + // exclusive end of its current terms, `None` past the live edge or unknown. + judged_end: Option, // Bumped on every insert into `drops`, so a cursor rescans it only after a change. drops_revision: u64, drops_changes: ledger::Changes, @@ -3790,10 +3793,12 @@ impl Producer { /// Forget every run held as stale: the terms a received copy's upstream judged them /// under were replaced, so they are unknown again, and a reader that wants one fills it. + /// The new terms' ranges end at `judged_end`, `None` past the live edge or unknown. #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] - pub(crate) fn supersede_stale(&self) { + pub(crate) fn supersede_stale(&self, judged_end: Option) { if let Ok(mut state) = self.modify() { state.supersede_stale(); + state.judged_end = judged_end; } } @@ -6110,22 +6115,41 @@ struct Drift { /// The budget is the track's max age bound, which no widening passes: its verdicts are final. bounded: bool, edge: Edge, - /// The lowest group from which this reader reads every group up past the live edge, - /// `None` while a range end or a cap bounds it. - covers_from: Option, + /// This reader's last contiguous run of groups: where it starts and its exclusive end, + /// `None` when it reads up past the live edge; see [`Coverage`]. + coverage: Coverage, +} + +/// A reader's last contiguous run of groups, its start and exclusive end (`None` past the +/// live edge), or `None` before its ranges resolve. +type Coverage = Option<(u64, Option)>; + +/// Whether a reader judging under `budget` with `coverage` is settled by `hole` over a run +/// from `start`, on a track whose upstream judged stale runs against ranges ending at +/// `judged_end` (`None` past the live edge, or unknown). Any hole but a received stale one +/// settles every reader. A stale verdict holds for a reader no wider than the budget it was +/// judged under, whose range covers the run and every group above it up to where the +/// upstream's range ended: its own successor and edge are then no older than those the +/// verdict was judged against. Any other reader treats the run as unknown, and fills it. +fn settles(budget: Duration, coverage: Coverage, start: u64, hole: &Hole, judged_end: Option) -> bool { + let Some(judged) = hole.stale_budget() else { + return true; + }; + let covers = coverage.is_some_and(|(from, until)| { + from <= start + && match (until, judged_end) { + (None, _) => true, + (Some(until), Some(end)) => until >= end, + (Some(_), None) => false, + } + }); + budget <= judged && covers } impl Drift { - /// Whether `hole`, over a run from `start`, settles this reader. Any hole but a received - /// stale one does. A stale verdict holds for a reader no wider than the budget it was - /// judged under, whose range covers the run and every group above it: its own successor - /// and edge are then no older than those the verdict was judged against, wherever the - /// upstream's range ended. Any other reader treats the run as unknown, and fills it. - fn settles(&self, start: u64, hole: &Hole) -> bool { - match hole.stale_budget() { - None => true, - Some(budget) => self.budget <= budget && self.covers_from.is_some_and(|from| from <= start), - } + /// Whether `hole`, over a run from `start`, settles this reader on `state`; see [`settles`]. + fn settles(&self, state: &TrackState, start: u64, hole: &Hole) -> bool { + settles(self.budget, self.coverage, start, hole, state.judged_end) } } @@ -6856,7 +6880,7 @@ impl Pending { /// The [`TrackState::drops_revision`], span, window revision, and the budget and coverage /// that settle received stale runs (see [`Drift::settles`]) a [`DropCursor`] folded in. -type Scan = (u64, ops::Range, u64, Option<(Duration, Option)>); +type Scan = (u64, ops::Range, u64, Option<(Duration, Coverage, Option)>); /// What a wire publisher's cursor gave up, found as it changes rather than by rescanning /// the cache on every poll. @@ -6953,7 +6977,9 @@ impl DropCursor { } // Rescan only after the drops change, the window moves, or what settles a stale run // does; out-of-window and unsettling runs were skipped and may be owed now. - let settles = within.drift.map(|drift| (drift.budget, drift.covers_from)); + let settles = within + .drift + .map(|drift| (drift.budget, drift.coverage, state.judged_end)); let scan = (state.drops_revision, span.clone(), within.revision, settles); if self.scanned.as_ref() != Some(&scan) { self.scanned = Some(scan); @@ -6970,7 +6996,7 @@ impl DropCursor { let run = dropped.start.max(span.start)..dropped.end.min(span.end); // A received stale run that does not settle this reader is unknown to it. let settled = match within.drift { - Some(drift) => drift.settles(run.start, hole), + Some(drift) => drift.settles(state, run.start, hole), None => hole.stale_budget().is_none(), }; if settled { @@ -7290,7 +7316,9 @@ impl Cursor { return false; } let given_up = self.misses.as_ref().map(|misses| misses.given_up()); + let coverage = self.coverage(); let state = self.state.read(); + let budget = clamp_max_delay(self.subscription.read().budget(), state.max_age_bound()); let accounted = |gap: ops::Range| { let mut left = RangeSet::new(); left.insert(gap.clone()); @@ -7302,12 +7330,12 @@ impl Cursor { left.remove(run.clone()); } } - // A bounded window never covers everything above a received stale run, so such a - // run does not settle it (see `Drift::settles`): the reader still wants those groups. + // A received stale run that does not settle this reader is unknown to it, so it + // still wants those groups. for (run, _) in state .drops .overlapping(&gap) - .filter(|(_, hole)| hole.stale_budget().is_none()) + .filter(|(run, hole)| settles(budget, coverage, run.start.max(gap.start), hole, state.judged_end)) { left.remove(run.clone()); } @@ -7579,6 +7607,21 @@ impl Cursor { std::mem::take(&mut self.stale) } + /// This reader's last contiguous run of groups, capped; see [`Coverage`]. + fn coverage(&self) -> Coverage { + let run = self.filter.window.groups()?.last(0, u64::MAX)?; + let end = match *run.end() { + u64::MAX => None, + last => Some(last + 1), + }; + let end = match (end, self.end_sequence) { + (end, None) => end, + (None, Some(cap)) => Some(cap), + (Some(end), Some(cap)) => Some(end.min(cap)), + }; + Some((*run.start(), end)) + } + /// This subscriber's clamped drift budget and the live edge to measure against, /// resolved once per poll. /// @@ -7595,24 +7638,14 @@ impl Cursor { max_delay = subscription.budget(); Poll::<()>::Pending }); - // Uncapped, an open last range reads everything above its start. - let covers_from = match self.end_sequence { - Some(_) => None, - None => self - .filter - .window - .groups() - .and_then(|groups| groups.last(0, u64::MAX)) - .filter(|run| *run.end() == u64::MAX) - .map(|run| *run.start()), - }; + let coverage = self.coverage(); self.poll(waiter, |state| { let bound = state.max_age_bound(); Poll::Ready(Ok(Drift { budget: clamp_max_delay(max_delay, bound), bounded: bound.is_some_and(|bound| max_delay >= bound), edge: state.drift_edge(servable), - covers_from, + coverage, })) }) } @@ -14521,7 +14554,7 @@ mod test { budget: Duration::ZERO, bounded: false, edge: state.drift_edge(&Servable::ALL), - covers_from: None, + coverage: None, }; assert!( state.is_stale(0, &drift.edge, drift.budget), @@ -23277,7 +23310,7 @@ mod test { ); let waiter = kio::Waiter::noop(); producer.give_up(1..2, Hole::stale(budget)); - producer.supersede_stale(); + producer.supersede_stale(None); assert!( subscriber.poll_drop(&waiter).is_pending(), "no verdict under the new terms" @@ -23307,29 +23340,49 @@ mod test { ); producer.give_up(3..7, Hole::stale(budget)); assert_eq!(subscriber.poll_drop(&waiter), Poll::Ready((3..4, Hole::stale(budget)))); - producer.supersede_stale(); + producer.supersede_stale(None); assert!(subscriber.poll_drop(&waiter).is_pending(), "5 and 6 are unknown again"); } - /// A bounded reader is never settled by a received stale run, so it still waits for the - /// groups the run names rather than end. + /// A bounded reader is settled by a received stale run only when its range reaches where + /// the upstream's did; otherwise it still waits for the groups the run names. #[test] #[cfg(feature = "async")] - fn a_received_stale_run_does_not_end_a_bounded_reader() { + fn a_received_stale_run_settles_a_bounded_reader_only_to_the_judged_end() { let producer = track_producer("test", Info::timed()); producer.received(); - let mut subscriber = producer.subscribe( - Subscription::default() - .with_delay(Delay::default().with_max(Duration::from_secs(5))) - .with_ranges([Range::groups(0..2)]), - ); + let bounded = |max: Duration| { + producer.subscribe( + Subscription::default() + .with_delay(Delay::default().with_max(max)) + .with_ranges([Range::groups(0..2)]), + ) + }; + let mut wider = bounded(Duration::from_secs(5)); + let mut matching = bounded(Duration::from_millis(500)); + let waiter = kio::Waiter::noop(); + // The upstream judged its own range of 0..2 under 500ms. + producer.supersede_stale(Some(2)); producer.give_up(0..2, Hole::stale(Duration::from_millis(500))); assert!( - subscriber.poll_recv_group(&kio::Waiter::noop()).is_pending(), - "the groups are still wanted" + wider.poll_recv_group(&waiter).is_pending(), + "a wider budget still wants the groups" ); + assert_eq!( + matching.poll_drop(&waiter), + Poll::Ready((0..2, Hole::stale(Duration::from_millis(500)))) + ); + assert!( + matches!(matching.poll_recv_group(&waiter), Poll::Ready(Ok(None))), + "nothing is owed" + ); + // Judged against an open range, a capped reader may see an older edge. + producer.supersede_stale(None); + producer.give_up(0..2, Hole::stale(Duration::from_millis(500))); + let mut capped = bounded(Duration::from_millis(500)); + assert!(capped.poll_drop(&waiter).is_pending()); write_group(&producer, 0); - assert_eq!(drain_groups(&mut subscriber), [0]); + assert_eq!(drain_groups(&mut wider), [0]); } /// A report describes a miss, not a delivery: a late group is still handed out once. diff --git a/rs/moq-net/src/model/track/miss.rs b/rs/moq-net/src/model/track/miss.rs index 5771b36e009..289b19e4109 100644 --- a/rs/moq-net/src/model/track/miss.rs +++ b/rs/moq-net/src/model/track/miss.rs @@ -1101,7 +1101,7 @@ impl Misses { left.insert(hole.clone()); // Only what overlaps the hole, so judging every run costs each one the same. for (run, gone) in state.drops.overlapping(&hole) { - if drift.settles(run.start.max(hole.start), gone) { + if drift.settles(state, run.start.max(hole.start), gone) { left.remove(run.clone()); } } @@ -1460,7 +1460,7 @@ mod tests { budget, bounded: false, edge: state.drift_edge(&Servable::ALL), - covers_from: None, + coverage: None, }; let asks = misses.judge( &state, @@ -1603,7 +1603,7 @@ mod tests { budget, bounded: false, edge: state.drift_edge(&Servable::ALL), - covers_from: None, + coverage: None, }; let asks = misses.judge( &state, @@ -1677,7 +1677,7 @@ mod tests { budget, bounded: false, edge: state.drift_edge(&Servable::ALL), - covers_from: None, + coverage: None, }; assert!( misses @@ -1785,7 +1785,7 @@ mod tests { budget, bounded: false, edge: state.drift_edge(&Servable::ALL), - covers_from: None, + coverage: None, }; let full = |span| [span].into_iter().collect(); assert!( @@ -1966,7 +1966,7 @@ mod tests { budget, bounded: false, edge: state.drift_edge(&servable), - covers_from: None, + coverage: None, }; let expected = each(&misses, &state, &drift); let found = misses.expired(&state, &drift, &kio::Waiter::noop()); @@ -2080,7 +2080,7 @@ mod tests { budget, bounded: false, edge: state.drift_edge(&Servable::ALL), - covers_from: None, + coverage: None, }; let asks = misses.judge( &state, From fe27d73d53ff1d751d534418b702889b7529f9fe Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 14:29:54 -0800 Subject: [PATCH 09/68] fix(net): keep a superseded stale run unknown in JS and seed new readers' judged end A stale drop no longer counts as receipt in the JS arrivals ledger, so new terms superseding it leave its groups wanted, as in Rust. A sink added later copies the judged end with the track's runs. Co-Authored-By: Claude Opus 5.5 --- js/net/src/track.test.ts | 24 ++++++++++++++++++++++++ js/net/src/track.ts | 6 ++++-- 2 files changed, 28 insertions(+), 2 deletions(-) diff --git a/js/net/src/track.test.ts b/js/net/src/track.test.ts index 79197d08c2d..259b087017e 100644 --- a/js/net/src/track.test.ts +++ b/js/net/src/track.test.ts @@ -3924,6 +3924,30 @@ test("a received stale run settles a bounded reader only to the judged end", () producer.close(); }); +test("a superseded stale run leaves a bounded reader waiting, and a later reader judges it too", () => { + const producer = new TrackProducer("video").accept({}); + hooks.receiveTrack(producer); + const bounded = (max: number) => + producer.subscribe({ + delay: { max: Milli(max) }, + ranges: [{ start: { absolute: 0n }, end: { absolute: 2n } }], + }); + const wider = bounded(5000); + hooks.supersedeStale(producer, 2n); + producer.createGroup(2n).finish(); + hooks.dropTrackGroups(producer, 0n, 2n, staleHole(500)); + // A reader added after the verdict judges it against the same judged end. + const later = bounded(500); + expect(hooks.tryDrop(later)).toMatchObject({ start: 0n, end: 2n, stale: 500 }); + expect(hooks.tryRecvGroup(wider, false)).toEqual({ kind: "idle" }); + // New terms make the groups unknown again: still wanted, never proven delivered. + hooks.supersedeStale(producer, 2n); + expect(hooks.tryRecvGroup(wider, false)).toEqual({ kind: "idle" }); + for (const reader of [wider, later]) reader.close(); + producer.finish(); + producer.close(); +}); + test("a wide drop is held and reported as one run", () => { const producer = new TrackProducer("video").accept({}); const reader = producer.subscribe({ delay: { max: Milli(60_000) } }); diff --git a/js/net/src/track.ts b/js/net/src/track.ts index c99fe823ec5..9fcfb80afcf 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -1070,8 +1070,9 @@ export class Producer { } } } - // A pending tail owes nothing the publisher gave up. - this.#arrive(start, end); + // A pending tail owes nothing the publisher gave up. A stale verdict is not receipt: new + // terms may supersede it, leaving those groups unknown again, as in Rust. + if (hole.stale === undefined) this.#arrive(start, end); } // A publisher may re-create a group it cut; the sequence is no longer given up. @@ -1262,6 +1263,7 @@ export class Producer { this.#prune(); for (const entry of this.#cache) this.#mirror(entry, sink); sink.drops.set(this.#state.drops.peek()); + sink.judgedEnd = this.#state.judgedEnd; sink.received.set(this.#received); sink.endError = this.#state.endError; From 163a89d5289cb02cda28a0a0061e485dbf8c763f Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 14:54:39 -0800 Subject: [PATCH 10/68] fix(net): owe a reversed stale drop again in the subscriber's tail A widening past the max age a stale drop was judged under makes a lite-07 publisher re-offer or drop it again, so the subscriber's tail reopens it (Ledger::unaccount) instead of settling early, now that SUBSCRIBE_FINISH keeps the stream open for the tail to decide. Co-Authored-By: Claude Opus 5.5 --- js/net/src/ledger.test.ts | 14 ++++++ js/net/src/ledger.ts | 32 ++++++++++++++ js/net/src/lite/subscriber.ts | 24 ++++++++++- js/net/src/lite/widened_max_age.test.ts | 46 ++++++++++++++++++++ js/net/src/tail.ts | 5 +++ rs/moq-net/src/ledger.rs | 57 +++++++++++++++++++++++++ rs/moq-net/src/lite/subscriber.rs | 42 +++++++++++++++--- rs/moq-net/src/tail.rs | 6 +++ 8 files changed, 218 insertions(+), 8 deletions(-) diff --git a/js/net/src/ledger.test.ts b/js/net/src/ledger.test.ts index d413dbcad26..27db4ab6e6c 100644 --- a/js/net/src/ledger.test.ts +++ b/js/net/src/ledger.test.ts @@ -111,3 +111,17 @@ test("shuffled arrivals and bulk merges agree with the accounted sequence set", expect(ledger.hasGaps()).toBe(false); expect(ledger.covers(0n, 4128n)).toBe(true); }); + +test("unaccount reopens a gap for a fresh grace", () => { + const ledger = new Ledger(); + const grace = Milli(1000); + ledger.account(0n, 10n, at(0)); + ledger.unaccount(3n, 5n, at(1000)); + expect(ledger.covers(0n, 3n) && ledger.covers(5n, 10n)).toBe(true); + expect(ledger.covers(3n, 5n)).toBe(false); + // Its grace runs from the reversal. + ledger.expire(at(1500), grace); + expect(ledger.covers(3n, 5n)).toBe(false); + ledger.expire(at(2000), grace); + expect(ledger.covers(0n, 10n)).toBe(true); +}); diff --git a/js/net/src/ledger.ts b/js/net/src/ledger.ts index 6ed51d28ec5..fda760e859a 100644 --- a/js/net/src/ledger.ts +++ b/js/net/src/ledger.ts @@ -154,6 +154,38 @@ export class Ledger { return gaps; } + /** + * Take `[start, end)` back out of what is accounted for, owed again from `at`: a verdict that + * accounted for it was reversed. The gap it leaves opens at `at`, so it is waited for a fresh + * grace. Mirrors the Rust `Ledger::unaccount`. + */ + unaccount(start: bigint, end: bigint, at: Instant): void { + if (start >= end) return; + const lower = this.#runs.getPairOrNextLower(start)?.[1]; + const overlapping: Run[] = []; + for (const [, run] of this.#runs.entries(lower?.start ?? start)) { + if (run.start >= end) break; + if (run.end > start) overlapping.push(run); + } + for (const run of overlapping) { + this.#remove(run); + if (this.#last === run) this.#last = undefined; + if (run.start < start) { + const below = { start: run.start, end: start, since: run.since }; + this.#runs.set(below.start, below); + this.#ages.set(below, undefined); + } + if (end < run.end) { + const above = { start: end, end: run.end, since: at }; + this.#runs.set(above.start, above); + this.#ages.set(above, undefined); + } + } + const last = this.#runs.maxKey(); + this.#last = last === undefined ? undefined : this.#runs.get(last); + this.#nextExpiry(); + } + /** Restart the age of every gap reaching into `[start, end)`, which the demand newly asks for. */ demand(start: bigint, end: bigint, at: Instant): void { if (start >= end) return; diff --git a/js/net/src/lite/subscriber.ts b/js/net/src/lite/subscriber.ts index b00ed7e61fa..60d8ad74b99 100644 --- a/js/net/src/lite/subscriber.ts +++ b/js/net/src/lite/subscriber.ts @@ -160,6 +160,11 @@ interface SubscribeEntry { // The range terms id of the latest SUBSCRIBE or SUBSCRIBE_UPDATE sent: a stale drop judged // under an earlier one is no verdict on the current terms. Undefined is the SUBSCRIBE's 0. terms?: bigint; + // The max delay the latest SUBSCRIBE or SUBSCRIBE_UPDATE asked for, in milliseconds. + maxDelay?: number; + // The runs `tail` accounts for by a stale drop, with the max age each was judged under: a + // widening past it reverses the verdict, so they are owed again. + stale?: { start: bigint; end: bigint; maxAge: number }[]; // Whether SUBSCRIBE_START arrived, which `start` alone cannot say when no range starts. started?: boolean; // Whether our side is FINned, which lite-07 does once the tail settles past SUBSCRIBE_FINISH. @@ -1426,6 +1431,7 @@ export class Subscriber { if (resolvesStart(this.version)) hooks.startTrack(producer, { requested: msg.startGroup }); const entry: SubscribeEntry = { track: producer, + maxDelay: msg.maxDelay, ...framing, // The effective max delay is the stopgap grace: the wrong clock (it bounds // presentation-time drift), but it is how long the subscriber was willing to wait @@ -1858,10 +1864,17 @@ export class Subscriber { // The publisher will never send these groups, so they are accounted for without a // stream, and a reader of one still arriving ends with the drop's error now. const { start, end, maxAge, terms } = resp.drop; - entry.tail.account(start, end + 1n); // A stale verdict under terms an update has since replaced says nothing of the // current ones: the groups are unknown, and a reader that wants one fills it. - if (maxAge !== undefined && terms < (entry.terms ?? 0n)) continue; + const superseded = maxAge !== undefined && terms < (entry.terms ?? 0n); + // One the update widened past is served or dropped again, so it is still owed. + const reversed = superseded && maxAge < (entry.maxDelay ?? 0); + if (!reversed) entry.tail.account(start, end + 1n); + if (maxAge !== undefined && !reversed) { + entry.stale ??= []; + entry.stale.push({ start, end: end + 1n, maxAge }); + } + if (superseded) continue; hooks.dropTrackGroups(entry.track, start, end + 1n, { error: resp.drop.cause(this.version), stale: maxAge, @@ -2050,7 +2063,14 @@ export class Subscriber { // may move the range it was judged against, so the runs held as stale become unknown // again: a reader fills them rather than trust an old verdict. entry.terms = update.terms; + entry.maxDelay = update.maxDelay; hooks.supersedeStale(entry.track, judgedEnd(ranges)); + // A lite-07 publisher re-offers or drops again what a widening reverses, so the tail owes + // the stale runs judged under a narrower max age again. + for (const run of entry.stale ?? []) { + if (run.maxAge < update.maxDelay) entry.tail.unaccount(run.start, run.end); + } + entry.stale = entry.stale?.filter((run) => run.maxAge >= update.maxDelay); // Encoded apart from the write, so a bound the wire can't carry still fails the track. const encoded = await Writer.collect(this.version, (w) => update.encode(w, this.version)); // A failed write means the publisher stopped reading, as it does once it finishes the diff --git a/js/net/src/lite/widened_max_age.test.ts b/js/net/src/lite/widened_max_age.test.ts index 1c3c4a9df4b..36002a6edb8 100644 --- a/js/net/src/lite/widened_max_age.test.ts +++ b/js/net/src/lite/widened_max_age.test.ts @@ -116,3 +116,49 @@ for (const [version, alpn] of [ server.abort(); }); } + +test("a widening keeps a bounded lite-07 subscription open for the stale groups it re-offers", async () => { + const pair = createMockTransportPair(ALPN_07_WIP); + const origin = new OriginProducer(); + const publication = origin.createBroadcast(Path.from("room")); + publication.announce(); + const broadcast = publication.broadcast; + const producer = broadcast.createTrack("video", { timescale: Timescale.MILLI }); + const write = (sequence: number, millis: number) => { + const group = producer.createGroup(BigInt(sequence)); + group.writeFrame({ payload: new Uint8Array([sequence]), timestamp: Timestamp.fromMillis(millis) }); + group.finish(); + }; + write(0, 0); + + const [client, server] = await Promise.all([ + connect({ url, transport: pair.client }), + accept({ transport: pair.server, url, version: Version.DRAFT_07, publish: origin.consume() }), + ]); + const remote = wireOf(client).consume(Path.from("room")); + const ranges = [{ start: { absolute: 0n }, end: { absolute: 5n } }]; + const reader = remote.track("video").subscribe({ delay: { max: Milli(1500) }, ranges }); + expect(await received(reader, 1)).toEqual([0n]); + // Groups 1 and 2 are reported stale after SUBSCRIBE_START. + for (const [sequence, millis] of [ + [1, 1000], + [2, 2000], + [3, 3000], + [4, 6000], + ]) { + write(sequence, millis); + } + expect(await received(reader, 2)).toEqual([3n, 4n]); + + // The widening re-offers them, and the track's end does not settle the tail before they arrive. + reader.update({ delay: { max: Milli(10_000) }, ranges }); + producer.finish(); + expect(await received(reader, 2)).toEqual([1n, 2n]); + + reader.close(); + remote.close(); + broadcast.close(); + origin.close(); + client.abort(); + server.abort(); +}); diff --git a/js/net/src/tail.ts b/js/net/src/tail.ts index 5c6bb52be7d..a948f6f0807 100644 --- a/js/net/src/tail.ts +++ b/js/net/src/tail.ts @@ -99,6 +99,11 @@ export class Tail { this.#account(start, end); } + /** Owe `[start, end)` again: a stale drop that accounted for it was reversed by a widening. */ + unaccount(start: bigint, end: bigint): void { + this.#ledger.unaccount(start, end, now()); + } + /** Restart the age of every gap reaching into `[start, end)`, which the demand newly asks for. */ demand(start: bigint, end: bigint): void { this.#ledger.demand(start, end, now()); diff --git a/rs/moq-net/src/ledger.rs b/rs/moq-net/src/ledger.rs index 002968647b1..50736298d7e 100644 --- a/rs/moq-net/src/ledger.rs +++ b/rs/moq-net/src/ledger.rs @@ -155,6 +155,47 @@ impl Ledger { self.changes.insert(first..end); } + /// Take `groups` back out of what is accounted for, owed again from `now`: a verdict that + /// accounted for them was reversed. The gap they leave opens now, so it is waited for a + /// fresh grace. + pub fn unaccount(&mut self, groups: Range, now: Instant) { + if groups.is_empty() { + return; + } + let overlapping: Vec = self.accounted(groups.clone()).map(|run| run.start).collect(); + for clipped in overlapping { + let start = self + .runs + .range(..=clipped) + .next_back() + .map(|(&start, _)| start) + .expect("an accounted run"); + let run = self.remove(start); + if start < groups.start { + self.ages.insert((run.since, start)); + self.runs.insert( + start, + Run { + end: groups.start, + since: run.since, + }, + ); + self.changes.insert(start..groups.start); + } + if groups.end < run.end { + self.ages.insert((now, groups.end)); + self.runs.insert( + groups.end, + Run { + end: run.end, + since: now, + }, + ); + self.changes.insert(groups.end..run.end); + } + } + } + /// Account a live arrival, refreshing a cached incarnation even if its sequence was /// accounted before. Its gap ages stay unchanged, but readers must revisit a refusal. pub fn arrive(&mut self, sequence: u64, now: Instant) { @@ -454,6 +495,22 @@ mod tests { ledger.expire(lowered + GRACE); assert!(ledger.covers(1..10)); } + /// Unaccounting reopens what a reversed verdict accounted for, as a gap waited for a fresh + /// grace, and leaves the rest of the run accounted. + #[test] + fn unaccount_reopens_a_gap_for_a_fresh_grace() { + let start = Instant::now(); + let mut ledger = Ledger::new(GRACE); + arrive(&mut ledger, 0..10, start); + ledger.unaccount(3..5, start + GRACE); + assert!(ledger.covers(0..3) && ledger.covers(5..10)); + assert!(!ledger.covers(3..5)); + ledger.expire(start + GRACE * 3 / 2); + assert!(!ledger.covers(3..5), "its grace runs from the reversal"); + ledger.expire(start + GRACE * 2); + assert!(ledger.covers(0..10)); + } + #[test] fn demand_can_move_an_older_gap_after_a_newer_one() { let start = Instant::now(); diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index e133a29ed40..1438e94f073 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -19,6 +19,7 @@ use super::Version; use crate::tail::{self, Reading, Settle, Tail}; use kio::Lock; +use rangemap::RangeMap; pub(super) struct SubscriberConfig { pub runtime: crate::time::Deadlines, @@ -2466,6 +2467,7 @@ mod tests { updated: false, _working: crate::session::Work::default().start(), terms: 0, + stale: RangeMap::new(), }) } @@ -2616,6 +2618,7 @@ mod tests { updated: false, _working: crate::session::Work::default().start(), terms: 0, + stale: RangeMap::new(), }); if update { h.serve @@ -3078,6 +3081,7 @@ mod tests { updated: false, _working: crate::session::Work::default().start(), terms: 0, + stale: RangeMap::new(), }; assert_eq!( sub.owed(None), @@ -7537,6 +7541,9 @@ struct SubStream { /// The range terms id of the latest SUBSCRIBE or SUBSCRIBE_UPDATE sent: a stale drop /// judged under an earlier one is no verdict on the current terms. terms: u64, + /// The runs `tail` accounts for by a stale drop, with the max age each was judged under: + /// a widening past it reverses the verdict, so they are owed again. + stale: RangeMap, /// Holds the session busy for as long as the subscription is open. _working: crate::session::Working, } @@ -8076,7 +8083,7 @@ impl TrackServe { if start_moved { let _ = producer.start_at(active.start); } - buffer_update(active, producer)?; + buffer_update(active, producer, self.subscriber.runtime.now())?; } Ok(Begin::None) } @@ -8348,9 +8355,26 @@ fn judged_end(ranges: &[lite::Range]) -> Option { /// A stale drop judged under earlier terms may still be in flight, and the update may have /// moved the range it was judged against, so the runs `serving` holds as stale become /// unknown again: a reader fills them by range rather than trust an old verdict. -fn buffer_update(active: &mut SubStream, serving: &track::Producer) -> Result<(), Error> { +/// +/// A lite-07 publisher re-offers or drops again what a widening reverses, so the tail owes +/// the stale runs judged under a narrower max age than this update's again, from `now`. +fn buffer_update(active: &mut SubStream, serving: &track::Producer, now: crate::time::Instant) -> Result<(), Error> { active.terms += 1; serving.supersede_stale(judged_end(&active.ranges)); + let reversed: Vec<_> = active + .stale + .iter() + .filter(|(_, judged)| **judged < active.max_delay) + .map(|(run, _)| run.clone()) + .collect(); + if let Ok(mut tail) = active.tail.write() { + for run in &reversed { + tail.unaccount(run.clone(), now); + } + } + for run in reversed { + active.stale.remove(run); + } active.stream.writer.buffer(&lite::SubscribeUpdate { wanted_group: active.wanted, priority: active.priority, @@ -8458,6 +8482,7 @@ impl Establish { wanted: None, updated: false, terms: 0, + stale: RangeMap::new(), _working: self.session.work().start(), } } @@ -9464,12 +9489,17 @@ impl ServeLoop { // one still arriving ends with the drop's error now. lite::SubscribeResponse::Drop(dropped) => { let groups = dropped.start..dropped.end.saturating_add(1); - if let Ok(mut tail) = active.tail.write() { - tail.account(groups.clone(), serve.subscriber.runtime.now()); - } // A stale verdict under terms an update has since replaced says nothing of the // current ones: the groups are unknown, and a reader that wants one fills it by range. let superseded = dropped.terms < active.terms; + // One the update widened past is served or dropped again, so it is still owed. + let reversed = superseded && dropped.max_age.is_some_and(|judged| judged < active.max_delay); + if !reversed && let Ok(mut tail) = active.tail.write() { + tail.account(groups.clone(), serve.subscriber.runtime.now()); + } + if let Some(judged) = dropped.max_age.filter(|_| !reversed) { + active.stale.insert(groups.clone(), judged); + } match dropped.max_age { Some(_) if superseded => {} stale => self.serving.give_up( @@ -9711,7 +9741,7 @@ impl ServeLoop { && active.wanted != Some(wanted) { active.wanted = Some(wanted); - if let Err(err) = buffer_update(active, &self.serving) { + if let Err(err) = buffer_update(active, &self.serving, serve.subscriber.runtime.now()) { return Poll::Ready(ServeEnd::GiveBack(err)); } continue; diff --git a/rs/moq-net/src/tail.rs b/rs/moq-net/src/tail.rs index 4eeb6532792..d6c36b39f84 100644 --- a/rs/moq-net/src/tail.rs +++ b/rs/moq-net/src/tail.rs @@ -71,6 +71,12 @@ impl Tail { self.expire(now); } + /// Owe `groups` again from `now`: a stale drop that accounted for them was reversed by a + /// widening, so the publisher serves or drops them anew. + pub fn unaccount(&mut self, groups: Range, now: Instant) { + self.ledger.unaccount(groups, now); + } + /// Restart the age of every gap reaching into `groups`, which the demand newly asks for. pub fn demand(&mut self, groups: Range, now: Instant) { self.ledger.demand(groups, now); From 81cef057e88ff89c6daa5cd704d79ff8a4d6eacb Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 15:04:54 -0800 Subject: [PATCH 11/68] fix(net): coalesce the JS subscriber's stale runs A peer repeating a stale drop now holds one run rather than one record per message, as Rust's range map does. Co-Authored-By: Claude Opus 5.5 --- js/net/src/lite/subscriber.ts | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/js/net/src/lite/subscriber.ts b/js/net/src/lite/subscriber.ts index 60d8ad74b99..d18d10e8392 100644 --- a/js/net/src/lite/subscriber.ts +++ b/js/net/src/lite/subscriber.ts @@ -23,14 +23,17 @@ import * as netGroup from "../group.ts"; import { Cost, type Hop, MAX_HOPS, type Route, routesEqual, UNKNOWN_HOP } from "../hop.ts"; import * as Instance from "../instance.ts"; import { + type DropRun, hiddenBelow, hooks, + insertDrop, maxDelayOf, minDelayOf, scopeCaptures, scopeHead, scopeOverlaps, skipHole, + staleHole, } from "../internal.ts"; import type { Lister } from "../listing.ts"; import * as Path from "../path.ts"; @@ -164,7 +167,8 @@ interface SubscribeEntry { maxDelay?: number; // The runs `tail` accounts for by a stale drop, with the max age each was judged under: a // widening past it reverses the verdict, so they are owed again. - stale?: { start: bigint; end: bigint; maxAge: number }[]; + // Coalesced, so a peer repeating a drop holds one run, not one per message. + stale?: DropRun[]; // Whether SUBSCRIBE_START arrived, which `start` alone cannot say when no range starts. started?: boolean; // Whether our side is FINned, which lite-07 does once the tail settles past SUBSCRIBE_FINISH. @@ -1870,10 +1874,8 @@ export class Subscriber { // One the update widened past is served or dropped again, so it is still owed. const reversed = superseded && maxAge < (entry.maxDelay ?? 0); if (!reversed) entry.tail.account(start, end + 1n); - if (maxAge !== undefined && !reversed) { - entry.stale ??= []; - entry.stale.push({ start, end: end + 1n, maxAge }); - } + if (maxAge !== undefined && !reversed) + entry.stale = insertDrop(entry.stale ?? [], start, end + 1n, staleHole(maxAge)); if (superseded) continue; hooks.dropTrackGroups(entry.track, start, end + 1n, { error: resp.drop.cause(this.version), @@ -2068,9 +2070,9 @@ export class Subscriber { // A lite-07 publisher re-offers or drops again what a widening reverses, so the tail owes // the stale runs judged under a narrower max age again. for (const run of entry.stale ?? []) { - if (run.maxAge < update.maxDelay) entry.tail.unaccount(run.start, run.end); + if ((run.stale ?? 0) < update.maxDelay) entry.tail.unaccount(run.start, run.end); } - entry.stale = entry.stale?.filter((run) => run.maxAge >= update.maxDelay); + entry.stale = entry.stale?.filter((run) => (run.stale ?? 0) >= update.maxDelay); // Encoded apart from the write, so a bound the wire can't carry still fails the track. const encoded = await Writer.collect(this.version, (w) => update.encode(w, this.version)); // A failed write means the publisher stopped reading, as it does once it finishes the From 67618139622bb30bad3ff959e81ed9a6c446ac8d Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 16:31:06 -0800 Subject: [PATCH 12/68] feat(net)!: lite-07 DROP names its Update; any update reopens stale drops Maintainer decisions of 2026-11-03 (/quest-complete): - Rename `Terms` to `Update` on lite-07 SUBSCRIBE, SUBSCRIBE_UPDATE and SUBSCRIBE_DROP, in the draft, Rust and JS. - Drop `Max Age` from SUBSCRIBE_DROP. A stale (OLD) drop is judged under the budget the subscriber sent with the update it names; one under a superseded update is no verdict. Any SUBSCRIBE_UPDATE reopens every outstanding OLD drop: the publisher re-offers what the new terms cover and drops again under the new number what is still stale, including a group whose stream already expired. Non-OLD drops are final. - A local producer's create_group skips the gap above its last group at once; a run already given up keeps its reason, and a late group still wins over its skip. - track::Producer::drop_groups is crate-private; moq-archive's replay skips what it holds nothing for up to a recorded end. Also fixes review findings: Misses::retry re-judges a retried range, the JS cursor clips received stale runs to its span, and the resume reader withdraws a queued received stale run new terms superseded. Co-Authored-By: Claude Opus 5.5 --- drafts/draft-lcurley-moq-lite.md | 50 ++--- js/net/src/internal.ts | 32 ++- js/net/src/lite/publisher.test.ts | 2 + js/net/src/lite/publisher.ts | 58 +++--- js/net/src/lite/subscribe.test.ts | 27 ++- js/net/src/lite/subscribe.ts | 89 +++----- js/net/src/lite/subscriber.ts | 63 +++--- js/net/src/lite/widened_max_age.test.ts | 50 +++++ js/net/src/resume.test.ts | 40 +++- js/net/src/resume.ts | 13 +- js/net/src/track.test.ts | 63 ++++++ js/net/src/track.ts | 97 +++++---- quest/m1/drop-set.md | 52 ++++- quest/m1/lite07-finalize.md | 4 +- quest/m1/time-addressing/model.md | 8 + quest/m2/ietf-hole-marker.md | 9 +- rs/moq-archive/src/consumer/tests.rs | 2 +- rs/moq-archive/src/consumer/track.rs | 7 +- rs/moq-net/src/fuzz.rs | 8 +- rs/moq-net/src/lite/publisher.rs | 247 ++++++++++++---------- rs/moq-net/src/lite/subscribe.rs | 145 +++++-------- rs/moq-net/src/lite/subscriber.rs | 232 ++++++++++++++++----- rs/moq-net/src/lite/version.rs | 6 +- rs/moq-net/src/model/resume.rs | 72 ++++++- rs/moq-net/src/model/resume_model.rs | 2 + rs/moq-net/src/model/skipped.rs | 13 ++ rs/moq-net/src/model/track.rs | 260 ++++++++++++++++-------- rs/moq-net/src/model/track/miss.rs | 63 ++++-- 28 files changed, 1117 insertions(+), 597 deletions(-) diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index a9c1f98df23..e55d2787232 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -335,9 +335,9 @@ Sent when resetting a stream (RESET_STREAM), or when refusing to receive one (ST | ------- | ------------- | ----------- | | 0x32 | GROUP_TOO_LARGE | The group grew past the publisher's cache budget and was aborted. | | ------- | ------------- | ----------- | -| 0x33 | NOT_FOUND | The requested group, track, or broadcast is not here. A sequence that never held a group is named by SUBSCRIBE_SKIP instead. | +| 0x33 | NOT_FOUND | The requested group, track, or broadcast is not here. A sequence its publisher never produced is named by SUBSCRIBE_SKIP instead; a relay that cannot tell whether its source produced one names it with NOT_FOUND. | | ------- | ------------- | ----------- | -| 0x34 | OLD | The group was superseded by a newer group and dropped. In SUBSCRIBE_DROP with a `Max Age`, a widening past that budget re-offers it. | +| 0x34 | OLD | The group was superseded by a newer group and dropped. In SUBSCRIBE_DROP, the next SUBSCRIBE_UPDATE reopens it: the publisher re-offers the group or drops it again. | | ------- | ------------- | ----------- | | 0x35 | EVICTED | The group was dropped under memory pressure. Unlike OLD it was still current, so it can be re-fetched. | | ------- | ------------- | ----------- | @@ -1251,7 +1251,7 @@ SUBSCRIBE Message { Subscriber Min Age (i) Range Count (i) Range (..) ... - Terms (i) + Update (i) } Range { @@ -1315,9 +1315,9 @@ Which fields follow, and what the Anchor names: Any other value is a PROTOCOL_VIOLATION. The publisher resolves a Relative or Time Anchor against the Track as it holds it. -**Terms**: -An identifier for the subscription's terms: its Ranges and `Subscriber Max Age`. -Each SUBSCRIBE_UPDATE carries a larger value than the one before it, and a stale [SUBSCRIBE_DROP](#subscribe-drop) echoes the value it was judged under, so the subscriber can tell a verdict on its current terms from one an update has since replaced. +**Update**: +The number of the subscription's terms this message sets: 0 on SUBSCRIBE, and on each SUBSCRIBE_UPDATE a larger value than the one before it. +A [SUBSCRIBE_DROP](#subscribe-drop) echoes the number in effect when its groups were given up, so the subscriber can tell a stale verdict on its current terms from one an update has since replaced. A `Start` is a floor, not a request; `Subscriber Max Age` is the only thing that asks for data. The publisher SHOULD start at the oldest group at or above the floor that [Expiration](#expiration) has not expired, and MUST NOT deliver an older one. @@ -1335,7 +1335,9 @@ A subscriber can modify a subscription with a SUBSCRIBE_UPDATE message. A subscriber MAY send multiple SUBSCRIBE_UPDATE messages to update the subscription. Its Ranges replace the previous ones, growing or shrinking the subscription. Lowering a `Start` covers the groups below the old one that the publisher still holds, not only those that arrive later; a group already sent is not sent again. -Growing the subscription asks for what it newly covers: the publisher MUST serve the newly covered Groups it holds and never delivered on this subscription, including Groups it named in a stale SUBSCRIBE_DROP that the new terms no longer convict, and MUST name again, under the new `Terms`, those it cannot serve. +Growing the subscription asks for what it newly covers: the publisher MUST serve the newly covered Groups it holds and never delivered on this subscription. +Every SUBSCRIBE_UPDATE, whatever it changes, reopens every stale (OLD) SUBSCRIBE_DROP sent before it: the publisher MUST serve again the Groups it named in one that it still holds and the new terms cover, and MUST name again, in a SUBSCRIBE_DROP carrying the new `Update`, those still stale. +A SUBSCRIBE_DROP with any other Error Code is final, and no update reopens it. A SUBSCRIBE_UPDATE never resets the subscription, since a reset would race Group Streams already in flight, and shrinking it sends no SUBSCRIBE_SKIP or SUBSCRIBE_DROP. ~~~ @@ -1347,13 +1349,13 @@ SUBSCRIBE_UPDATE Message { Subscriber Min Age (i) Range Count (i) Range (..) ... - Terms (i) + Update (i) Wanted Group (i) } ~~~ See [SUBSCRIBE](#subscribe) for information about each field. -A `Terms` no larger than the previous one only confuses the subscriber's own accounting of stale drops; the publisher echoes the latest value it read. +An `Update` no larger than the previous one only confuses the subscriber's own accounting of stale drops; the publisher echoes the latest value it read. Moving a `Start` forward within a group that is already being delivered does not retract frames the publisher has already sent; it only bounds what the publisher sends from here on. @@ -1568,6 +1570,7 @@ The publisher SHOULD send SUBSCRIBE_FINISH as soon as the exclusive final group A publisher names groups that existed but that it will not deliver with SUBSCRIBE_DROP on the Subscribe Stream: an aborted or reset group, one evicted or expired with no copy left to serve it, or one past the subscriber's budget (OLD). Sequences the publisher never produced are named by [SUBSCRIBE_SKIP](#subscribe-skip) instead. +A relay cannot tell a sequence its source never produced from one it never received, so it names a sequence it never got in SUBSCRIBE_DROP with NOT_FOUND. ~~~ SUBSCRIBE_DROP Message { @@ -1576,27 +1579,27 @@ SUBSCRIBE_DROP Message { Group Start (i) Group End (i) Error Code (i) - Max Age (i) - Terms (i) + Update (i) } ~~~ **Group Start** and **Group End** are the inclusive range of group sequences the publisher gave up. `Group Start = Group End + 1` is an empty range; a larger Group Start is a PROTOCOL_VIOLATION. -**Error Code** is a [stream error code](#error-codes) saying why the groups were given up, as a reset retaining the Group Stream header would: for example OLD for one past the subscriber's budget, or EVICTED for one the publisher no longer holds. +**Error Code** is a [stream error code](#error-codes) saying why the groups were given up, as a reset retaining the Group Stream header would: for example OLD for one past the subscriber's budget, EVICTED for one the publisher no longer holds, or NOT_FOUND for one a relay never received. A subscriber MUST end the readers of a dropped group it holds with that error, including a group whose stream is still arriving, rather than end it cleanly. -**Max Age**: -The `Subscriber Max Age` a staleness verdict was judged under, in milliseconds plus one, so a verdict on a budget of zero is 1. -0 means the drop does not depend on the subscriber's budget, such as an aborted group or one evicted or expired with no copy left, and no SUBSCRIBE_UPDATE reverses it. +**Update**: +The [SUBSCRIBE](#subscribe) or [SUBSCRIBE_UPDATE](#subscribe-update) `Update` in effect when the publisher gave the groups up. -**Terms**: -The [SUBSCRIBE](#subscribe) `Terms` in effect when the publisher judged the drop. -A stale drop holds only under those terms: a subscriber treats one whose `Terms` an update has since replaced as no verdict, since the update may have moved the Range the verdict was judged against, and a relay asks for such a group again rather than drop it for its own readers. -A relay that received a stale drop settles with it only a reader whose own budget is no wider than its `Max Age` and whose Ranges cover the relay's own, which the verdict was judged against. +A drop whose Error Code is OLD is a stale verdict, which holds only under the update it names. +The subscriber judges it against the `Subscriber Max Age` it sent with that update, from its own record: the drop does not carry the budget. +A stale drop whose `Update` a later update has replaced is no verdict, since the update may have moved the Range or budget it was judged against: the subscriber still owes those groups, and a relay asks for such a group again rather than drop it for its own readers. +The next SUBSCRIBE_UPDATE reopens every stale drop the subscriber accounted for, which it owes again until the publisher serves the group or drops it again under the new `Update`, as described in [SUBSCRIBE_UPDATE](#subscribe-update). +A relay that received a stale drop settles with it only a reader whose own budget is no wider than the one the relay sent with that update and whose Ranges cover the relay's own, which the verdict was judged against. + +A drop with any other Error Code is final, whatever its `Update`. -A SUBSCRIBE_UPDATE that widens the terms re-offers the stale drops they no longer convict, as described in [SUBSCRIBE_UPDATE](#subscribe-update). A Group Stream for a sequence the subscriber already saw dropped, as after such a re-offer, wins over the drop: the subscriber delivers it. For every sequence in the subscription's range, the publisher MUST send a Group Stream, send a datagram, or name it in SUBSCRIBE_SKIP or SUBSCRIBE_DROP. @@ -1605,7 +1608,7 @@ A reliably delivered reset retaining the Group Stream header accounts for that g The publisher SHOULD name a sequence as soon as it gives it up, without waiting for SUBSCRIBE_FINISH. A datagram counts as sent even if it is lost; a subscriber SHOULD bound its wait for unaccounted sequences, for example by Subscriber Max Age. -Before moq-lite-07, SUBSCRIBE_DROP carries neither `Max Age` nor `Terms`, its Error Code is application-specific, and it also names the sequences moq-lite-07 names in SUBSCRIBE_SKIP; every such drop is final. +Before moq-lite-07, SUBSCRIBE_DROP carries no `Update`, its Error Code is application-specific, and it also names the sequences moq-lite-07 names in SUBSCRIBE_SKIP; every such drop is final. ## SUBSCRIBE_SKIP {#subscribe-skip} @@ -1625,7 +1628,8 @@ SUBSCRIBE_SKIP Message { A skip is final: no Group will ever exist at those sequences, whatever the terms. It carries no error and ends no reader, since there is nothing to read; a subscriber stops waiting for those sequences. -A relay forwards it unchanged. +A publisher SHOULD name a gap it leaves in its own sequence numbers as soon as it creates the Group above it, rather than judge the gap stale later. +A relay forwards it unchanged; a relay that cannot tell whether its source produced a sequence uses SUBSCRIBE_DROP instead (see [SUBSCRIBE_DROP](#subscribe-drop)). ## FETCH FETCH is sent by a subscriber to request a single group from a track. @@ -1952,7 +1956,7 @@ The `Message Length` describes the payload size on the wire. - SUBSCRIBE_DROP's Error Code is a stream error code, which ends the dropped groups' readers, including one still arriving. - Refuse a SUBSCRIBE_DROP whose Group Start is greater than Group End plus one on every version that carries it. - Added SUBSCRIBE_SKIP (type 0x3) for sequences the publisher never produced, which ends no reader and is final. SUBSCRIBE_DROP now names only groups that existed. -- Added `Max Age` and `Terms` to SUBSCRIBE_DROP, and `Terms` to SUBSCRIBE and SUBSCRIBE_UPDATE: a stale drop names the budget and the terms it was judged under, holds only under them, and a widening re-offers it. A Group Stream wins over its earlier drop. +- Added `Update` to SUBSCRIBE, SUBSCRIBE_UPDATE, and SUBSCRIBE_DROP: the number of the terms in effect. A stale (OLD) drop holds only under the update it names, judged against the budget the subscriber sent with it; every later SUBSCRIBE_UPDATE reopens it, and the publisher re-offers the group or drops it again under the new `Update`. Any other drop is final. A Group Stream wins over its earlier drop. - SUBSCRIBE_FINISH declares the final boundary while lower groups can still be in flight, and no longer closes the Subscribe Stream: the publisher keeps reading SUBSCRIBE_UPDATE, so an update can reach an ended Track's history. - A covered subscription, whose every Range has an End and whose every Group was sent or dropped, also stays open for an update; the publisher never FINs first. - A publisher that withheld SUBSCRIBE_OK sends it, then the drops it withheld, once no group can resolve the start: every Group of a bounded subscription was given up, or the Track finished. diff --git a/js/net/src/internal.ts b/js/net/src/internal.ts index a0d4d2c2a0f..5d3c47ab935 100644 --- a/js/net/src/internal.ts +++ b/js/net/src/internal.ts @@ -642,18 +642,12 @@ export const hooks: { */ unreport: (subscriber: Subscriber, start: bigint, end: bigint) => void; /** - * The budget in milliseconds a subscriber judges staleness under: its max delay, clamped to the - * track's max age. Mirrors the Rust crate-private `track::Subscriber::budget`. + * Re-offer what a subscriber gave up as stale through dispositions, as a wire whose every update + * reopens its stale drops does: each run is judged again under the new terms. Returns the runs + * taken back, which are owed again. Mirrors the Rust crate-private + * `track::Subscriber::reoffer_stale`. */ - budget: (subscriber: Subscriber) => number; - /** - * Have a widening re-offer what a subscriber reported stale through dispositions, as a wire - * whose drops a widening reverses does. Returns the subscriber. Mirrors the Rust - * crate-private `track::Subscriber::reoffers_reported`. - */ - reoffersReported: (subscriber: Subscriber) => Subscriber; - /** The runs a subscriber reported stale that a widening took back since the last call. */ - takeUnreported: (subscriber: Subscriber) => Run[]; + reofferStale: (subscriber: Subscriber) => Run[]; /** * Whether a subscriber reads a copy received from a remote publisher, which cannot tell a group * its source never produced from one it never received. A resumed reader always does. @@ -667,6 +661,11 @@ export const hooks: { holdsGroup: (subscriber: Subscriber, sequence: bigint) => boolean; /** The error the subscriber's track gave `sequence` up with, if it did, and what changes it. */ givenUp: (subscriber: Subscriber, sequence: bigint) => { error?: Error; changed: Getter }; + /** + * The parts of `[start, end)` the subscriber's track still holds as a received stale run: one + * new terms superseded is gone. Mirrors the Rust crate-private `track::Subscriber::stale_held`. + */ + staleHeld: (subscriber: Subscriber, start: bigint, end: bigint) => Run[]; /** Whether a group may still land on the live feed of the subscriber's track; see {@link Land}. */ mayLand: (subscriber: Subscriber, land: Land) => boolean; /** A subscriber's largest location now, and whether it can still advance. */ @@ -872,13 +871,7 @@ export const hooks: { supersedeStale: () => { throw new Error("track.ts not loaded"); }, - budget: () => { - throw new Error("track.ts not loaded"); - }, - reoffersReported: () => { - throw new Error("track.ts not loaded"); - }, - takeUnreported: () => { + reofferStale: () => { throw new Error("track.ts not loaded"); }, isReceived: () => { @@ -917,6 +910,9 @@ export const hooks: { holdsGroup: () => { throw new Error("track.ts not loaded"); }, + staleHeld: () => { + throw new Error("track.ts not loaded"); + }, givenUp: () => { throw new Error("track.ts not loaded"); }, diff --git a/js/net/src/lite/publisher.test.ts b/js/net/src/lite/publisher.test.ts index 24f28a9088b..9de90901604 100644 --- a/js/net/src/lite/publisher.test.ts +++ b/js/net/src/lite/publisher.test.ts @@ -935,6 +935,8 @@ test("lite-07 serves a time range from the on-demand track its lookup resolved", // deliver it; a sequence cursor would skip it permanently. test("lite draft-05: a late-arriving older group is still served", async () => { const sub = await servedSubscription({ startGroup: 1n }); + // A received copy: a local publisher's gap is skipped at once, so its late group is not. + hooks.receiveTrack(sub.track); try { // Serve one at a time so arrival order at the publisher is the order given. sub.serve(1n); diff --git a/js/net/src/lite/publisher.ts b/js/net/src/lite/publisher.ts index d3ff44d1027..e090654184c 100644 --- a/js/net/src/lite/publisher.ts +++ b/js/net/src/lite/publisher.ts @@ -955,8 +955,6 @@ export class Publisher { } // Parked past a bounded window's end: the peer reads this stream's end as the track's. const track = hooks.parked(consumer.subscribe(subscription(ranges))); - // A lite-07 widening re-offers what was reported stale; earlier versions' drops are final. - if (hasSkip(this.version)) hooks.reoffersReported(track); serving(track); if (timed) { try { @@ -1084,7 +1082,7 @@ export class Publisher { served, floor, bounds, - terms: msg.terms, + update: msg.update, }); console.debug(`publish done: broadcast=${msg.broadcast} track=${track.name}`); @@ -1250,12 +1248,12 @@ export class Publisher { // The read cursor's floor; see {@link cursorFloor}. floor: bigint; bounds: FrameBounds; - /** The SUBSCRIBE's range terms id, which a stale drop echoes until an update replaces it. */ - terms: bigint; + /** The SUBSCRIBE's update number (0), which a drop echoes until an update replaces it. */ + update: bigint; }, ): Promise { const { sub, broadcast, stamp, publisher, compress, served, bounds } = serving; - let terms = serving.terms; + let updateNumber = serving.update; // The read cursor's floor; the subscription's ranges cap it. let floor = serving.floor; const moveCursor = () => track.withRange({ start: { included: floor } }); @@ -1283,7 +1281,7 @@ export class Publisher { const holeMessage = (start: bigint, end: bigint, hole: Hole): SubscribeResponse => hole.skip && hasSkip(this.version) ? { skip: new SubscribeSkip({ start, end }) } - : { drop: SubscribeDrop.given(start, end, hole.error, this.version, { maxAge: hole.stale, terms }) }; + : { drop: SubscribeDrop.given(start, end, hole.error, this.version, updateNumber) }; // Give up, as `hole` says, whatever part of `[start, end)` this subscription has not been // served or given up yet. @@ -1344,7 +1342,11 @@ export class Publisher { // Every group this subscription started serving, until its stream finishes or resets. const groups = new Set>(); const drops = new Signal(0); - const pendingDrops: SubscribeDrop[] = []; + // Groups whose stream failed, with why, for the control stream to name. + const pendingDrops: { sequence: bigint; error: Error }[] = []; + // Lite-07 groups whose stream expired as stale, which every update drops again under its + // number: the subscriber reopens each stale drop then, and the stream already reset. + const expired: Run[] = []; // Cancels groups still queued for a stream slot. Only the subscriber leaving counts: // a track that ran out of groups still has to flush the ones already queued, and the @@ -1366,7 +1368,11 @@ export class Publisher { skipped.length = 0; } if (pendingDrops.length > 0) { - for (const dropped of pendingDrops.splice(0)) { + for (const { sequence, error } of pendingDrops.splice(0)) { + // A stream expired under this subscription's budget is a stale verdict. + if (hasSkip(this.version) && toStreamCode(error) === StreamCode.Old) + insertRun(expired, sequence, sequence + 1n); + const dropped = SubscribeDrop.given(sequence, sequence, error, this.version, updateNumber); if ( !(await controls.response(encodeSubscribeResponse(stream, { drop: dropped }, this.version))) ) @@ -1392,7 +1398,22 @@ export class Publisher { // An update may reach the finished track's history, so it is served again. finished = false; const update = control.update; - terms = update.terms; + updateNumber = update.update; + // Lite-07 reopens every stale drop on any update: what was reported stale is + // offered again, and dropped again under this update's number if still stale. + if (hasSkip(this.version)) { + const reopened = hooks.reofferStale(track); + if (served) { + for (const run of reopened) + served.splice(0, served.length, ...removeRun(served, run.start, run.end)); + } + // A stream already reset as stale is not served again: dropped again. + for (const run of expired) { + for (let sequence = run.start; sequence < run.end; sequence++) + pendingDrops.push({ sequence, error: new StreamError(StreamCode.Old) }); + } + expired.length = 0; + } console.debug(`subscribe update: broadcast=${broadcast} track=${track.name}`); // `cursorFloor` pins an unfloored pre-06 SUBSCRIBE to the latest // group, so an update that drops a named floor re-pins there too. On @@ -1446,12 +1467,6 @@ export class Publisher { ? ({ kind: "group", group: first.group } as const) : hooks.tryRecvGroup(track, emitRange); first = undefined; - // A stale drop a lite-07 widening reversed is owed again: re-offered if still held, - // and dropped again under the new terms otherwise. - if (served) { - for (const run of hooks.takeUnreported(track)) - served.splice(0, served.length, ...removeRun(served, run.start, run.end)); - } switch (recv.kind) { case "error": throw recv.error; @@ -1604,7 +1619,7 @@ export class Publisher { const group = recv.group; // Never a second stream for a sequence this subscription already accounted for // (served, sent as a datagram, or dropped): before lite-07 a subscriber settles a - // drop for good. A stale drop a lite-07 widening reversed is owed again above. + // drop for good. A stale drop a lite-07 update reopened is owed again (see the update above). if (served && runGaps(served, group.sequence, group.sequence + 1n).length === 0) { group.close(); continue; @@ -1665,14 +1680,7 @@ export class Publisher { const task = opened.then(async (writer) => { const failed = writer instanceof Error ? writer : await this.#serveGroup(writer, options); if (emitRange && failed instanceof Error) { - // A stream expired under this subscription's budget is a stale verdict. - const maxAge = toStreamCode(failed) === StreamCode.Old ? hooks.budget(track) : undefined; - pendingDrops.push( - SubscribeDrop.given(group.sequence, group.sequence, failed, this.version, { - maxAge, - terms, - }), - ); + pendingDrops.push({ sequence: group.sequence, error: failed }); drops.update((revision) => revision + 1); } }); diff --git a/js/net/src/lite/subscribe.test.ts b/js/net/src/lite/subscribe.test.ts index bb0c4895516..22a33dae494 100644 --- a/js/net/src/lite/subscribe.test.ts +++ b/js/net/src/lite/subscribe.test.ts @@ -196,7 +196,7 @@ test("lite-07 carries the span as its only range, exclusive at the end", async ( test("a lite-07 list decodes as the span covering it", async () => { // Priority, order, max delay, and an unstated min delay as SubscribeUpdate writes them, - // then the ranges, the terms, and an absent wanted group. + // then the ranges, the update number, and an absent wanted group. const head = (await encodeMessage(Version.DRAFT_07, new SubscribeUpdate({ priority: 0 }))).slice(1, 5); const update = (ranges: number[]) => { const body = [...head, ...ranges, 0, 0]; @@ -328,25 +328,24 @@ test("SubscribeEnd carries its error on draft-07", async () => { test("SubscribeDrop is restored on draft-07", async () => { const drop: SubscribeResponse = { drop: new SubscribeDrop({ start: 1n, end: 3n, error: 7 }) }; const wire = await encode(Version.DRAFT_07, drop); - // A drop no budget reverses has a Max Age of 0. - expect(wire).toEqual(new Uint8Array([2, 5, 1, 3, 7, 0, 0])); + // The drop names the SUBSCRIBE's update, 0. + expect(wire).toEqual(new Uint8Array([2, 4, 1, 3, 7, 0])); const got = await responseRoundtrip(Version.DRAFT_07, drop); if (!("drop" in got)) throw new Error("expected drop"); expect([got.drop.start, got.drop.end, got.drop.error]).toEqual([1n, 3n, 7]); }); -test("a stale drop names its max age and terms on draft-07 alone", async () => { - for (const maxAge of [0, 300]) { - const drop = new SubscribeDrop({ start: 4n, end: 4n, error: 0x34, maxAge, terms: 2n }); - const got = await responseRoundtrip(Version.DRAFT_07, { drop }); - if (!("drop" in got)) throw new Error("expected drop"); - expect([got.drop.maxAge, got.drop.terms]).toEqual([maxAge, 2n]); - } - const drop = new SubscribeDrop({ start: 4n, end: 4n, error: 0, maxAge: 300, terms: 2n }); - expect(await encode(Version.DRAFT_06, { drop })).toEqual(new Uint8Array([2, 3, 4, 4, 0])); - const got = await responseRoundtrip(Version.DRAFT_06, { drop }); +test("a drop names its update on draft-07 alone", async () => { + const drop = new SubscribeDrop({ start: 4n, end: 4n, error: 0x34, update: 2n }); + expect(await encode(Version.DRAFT_07, { drop })).toEqual(new Uint8Array([2, 4, 4, 4, 0x34, 2])); + const got = await responseRoundtrip(Version.DRAFT_07, { drop }); if (!("drop" in got)) throw new Error("expected drop"); - expect([got.drop.maxAge, got.drop.terms]).toEqual([undefined, 0n]); + expect(got.drop.update).toBe(2n); + const old = new SubscribeDrop({ start: 4n, end: 4n, error: 0, update: 2n }); + expect(await encode(Version.DRAFT_06, { drop: old })).toEqual(new Uint8Array([2, 3, 4, 4, 0])); + const back = await responseRoundtrip(Version.DRAFT_06, { drop: old }); + if (!("drop" in back)) throw new Error("expected drop"); + expect(back.drop.update).toBe(0n); }); test("SubscribeSkip is type 3 on draft-07 alone", async () => { diff --git a/js/net/src/lite/subscribe.ts b/js/net/src/lite/subscribe.ts index 6000d0628ea..9a4ede30562 100644 --- a/js/net/src/lite/subscribe.ts +++ b/js/net/src/lite/subscribe.ts @@ -28,14 +28,14 @@ import { Version, } from "./version.ts"; -/** Decode the range `Terms` after the ranges. A wire without the field decodes as 0. */ -async function decodeTerms(r: Reader, version: Version): Promise { +/** Decode the `Update` number after the ranges. A wire without the field decodes as 0. */ +async function decodeUpdate(r: Reader, version: Version): Promise { return hasSkip(version) ? await r.u62() : 0n; } -/** Encode the range `Terms`, dropping it on a wire without the field. */ -async function encodeTerms(w: Writer, version: Version, terms: bigint) { - if (hasSkip(version)) await w.u62(terms); +/** Encode the `Update` number, dropping it on a wire without the field. */ +async function encodeUpdate(w: Writer, version: Version, update: bigint) { + if (hasSkip(version)) await w.u62(update); } /** @@ -441,8 +441,8 @@ export class SubscribeUpdate { endFrame?: number; /** See {@link Subscribe.ranges}. */ ranges?: WireRange[]; - /** See {@link Subscribe.terms}: larger than the subscription's previous terms. */ - terms: bigint; + /** See {@link Subscribe.update}: larger than the subscription's previous one. */ + update: bigint; constructor(props: { priority: number; @@ -455,7 +455,7 @@ export class SubscribeUpdate { startFrame?: number; endFrame?: number; ranges?: WireRange[]; - terms?: bigint; + update?: bigint; }) { if (props.wantedGroup !== undefined && props.wantedGroup < 0n) throw new RangeError("wanted group must be non-negative"); @@ -469,7 +469,7 @@ export class SubscribeUpdate { this.startFrame = props.startFrame ?? 0; this.endFrame = props.endFrame; this.ranges = props.ranges; - this.terms = props.terms ?? 0n; + this.update = props.update ?? 0n; } async #encode(w: Writer, version: Version) { @@ -484,7 +484,7 @@ export class SubscribeUpdate { await w.u53(this.maxDelay); await encodeMinDelay(w, version, this.minDelay); await encodeSpan(w, version, this, this.ranges); - await encodeTerms(w, version, this.terms); + await encodeUpdate(w, version, this.update); if (hasRefresh(version)) await w.u62(this.wantedGroup === undefined ? 0n : this.wantedGroup + 1n); break; } @@ -501,7 +501,7 @@ export class SubscribeUpdate { const maxDelay = await r.u53(); const minDelay = await decodeMinDelay(r, version); const span = await decodeSpan(r, version); - const terms = await decodeTerms(r, version); + const update = await decodeUpdate(r, version); const wanted = hasRefresh(version) ? await r.u62() : 0n; return new SubscribeUpdate({ wantedGroup: wanted === 0n ? undefined : wanted - 1n, @@ -510,7 +510,7 @@ export class SubscribeUpdate { maxDelay, minDelay, ...span, - terms, + update, }); } } @@ -574,11 +574,11 @@ export class Subscribe { ranges?: WireRange[]; /** - * The range terms id: every SUBSCRIBE_UPDATE carries a larger one, and a stale - * {@link SubscribeDrop} echoes the one it was judged under. Lite-07+; earlier wires decode - * as 0 and drop it on encode. + * The number of the terms in effect: the SUBSCRIBE is 0 and every SUBSCRIBE_UPDATE carries a + * larger one, which a {@link SubscribeDrop} echoes. Lite-07+; earlier wires decode as 0 and + * drop it on encode. */ - terms: bigint; + update: bigint; constructor(props: { id: bigint; @@ -594,7 +594,7 @@ export class Subscribe { startFrame?: number; endFrame?: number; ranges?: WireRange[]; - terms?: bigint; + update?: bigint; }) { this.id = props.id; this.broadcast = props.broadcast; @@ -609,7 +609,7 @@ export class Subscribe { this.startFrame = props.startFrame ?? 0; this.endFrame = props.endFrame; this.ranges = props.ranges; - this.terms = props.terms ?? 0n; + this.update = props.update ?? 0n; } async #encode(w: Writer, version: Version) { @@ -630,7 +630,7 @@ export class Subscribe { await encodeSpan(w, version, this, this.ranges); break; } - await encodeTerms(w, version, this.terms); + await encodeUpdate(w, version, this.update); } static async #decode(r: Reader, version: Version): Promise { @@ -649,7 +649,7 @@ export class Subscribe { const maxDelay = await r.u53(); const minDelay = await decodeMinDelay(r, version); const span = await decodeSpan(r, version); - const terms = await decodeTerms(r, version); + const update = await decodeUpdate(r, version); return new Subscribe({ id, broadcast, @@ -660,7 +660,7 @@ export class Subscribe { maxDelay, minDelay, ...span, - terms, + update, }); } } @@ -881,40 +881,27 @@ export class SubscribeDrop { end: bigint; error: number; /** - * The budget in milliseconds a stale verdict was judged under, undefined for a drop that does - * not depend on it. Lite-07+, encoded as the value plus one; earlier wires decode as undefined. + * The number of the SUBSCRIBE or SUBSCRIBE_UPDATE in effect when the groups were given up + * ({@link Subscribe.update}): a stale (OLD) drop under a superseded one is no verdict. + * Lite-07+; earlier wires decode as 0. */ - maxAge?: number; - /** The range terms id ({@link Subscribe.terms}) a stale verdict was judged under. Lite-07+. */ - terms: bigint; + update: bigint; - constructor(props: { start: bigint; end: bigint; error: number; maxAge?: number; terms?: bigint }) { + constructor(props: { start: bigint; end: bigint; error: number; update?: bigint }) { this.start = props.start; this.end = props.end; this.error = props.error; - this.maxAge = props.maxAge; - this.terms = props.terms ?? 0n; + this.update = props.update ?? 0n; } - /** - * Name `[start, end]` given up with `err`, as `version` carries it, judged under `maxAge` and - * `terms` when the verdict depends on the budget. - */ - static given( - start: bigint, - end: bigint, - err: unknown, - version: Version, - judged?: { maxAge?: number; terms: bigint }, - ): SubscribeDrop { - const skip = hasSkip(version); + /** Name `[start, end]` given up with `err` under the terms numbered `update`, as `version` carries it. */ + static given(start: bigint, end: bigint, err: unknown, version: Version, update = 0n): SubscribeDrop { return new SubscribeDrop({ start, end, // Before lite-07 the code is the application's, so it says nothing of why. error: hasDropCode(version) ? toStreamCode(err) : 0, - maxAge: skip ? judged?.maxAge : undefined, - terms: skip ? (judged?.terms ?? 0n) : 0n, + update: hasSkip(version) ? update : 0n, }); } @@ -936,10 +923,7 @@ export class SubscribeDrop { await w.u62(this.start); await w.u62(this.end); await w.u53(this.error); - if (hasSkip(version)) { - await w.u62(this.maxAge === undefined ? 0n : BigInt(this.maxAge) + 1n); - await w.u62(this.terms); - } + await encodeUpdate(w, version, this.update); } static async #decode(r: Reader, version: Version): Promise { @@ -947,15 +931,8 @@ export class SubscribeDrop { const end = await r.u62(); if (start - end > 1n) throw new ProtocolViolation("invalid dropped range"); const error = await r.u53(); - let maxAge: number | undefined; - let terms = 0n; - if (hasSkip(version)) { - const age = await r.u62(); - if (age > BigInt(Number.MAX_SAFE_INTEGER) + 1n) throw new RangeError("max age exceeds safe integer"); - maxAge = age === 0n ? undefined : Number(age - 1n); - terms = await r.u62(); - } - return new SubscribeDrop({ start, end, error, maxAge, terms }); + const update = await decodeUpdate(r, version); + return new SubscribeDrop({ start, end, error, update }); } async encode(w: Writer, version: Version): Promise { diff --git a/js/net/src/lite/subscriber.ts b/js/net/src/lite/subscriber.ts index d18d10e8392..16b70bf1401 100644 --- a/js/net/src/lite/subscriber.ts +++ b/js/net/src/lite/subscriber.ts @@ -17,6 +17,7 @@ import { StreamCode, StreamError, sessionCause, + toStreamCode, unauthorized, } from "../error.ts"; import * as netGroup from "../group.ts"; @@ -89,6 +90,7 @@ import { hasRanges, hasRefresh, hasSetupStream, + hasSkip, hasSynced, hasTrackStream, hasTracks, @@ -160,13 +162,14 @@ interface SubscribeEntry { starts?: (bigint | undefined)[]; // Whether a SUBSCRIBE_UPDATE went out, which the publisher may have read before answering. updated?: boolean; - // The range terms id of the latest SUBSCRIBE or SUBSCRIBE_UPDATE sent: a stale drop judged - // under an earlier one is no verdict on the current terms. Undefined is the SUBSCRIBE's 0. - terms?: bigint; - // The max delay the latest SUBSCRIBE or SUBSCRIBE_UPDATE asked for, in milliseconds. + // The number of the latest SUBSCRIBE or SUBSCRIBE_UPDATE sent: a stale drop given up under an + // earlier one is no verdict on the current terms. Undefined is the SUBSCRIBE's 0. + update?: bigint; + // The max delay the latest SUBSCRIBE or SUBSCRIBE_UPDATE asked for, in milliseconds: the budget + // a stale drop under it was judged by. maxDelay?: number; - // The runs `tail` accounts for by a stale drop, with the max age each was judged under: a - // widening past it reverses the verdict, so they are owed again. + // The runs `tail` accounts for by a stale drop under `update`: the next update reopens them, + // since the publisher re-offers or drops them again under it. // Coalesced, so a peer repeating a drop holds one run, not one per message. stale?: DropRun[]; // Whether SUBSCRIBE_START arrived, which `start` alone cannot say when no range starts. @@ -1867,20 +1870,20 @@ export class Subscriber { } else if ("drop" in resp) { // The publisher will never send these groups, so they are accounted for without a // stream, and a reader of one still arriving ends with the drop's error now. - const { start, end, maxAge, terms } = resp.drop; - // A stale verdict under terms an update has since replaced says nothing of the - // current ones: the groups are unknown, and a reader that wants one fills it. - const superseded = maxAge !== undefined && terms < (entry.terms ?? 0n); - // One the update widened past is served or dropped again, so it is still owed. - const reversed = superseded && maxAge < (entry.maxDelay ?? 0); - if (!reversed) entry.tail.account(start, end + 1n); - if (maxAge !== undefined && !reversed) - entry.stale = insertDrop(entry.stale ?? [], start, end + 1n, staleHole(maxAge)); - if (superseded) continue; - hooks.dropTrackGroups(entry.track, start, end + 1n, { - error: resp.drop.cause(this.version), - stale: maxAge, - }); + const { start, end, update } = resp.drop; + const error = resp.drop.cause(this.version); + // On lite-07 an OLD drop is a stale verdict, which the next update reopens; every other + // drop is final, whatever update it names. + const stale = hasSkip(this.version) && toStreamCode(error) === StreamCode.Old; + // One given up under an update since replaced says nothing of the current terms: the + // publisher re-offers it or drops it again under the new number, so it is still owed, + // and a reader that wants it fills it meanwhile. + if (stale && update < (entry.update ?? 0n)) continue; + entry.tail.account(start, end + 1n); + // Judged under the budget this subscriber sent with that update. + const hole = stale ? staleHole(entry.maxDelay ?? 0) : { error }; + if (stale) entry.stale = insertDrop(entry.stale ?? [], start, end + 1n, hole); + hooks.dropTrackGroups(entry.track, start, end + 1n, hole); } else if ("skip" in resp) { // Never produced, so nothing is waited for and no reader ends. entry.tail.account(resp.skip.start, resp.skip.end + 1n); @@ -2059,20 +2062,18 @@ export class Subscriber { startGroup: span.start, endGroup: inclusiveGroupEnd(span.end), ranges: hasRanges(this.version) ? toWireRanges(ranges, entry.scale) : undefined, - terms: (entry.terms ?? 0n) + 1n, + update: (entry.update ?? 0n) + 1n, }); - // A stale drop judged under the previous terms may still be in flight, and this update - // may move the range it was judged against, so the runs held as stale become unknown - // again: a reader fills them rather than trust an old verdict. - entry.terms = update.terms; + // A stale drop given up under the previous update may still be in flight, and this one + // may move the range or budget it was judged against, so the runs held as stale become + // unknown again: a reader fills them rather than trust an old verdict. + entry.update = update.update; entry.maxDelay = update.maxDelay; hooks.supersedeStale(entry.track, judgedEnd(ranges)); - // A lite-07 publisher re-offers or drops again what a widening reverses, so the tail owes - // the stale runs judged under a narrower max age again. - for (const run of entry.stale ?? []) { - if ((run.stale ?? 0) < update.maxDelay) entry.tail.unaccount(run.start, run.end); - } - entry.stale = entry.stale?.filter((run) => (run.stale ?? 0) >= update.maxDelay); + // A lite-07 publisher re-offers every group it reported stale on any update, and drops + // again under the new number what is still stale, so the tail owes every stale run again. + for (const run of entry.stale ?? []) entry.tail.unaccount(run.start, run.end); + entry.stale = undefined; // Encoded apart from the write, so a bound the wire can't carry still fails the track. const encoded = await Writer.collect(this.version, (w) => update.encode(w, this.version)); // A failed write means the publisher stopped reading, as it does once it finishes the diff --git a/js/net/src/lite/widened_max_age.test.ts b/js/net/src/lite/widened_max_age.test.ts index 36002a6edb8..5a221f36847 100644 --- a/js/net/src/lite/widened_max_age.test.ts +++ b/js/net/src/lite/widened_max_age.test.ts @@ -162,3 +162,53 @@ test("a widening keeps a bounded lite-07 subscription open for the stale groups client.abort(); server.abort(); }); + +// Every lite-07 update reopens the stale drops, widening or not: the publisher drops again, under +// the new update's number, what is still stale, so the subscriber settles at once rather than +// waiting out its grace for a re-offer that never comes. +test("an update keeping the budget drops the stale groups again on lite-07", async () => { + const pair = createMockTransportPair(ALPN_07_WIP); + const origin = new OriginProducer(); + const publication = origin.createBroadcast(Path.from("room")); + publication.announce(); + const broadcast = publication.broadcast; + const producer = broadcast.createTrack("video", { timescale: Timescale.MILLI }); + const write = (sequence: number, millis: number) => { + const group = producer.createGroup(BigInt(sequence)); + group.writeFrame({ payload: new Uint8Array([sequence]), timestamp: Timestamp.fromMillis(millis) }); + group.finish(); + }; + write(0, 0); + + const [client, server] = await Promise.all([ + connect({ url, transport: pair.client }), + accept({ transport: pair.server, url, version: Version.DRAFT_07, publish: origin.consume() }), + ]); + const remote = wireOf(client).consume(Path.from("room")); + const reader = remote.track("video").subscribe({ delay: { max: Milli(1500) } }); + expect(await received(reader, 1)).toEqual([0n]); + for (const [sequence, millis] of [ + [1, 1000], + [2, 2000], + [3, 3000], + [4, 6000], + ]) { + write(sequence, millis); + } + expect(await received(reader, 2)).toEqual([3n, 4n]); + + // Only the priority changes: groups 1 and 2 are still stale, so they are dropped again. + reader.update({ priority: 5 }); + const before = performance.now(); + producer.finish(); + producer.close(); + expect(await reader.recvGroup()).toBeUndefined(); + expect(performance.now() - before).toBeLessThan(500); + + reader.close(); + remote.close(); + broadcast.close(); + origin.close(); + client.abort(); + server.abort(); +}); diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts index 82cb12a184c..0e4fc8b0780 100644 --- a/js/net/src/resume.test.ts +++ b/js/net/src/resume.test.ts @@ -8,7 +8,7 @@ import type { Session as IetfSession } from "./ietf/session.ts"; import { Subscriber as IetfSubscriber } from "./ietf/subscriber.ts"; import { Version as IetfVersion } from "./ietf/version.ts"; import * as Instance from "./instance.ts"; -import { hooks, MAX_DATAGRAMS } from "./internal.ts"; +import { hooks, MAX_DATAGRAMS, staleHole } from "./internal.ts"; import type { Lister } from "./listing.ts"; import { Producer } from "./origin.ts"; import * as Path from "./path.ts"; @@ -49,6 +49,8 @@ function serve(origin: Producer, source: BroadcastProducer, cost: bigint) { /** A copy of the logical track: every route's copy holds the same groups under the same sequences. */ function copy(source: BroadcastProducer) { const producer = source.createTrack("seq"); + // A route's copy is received, so its groups arrive in any order. + hooks.receiveTrack(producer); return { producer, write(sequence: bigint, frames = 1, close = true) { @@ -1372,6 +1374,42 @@ test("a drop waits while a copy the serving one replaced may still deliver it", } }); +// A received stale run the reader held back is unknown again once new terms supersede it on the +// serving copy: its upstream re-offers or drops it again, so it is not reported meanwhile. +test("a held-back stale run new terms supersede is not reported", async () => { + const clock = new Manual(); + const installed = clock.install(); + try { + const front = new Front(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + const ta = copy(a); + const tb = copy(b); + front.serve(a.consume()); + const sub = front.consumer.track("seq").subscribe({ delay: { max: Milli(1_000) } }); + ta.write(0n); + expect(recvDisposition(sub)).toBe(0n); + front.serve(b.consume()); + await settle(); + // The new route's upstream judged groups 1 and 2 stale; the old route may still deliver them. + hooks.dropTrackGroups(tb.producer, 1n, 3n, staleHole(1_000)); + expect(recvDisposition(sub)).toBe("idle"); + tb.write(3n); + expect(recvDisposition(sub)).toBe(3n); + // The route's subscription updates before the old copy is past the budget. + hooks.supersedeStale(tb.producer, undefined); + await clock.advance(Nano(1_000_000_000)); + expect(recvDisposition(sub)).toBe("idle"); + + sub.close(); + front.consumer.close(); + a.close(); + b.close(); + } finally { + installed.close(); + } +}); + test("a drop a replaced route held back is no verdict on the route serving next", async () => { const front = new Front(); const a = new BroadcastProducer(); diff --git a/js/net/src/resume.ts b/js/net/src/resume.ts index 5e76e75284f..c9c6c6bc8ad 100644 --- a/js/net/src/resume.ts +++ b/js/net/src/resume.ts @@ -730,11 +730,22 @@ class Reader implements ResumeSource { if (this.#closed) return undefined; const serving = this.#serving(); if (serving) { + // A received stale run queued here that new terms superseded since is unknown again: the + // copy no longer holds it, and its upstream re-offers or drops it again. + for (const run of this.#drops) { + if (!run.received) continue; + const held = hooks.staleHeld(serving.sub, run.start, run.end); + for (const gap of runGaps(held, run.start, run.end)) + this.#drops = removeRun(this.#drops, gap.start, gap.end); + } for (let dropped = hooks.tryDrop(serving.sub); dropped; dropped = hooks.tryDrop(serving.sub)) { // A route failing is no verdict: another route may still deliver the group, and // one it failed while buffered is handed out to continue through its recovery. if (routeFailed(dropped.error)) continue; - this.#drops = insertDrop(this.#drops, dropped.start, dropped.end, dropped); + // A stale run the copy holds came from its upstream, which new terms may supersede. + const received = + dropped.stale !== undefined && hooks.staleHeld(serving.sub, dropped.start, dropped.end).length > 0; + this.#drops = insertDrop(this.#drops, dropped.start, dropped.end, { ...dropped, received }); } } this.#drops = this.#drops.filter((run) => !delivered.covers(run.start, run.end)); diff --git a/js/net/src/track.test.ts b/js/net/src/track.test.ts index 95f4d01e12a..c3ce14dae49 100644 --- a/js/net/src/track.test.ts +++ b/js/net/src/track.test.ts @@ -4352,6 +4352,8 @@ test("replay and backlogs deliver sparse out-of-order arrivals ascending without // pending, so lower groups may still arrive behind a higher one. function pendingTail(end: bigint, maxDelay = Milli(60_000)) { const producer = new TrackProducer("test"); + // A received copy, whose groups arrive in any order. + hooks.receiveTrack(producer); const reader = producer.subscribe({ delay: { max: maxDelay } }); hooks.startTrack(producer, { start: 0n }); hooks.endTrack(producer, { group: end, frame: 0 }); @@ -4850,3 +4852,64 @@ test("an errored end keeps its history", async () => { expect(polledOrdered.tryNextGroup()?.sequence).toBe(1n); expect(() => polledOrdered.tryNextGroup()).toThrow(failed); }); + +// A local publisher creates its groups in order, so the gap a created group leaves is skipped at +// once rather than waited for or judged stale; a reason already given keeps it. A received copy's +// groups arrive in any order, so its gap is not skipped. Mirrors Rust. +test("a local producer skips the gap a created group leaves at once", () => { + const producer = new TrackProducer("test"); + producer.accept({}); + const reader = producer.subscribe({ delay: { max: null } }); + producer.createGroup(0n).finish(); + expect(hooks.tryRecvGroup(reader, true)).toMatchObject({ kind: "group", group: { sequence: 0n } }); + hooks.dropTrackGroups(producer, 2n, 3n, { error: DROPPED }); + producer.createGroup(4n).finish(); + expect(hooks.tryRecvGroup(reader, true)).toMatchObject({ kind: "drop", start: 1n, end: 2n, skip: true }); + expect(hooks.tryRecvGroup(reader, true)).toEqual({ kind: "drop", start: 2n, end: 3n, error: DROPPED }); + expect(hooks.tryRecvGroup(reader, true)).toMatchObject({ kind: "drop", start: 3n, end: 4n, skip: true }); + expect(hooks.tryRecvGroup(reader, true)).toMatchObject({ kind: "group", group: { sequence: 4n } }); + // A group appended next leaves no gap. + producer.appendGroup().finish(); + expect(hooks.tryRecvGroup(reader, true)).toMatchObject({ kind: "group", group: { sequence: 5n } }); + + const copy = new TrackProducer("test"); + hooks.receiveTrack(copy); + copy.accept({}); + const late = copy.subscribe({ delay: { max: null } }); + copy.createGroup(0n).finish(); + copy.createGroup(2n).finish(); + expect(hooks.tryRecvGroup(late, true)).toMatchObject({ kind: "group" }); + expect(hooks.tryRecvGroup(late, true)).toMatchObject({ kind: "group" }); + expect(hooks.tryRecvGroup(late, true)).toEqual({ kind: "idle" }); +}); + +// A late group the local publisher still creates in a skipped gap reaches a reader that has not +// moved past it, as a group stream wins over a drop. +test("a late local group wins over its skip for a reader that has not reported it", () => { + const producer = new TrackProducer("test"); + producer.accept({}); + producer.createGroup(0n).finish(); + producer.createGroup(2n).finish(); + const reader = producer.subscribe({ delay: { max: null }, order: "ascending" }); + producer.createGroup(1n).finish(); + const got: bigint[] = []; + for (let group = reader.tryRecvGroup(); group; group = reader.tryRecvGroup()) got.push(group.sequence); + expect(got).toEqual([0n, 1n, 2n]); +}); + +// A received stale run is judged from where it meets the reader's span, as Rust clips it: a reader +// asking only its suffix, up to where the upstream's range ended, is settled by that suffix. +test("a received stale run settles the suffix a narrower range asks for", () => { + const producer = new TrackProducer("video").accept({}); + hooks.receiveTrack(producer); + hooks.supersedeStale(producer, 10n); + const reader = producer.subscribe({ + delay: { max: Milli(500) }, + ranges: [{ start: { absolute: 5n }, end: { absolute: 10n } }], + }); + hooks.dropTrackGroups(producer, 0n, 10n, staleHole(500)); + expect(hooks.tryDrop(reader)).toMatchObject({ kind: "drop", start: 5n, end: 10n, stale: 500 }); + expect(hooks.tryDrop(reader)).toBeUndefined(); + reader.close(); + producer.close(); +}); diff --git a/js/net/src/track.ts b/js/net/src/track.ts index 39cbaba400c..3a8e1497ce3 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -40,6 +40,7 @@ import { removeRun, runGaps, type StagedTrack, + skipHole, staleHole, type TrackRequestOptions, } from "./internal.ts"; @@ -1633,6 +1634,10 @@ export class Producer { /** * Create a new group with the given sequence number. * + * A gap it leaves above the last group is skipped at once: readers move past it without + * waiting, and no reader ends. A group created in that gap later still reaches readers that + * have not moved past it. + * * Throws on a sequence that is still cached: a live duplicate would fan out to every * subscriber twice. An aborted incarnation is evicted so a fresh group can serve the * sequence again. Best effort (mirrors Rust): nothing remembers a sequence whose cache @@ -1645,6 +1650,14 @@ export class Producer { #createGroup(sequence: bigint, start: number): GroupProducer { this.#writable(sequence); + // What this track's own publisher passes over when it creates a group is skipped at once, + // never judged stale; a received copy's groups arrive in any order. A late group it still + // creates below wins over its skip, as a group stream wins over a drop. Mirrors Rust. + // A run already given up keeps its reason. + if (!this.#subscribed && this.#received > 0n) { + for (const gap of runGaps(this.#state.drops.peek(), this.#received, sequence)) + this.#drop(gap.start, gap.end, skipHole()); + } const existing = this.#cached.get(sequence); if (existing) { if (!(existing.group.closed.peek() instanceof Error)) { @@ -2065,15 +2078,9 @@ export class Subscriber { judgedEnd?: bigint; }; #reported: Run[] = []; - // The reported runs judged stale, with the budget each was judged under, for a wire whose - // widening re-offers them; see `hooks.reoffersReported`. - #reportedStale: DropRun[] = []; - // Whether a widening re-offers what this cursor reported stale, and the runs it took back - // since the owner last drained them. - #reoffersReported = false; - #unreported: Run[] = []; - // The budget the reported stale runs were last checked against: only a change can reverse one. - #reofferedAt?: number; + // The reported runs judged stale, for a wire whose update re-offers them; see + // `hooks.reofferStale`. + #reportedStale: Run[] = []; // The group the frame-level helpers are currently draining, acquired through the // sequence cursor so frame reads and {@link Ordered.nextGroup} share one floor. #frameGroup?: GroupConsumer; @@ -2352,19 +2359,6 @@ export class Subscriber { // skipped again, so a newer one handed out first never folds it into #delivered. Costs the // skipped groups, never the cache. #reoffer(budget: number): void { - // A reported stale run reversed is judged again, and owed again by the wire. - const changed = this.#reofferedAt !== budget; - this.#reofferedAt = budget; - if (this.#reoffersReported && changed && this.#reportedStale.some((run) => (run.stale ?? budget) < budget)) { - for (const run of this.#reportedStale) { - if ((run.stale ?? budget) >= budget) continue; - this.#reported = removeRun(this.#reported, run.start, run.end); - insertRun(this.#unreported, run.start, run.end); - } - this.#reportedStale = this.#reportedStale.filter((run) => (run.stale ?? budget) >= budget); - this.#pending = this.#pending.filter((run) => (run.stale ?? budget) >= budget); - this.#dropsSeen = undefined; - } if (this.#skipped.size === 0 || budget <= this.#narrowest) return; const sequences: bigint[] = []; let narrowest = Number.POSITIVE_INFINITY; @@ -2390,6 +2384,30 @@ export class Subscriber { if (producer && !this.#closed) reofferSequences(producer, this.#state, sequences); } + // Owe again every run this cursor gave up as stale, reported or not: an update reopens the + // verdict. A group still cached is offered again as a widened skip is, and whatever is still + // stale under the new terms is given up and reported again. Returns the runs taken back. A + // resumed reader reports only what its routes' copies give up, so it keeps its reports. + #reofferStale(): Run[] { + if (this.#resume) return []; + const taken: Run[] = []; + for (const run of this.#reportedStale) { + this.#reported = removeRun(this.#reported, run.start, run.end); + insertRun(taken, run.start, run.end); + } + this.#reportedStale = []; + for (const run of this.#pending) if (run.stale !== undefined) insertRun(taken, run.start, run.end); + this.#pending = this.#pending.filter((run) => run.stale === undefined); + // A received stale run is folded in again by the next scan, if the track still holds it. + this.#dropsSeen = undefined; + const sequences = [...this.#skipped.keys()].filter((sequence) => + taken.some((run) => run.start <= sequence && sequence < run.end), + ); + const producer = this.#state.producer; + if (sequences.length > 0 && producer && !this.#closed) reofferSequences(producer, this.#state, sequences); + return taken; + } + private constructor(name: string, state: TrackState) { this.name = name; this.#state = state; @@ -2531,13 +2549,17 @@ export class Subscriber { const budget = this.#drift().budget; const coverage = this.#coverage(); const all = this.#state.drops.peek(); - const drops = all.filter((run) => settles(run, budget, coverage, this.#state.judgedEnd)); - const unknown = all.filter((run) => !settles(run, budget, coverage, this.#state.judgedEnd)); const producer = this.#state.producer; for (const run of this.#window.runs(this.#window.floor() ?? 0n, end)) { let gaps = [run]; for (const delivered of taken) gaps = gaps.flatMap((gap) => delivered.gaps(gap.start, gap.end)); for (const gap of gaps) { + // Each run judged from where it meets the gap, as the reader would read it. + const overlapping = all + .filter((run) => run.start < gap.end && gap.start < run.end) + .map((run) => ({ ...run, start: max(run.start, gap.start), end: min(run.end, gap.end) })); + const drops = overlapping.filter((run) => settles(run, budget, coverage, this.#state.judgedEnd)); + const unknown = overlapping.filter((run) => !settles(run, budget, coverage, this.#state.judgedEnd)); for (const open of runGaps(drops, gap.start, gap.end)) { if (unknown.some((run) => run.start < open.end && open.start < run.end)) return false; // Not received yet, so it may still arrive. @@ -2617,16 +2639,7 @@ export class Subscriber { hooks.unreport = (subscriber, start, end) => { subscriber.#reported = removeRun(subscriber.#reported, start, end); }; - hooks.budget = (subscriber) => subscriber.#drift().budget; - hooks.reoffersReported = (subscriber) => { - subscriber.#reoffersReported = true; - return subscriber; - }; - hooks.takeUnreported = (subscriber) => { - const unreported = subscriber.#unreported; - subscriber.#unreported = []; - return unreported; - }; + hooks.reofferStale = (subscriber) => subscriber.#reofferStale(); hooks.isReceived = (subscriber) => { if (subscriber.#resume) return true; const producer = subscriber.#state.producer; @@ -2693,6 +2706,14 @@ export class Subscriber { const producer = subscriber.#state.producer; return producer !== undefined && cachedHolds(producer, sequence); }; + hooks.staleHeld = (subscriber, start, end) => { + const held: Run[] = []; + for (const run of subscriber.#state.drops.peek()) { + if (run.stale === undefined || run.end <= start || run.start >= end) continue; + insertRun(held, max(run.start, start), min(run.end, end)); + } + return held; + }; hooks.givenUp = (subscriber, sequence) => { const drops = subscriber.#state.drops; return { error: dropAt(drops.peek(), sequence), changed: drops }; @@ -3111,8 +3132,7 @@ export class Subscriber { const [gap] = runGaps(this.#reported, run.start, run.end); if (!gap) continue; insertRun(this.#reported, gap.start, gap.end); - if (hole.stale !== undefined) - this.#reportedStale = insertDrop(this.#reportedStale, gap.start, gap.end, hole); + if (hole.stale !== undefined) insertRun(this.#reportedStale, gap.start, gap.end); return { kind: "drop", ...gap, error: hole.error, skip: hole.skip, stale: hole.stale }; } return undefined; @@ -3150,7 +3170,10 @@ export class Subscriber { // The track's runs are folded in afresh, so one it forgot, such as a stale run new terms // superseded, or one that stopped settling this reader, leaves nothing behind. this.#pending = this.#pending.filter((run) => !run.received); - for (const run of drops) { + for (const dropped of drops) { + // Judged from where it meets this cursor's span, as Rust clips it. + const run = { ...dropped, start: max(dropped.start, start), end: min(dropped.end, end ?? dropped.end) }; + if (run.start >= run.end) continue; // A received stale run that does not settle this reader is unknown to it. if (!settles(run, budget, coverage, judgedEnd)) continue; this.#pending = insertDrop(this.#pending, run.start, run.end, { ...run, received: true }); diff --git a/quest/m1/drop-set.md b/quest/m1/drop-set.md index 11928b9bf42..8021e4d1b08 100644 --- a/quest/m1/drop-set.md +++ b/quest/m1/drop-set.md @@ -13,13 +13,13 @@ subscription with `TooFarBehind`. ## Plan [moq-dev/moq#7670](https://github.com/moq-dev/moq/pull/7670) landed the -behavior: lite-07 SUBSCRIBE_SKIP and SUBSCRIBE_DROP with `Max Age` and -`Terms`, a per-run reason (`Hole` in `rs/moq-net/src/model/track/hole.rs`, +behavior: lite-07 SUBSCRIBE_SKIP and SUBSCRIBE_DROP with `Update`, a per-run +reason (`Hole` in `rs/moq-net/src/model/track/hole.rs`, `Hole` in `js/net/src/internal.ts`), relays settling a received stale drop only for readers it covers (`Drift::settles`, `settles` in `js/net/src/track.ts`) and forgetting stale runs when they send new terms -(`supersede_stale`), and lite-07 widenings re-offering what was reported -stale (`reoffers_reported`). It kept today's storage: `TrackState::drops`, +(`supersede_stale`), and every lite-07 update re-offering what was reported +stale (`reoffer_stale`). It kept today's storage: `TrackState::drops`, `DropCursor` with its `reported` and `reported_stale` sets, `Skipped`, `Delivered`, `TrackRun::served`, and the JS `drops` signal, `#pending`, `#reported`, and `#reportedStale`, none billed to anything. This quest is @@ -38,7 +38,8 @@ chosen: a per-track run cap; upstream backpressure. The pool port itself is [JS cache pool](/quest/m1/js-cache-pool.md), split out on 2026-11-03 since it lands on its own; this quest bills the runs to it. -Decided 2026-11-03 (maintainer, /quest-unblock): range terms carry an +Decided 2026-11-03 (maintainer, /quest-unblock), renamed and narrowed by the +`/quest-complete` decisions below: range terms carry an explicit update id. Each lite-07 SUBSCRIBE and SUBSCRIBE_UPDATE carries an increasing range-terms id, and a stale drop echoes the id it was judged under. A relay treats a drop whose id has been superseded as unknown and @@ -50,8 +51,9 @@ group 2 under the open range and group 0 under a range capped before group conservative refill with bounded remembered ranges; treating every range update as invalidating in-flight drops. -Decided 2026-11-03 (maintainer, /quest-unblock): lite-07 splits its hole -message by whether the group ever existed. `SUBSCRIBE_SKIP {Group Start, +Decided 2026-11-03 (maintainer, /quest-unblock), with `Max Age` dropped and +`Terms` renamed `Update` by the `/quest-complete` decisions below: lite-07 +splits its hole message by whether the group ever existed. `SUBSCRIBE_SKIP {Group Start, Group End}` names sequences the publisher never produced, such as skipped sequence numbers or ones past a cut: it has no error code, ends no reader, and is final. `SUBSCRIBE_DROP {Group Start, Group End, Error Code, Max Age, @@ -66,6 +68,33 @@ and no budget to qualify, so one message conflated a final, code-free verdict with a reversible, reader-ending one. Not chosen: stale in SKIP; a single message with a reason enum; renaming only. +Decided 2026-11-03 (maintainer, /quest-complete), changing the design +[moq-dev/moq#7670](https://github.com/moq-dev/moq/pull/7670) first shipped: + +- **`Terms` is renamed `Update`** on lite-07 SUBSCRIBE, SUBSCRIBE_UPDATE and + SUBSCRIBE_DROP: the number of the SUBSCRIBE or SUBSCRIBE_UPDATE in effect, + in the Rust and JS code too. +- **SUBSCRIBE_DROP drops `Max Age`:** it is {Group Start, Group End, Error + Code, Update}. The budget a stale (OLD) drop was judged under comes from + the subscriber's own record of what it sent with that Update, and a drop + whose Update is superseded is no verdict. Any SUBSCRIBE_UPDATE reopens every + outstanding OLD drop in the subscriber's tail accounting; the publisher + re-offers what the new terms cover and drops again, under the new Update, + what is still stale. This replaces reopening only when the max delay widens + past the drop's. A non-OLD drop is final, whatever its Update. +- **SKIP comes from the publisher's own group creation, eagerly:** when a + local (non-received) track producer creates a group above the next + sequence, the gap is recorded as `Hole::Skip` at once, not judged stale + until the track ends, and `append_group` never leaves a gap. A later + `create_group` below the high-water mark still works and wins over the skip, + as a Group Stream wins over a drop: in-tree callers re-create aborted + sequences (`claim_sequence`) and deliver late lower groups, and 85 moq-net + tests rely on it, so refusing it was not chosen. +- **`track::Producer::drop_groups` is crate-private.** `skip_groups` stays + public for moq-archive, whose replay names the sequences it holds nothing + for up to a recorded end above its last group; a SKIP, matching the gaps + between stored groups that eager skipping already names. + Decided 2026-10-27 (maintainer): this runs after the range count cost work, which finished in [moq-dev/moq#7542](https://github.com/moq-dev/moq/pull/7542), and before [Rust groups by start](/quest/m1/time-addressing/model.md), which @@ -134,13 +163,14 @@ Still to cover, beyond the memory work: Tests, on mock time through a relay, in Rust and mirrored in `js/net`: - After a widening from B1 to B2 for a group the upstream no longer holds, an - in-flight drop tagged B1 settles only the B1 reader, and the re-drop tagged - B2 settles the B2 reader, over the wire. + in-flight drop under the B1 update settles no reader once the B2 update is + sent, and the re-drop under the B2 update settles the B2 reader, over the + wire. - A group a relay evicted while the upstream still holds it is never reported dropped: a new, a widened, and an owed reader each fill it by range. - Rust and `js/net` encode and decode an origin's eviction and retention - expiry drop with `Max Age` 0 across languages; a relay forwards them - unchanged, and a widening re-offers neither. + expiry drop (EVICTED, not OLD) across languages; a relay forwards them + unchanged, and no update re-offers either. - Through a lite-06 hop, a reported group stays dropped for every reader, with no session error; a shrink sends no drops. - Under pool pressure, delivered runs and budget-independent drops fold into diff --git a/quest/m1/lite07-finalize.md b/quest/m1/lite07-finalize.md index 02a6d46a94e..1dba587bf06 100644 --- a/quest/m1/lite07-finalize.md +++ b/quest/m1/lite07-finalize.md @@ -63,8 +63,8 @@ The cut: names that rejection, not only the cap. Decided 2026-10-20 (maintainer): the drop set joined Required, because it -adds `Max Age` to lite-07's SUBSCRIBE_DROP. That wire work, with -SUBSCRIBE_SKIP and range `Terms`, landed in +changes lite-07's SUBSCRIBE_DROP. That wire work, SUBSCRIBE_SKIP and the +`Update` number on SUBSCRIBE, SUBSCRIBE_UPDATE, and SUBSCRIBE_DROP, landed in [moq-dev/moq#7670](https://github.com/moq-dev/moq/pull/7670); what remains of the drop set is memory accounting with no wire change, so it no longer gates the cut. Nothing diff --git a/quest/m1/time-addressing/model.md b/quest/m1/time-addressing/model.md index cdad5d1bfe9..a6f307c98c9 100644 --- a/quest/m1/time-addressing/model.md +++ b/quest/m1/time-addressing/model.md @@ -63,6 +63,14 @@ a producer would need per-track memory of its last number. one FETCH per tick in the hole. Accepted: "This was a lingering problem. The entire point of (multiple) ranges is to avoid this issue." +- **Eager skips** (decided 2026-11-03, maintainer, /quest-complete): a local + producer's `create_group` skips the gap it leaves above its last group at + once, and `append_group` never leaves one + ([moq-dev/moq#7670](https://github.com/moq-dev/moq/pull/7670)). + `append_group(timestamp)` follows the same rule: every start between the + previous group's and the new one is skipped when the group is created, + never judged stale later. + Every current role of the sequence needs a replacement or a reason to drop it: diff --git a/quest/m2/ietf-hole-marker.md b/quest/m2/ietf-hole-marker.md index 77cf6ed36f9..6e38e19a030 100644 --- a/quest/m2/ietf-hole-marker.md +++ b/quest/m2/ietf-hole-marker.md @@ -13,8 +13,9 @@ the cache knows every group in the range: a local, moq-lite, or extension-carrying copy. Groups go out in order. End of Non-Existent Range (`0x8C`) marks only a proven hole: a group never produced, or a budget-independent drop (a cut, an origin's eviction or expiry). A stale -drop judged under a narrower budget or range, as lite-07's `Max Age` and -`Terms` qualify it, proves nothing for a wider FETCH. That +drop judged under a narrower budget or range, as lite-07's `Update` and the +subscriber's record of that update's terms qualify it, proves nothing for a +wider FETCH. That group is unknown, so the publisher fills it or the range is refused. Any other range stays refused, as [moq-dev/moq#7204](https://github.com/moq-dev/moq/pull/7204) @@ -60,8 +61,8 @@ Open, for this quest's interview: - The carrier: a message on the SUBSCRIBE's request stream (drafts 18 and 22 give each request its own stream), or a status on a group stream. -- Whether the drop names its budget and terms, as lite-07's SUBSCRIBE_DROP - `Max Age` and `Terms` do, so a stale drop doesn't settle a wider reader. +- Whether the drop names the terms it was judged under, as lite-07's + SUBSCRIBE_DROP `Update` does, so a stale drop doesn't settle a wider reader. - How a publisher answering a range walks the drop set's runs in group order, and where FETCH_OK's End Location falls. diff --git a/rs/moq-archive/src/consumer/tests.rs b/rs/moq-archive/src/consumer/tests.rs index e71fffe90df..ba52791f9df 100644 --- a/rs/moq-archive/src/consumer/tests.rs +++ b/rs/moq-archive/src/consumer/tests.rs @@ -580,7 +580,7 @@ async fn a_completed_archive_ends_as_recorded() { ); assert_eq!(fetch(&broadcast, "errored", 0).await.unwrap(), expected(true, 0, 0..1)); - // The sequences the archive holds nothing for are dropped, and the end settles over them. + // The sequences the archive holds nothing for are skipped, and the end settles over them. let mut reader = subscribe(&broadcast, "tail").await; assert_eq!(next(&mut reader).await.unwrap(), (0, expected(true, 0, 0..1))); assert!(next(&mut reader).await.is_none()); diff --git a/rs/moq-archive/src/consumer/track.rs b/rs/moq-archive/src/consumer/track.rs index 2ec3d1f2b97..4eb0ea60548 100644 --- a/rs/moq-archive/src/consumer/track.rs +++ b/rs/moq-archive/src/consumer/track.rs @@ -141,11 +141,12 @@ impl Live { }, None, )?; - // An empty group stores no object, so what the archive holds nothing for up to the - // end is not found here, and the end settles over it. + // An empty group stores no object, so the replay never produces what the archive + // holds nothing for up to the end, as it skips the gaps between stored groups, and + // the end settles over it. let fin = at.group + u64::from(at.frame != 0); let from = self.next.unwrap_or(0); - self.producer.drop_groups(from..fin, moq_net::StreamError::NotFound); + self.producer.skip_groups(from..fin); } End::Error(code) => { // After the newest stored sequence, keeping the history an abort would refuse. diff --git a/rs/moq-net/src/fuzz.rs b/rs/moq-net/src/fuzz.rs index d182e6212d6..9337310a280 100644 --- a/rs/moq-net/src/fuzz.rs +++ b/rs/moq-net/src/fuzz.rs @@ -648,7 +648,7 @@ impl LiteSample { max_delay: std::time::Duration::from_secs(10), min_delay: None, ranges: vec![lite::Range::OLDEST], - terms: 0, + update: 0, } .encode(w, version) .unwrap(), @@ -858,7 +858,7 @@ impl Default for Messages { start: Some(lite::Anchor::Absolute(crate::track::Position::group(1_000))), end: None, }], - terms: 0, + update: 0, }, lite_update: lite::SubscribeUpdate { wanted_group: None, @@ -870,7 +870,7 @@ impl Default for Messages { start: Some(lite::Anchor::Absolute(crate::track::Position::group(1_000))), end: Some(lite::Anchor::Absolute(crate::track::Position::group(2_001))), }], - terms: 0, + update: 0, }, lite_start: lite::SubscribeResponse::Start(lite::SubscribeStart { largest: None, @@ -1037,7 +1037,7 @@ pub fn encode_subscribe(broadcast: &Path, track: &str, ietf: bool, out: &mut Vec min_delay: None, ranges: vec![lite::Range::OLDEST], order: track::Order::Descending, - terms: 0, + update: 0, } .encode(w, BENCH_LITE) .unwrap(), diff --git a/rs/moq-net/src/lite/publisher.rs b/rs/moq-net/src/lite/publisher.rs index 74be35c40be..ed671a8d4e5 100644 --- a/rs/moq-net/src/lite/publisher.rs +++ b/rs/moq-net/src/lite/publisher.rs @@ -2263,7 +2263,7 @@ impl Request for SubscribeServe { }; let floor = cursor_floor(shared.version, bounds.groups().start(), latest); let mut run = TrackRun::new(sub, track, bounds, floor, steer_tx); - run.terms = msg.terms; + run.update = msg.update; if let Some(update) = update { run.update(update); } @@ -4117,8 +4117,11 @@ struct TrackRun { first: Option, // Groups given up before the start resolved, which it must not name, with why. skipped: RangeMap, - // The range terms id of the SUBSCRIBE or latest SUBSCRIBE_UPDATE, which a drop echoes. - terms: u64, + // The number of the SUBSCRIBE (0) or latest SUBSCRIBE_UPDATE, which a drop echoes. + update: u64, + // Lite-07 groups whose stream expired as stale, which every update drops again under its + // number: the subscriber reopens each stale drop then, and the stream already reset. + expired: RangeSet, end_sent: bool, // Lite-07 with every range bounded: once every group in its ranges is given up, it is // answered without a group to resolve its start, rather than left unanswered. @@ -4157,12 +4160,8 @@ impl TrackRun { // flow through the SUBSCRIBE_UPDATE arm below. let groups = bounds.groups(); let cap = groups.end().map_or(Bound::Unbounded, Bound::Excluded); - // The FIN of a covered subscription is the wire's to send, below. A lite-07 widening - // re-offers what was reported stale. - let mut track = track.with_range((Bound::Included(floor), cap)).parked(); - if ctx.version.has_skip() { - track = track.reoffers_reported(); - } + // The FIN of a covered subscription is the wire's to send, below. + let track = track.with_range((Bound::Included(floor), cap)).parked(); let emit_range = ctx.version.has_track_stream(); let datagrams = ctx.version.has_datagrams() && ctx.session.max_datagram_size() > 0; @@ -4184,7 +4183,8 @@ impl TrackRun { start: None, first: None, skipped: RangeMap::new(), - terms: 0, + update: 0, + expired: RangeSet::new(), end_sent: false, served, datagrams, @@ -4211,7 +4211,22 @@ impl TrackRun { // The update moves the cursor and the bounds the track side judges against, which // notifies nothing it parked on. Idle::disarm(&mut self.idle); - self.terms = upd.terms; + self.update = upd.update; + // Lite-07 reopens every stale drop on any update: what was reported stale is offered + // again, and dropped again under this update's number if still stale. + if self.ctx.version.has_skip() { + let reopened = self.track.reoffer_stale(); + self.owe(reopened); + // A stream already reset as stale is not served again: dropped again below. + let expired = std::mem::take(&mut self.expired); + let mut dropped = self.ctx.outcomes.dropped.lock().unwrap(); + dropped.extend( + expired + .iter() + .flat_map(|run| run.clone()) + .map(|sequence| (sequence, StreamError::Old)), + ); + } // Every update restates the wanted group. Past our largest it is a join, which // repeats the pending ask; at or below it the group already exists. if upd @@ -4373,7 +4388,14 @@ impl TrackRun { for (sequence, err) in dropped { // A stream expired under this subscription's budget is a stale verdict. let hole = match err { - StreamError::Old => Hole::stale(self.track.budget()), + StreamError::Old => { + if self.ctx.version.has_skip() + && let Some(end) = sequence.checked_add(1) + { + self.expired.insert(sequence..end); + } + Hole::stale(self.track.budget()) + } err => Hole::drop(err), }; writer.buffer(&self.hole(sequence..=sequence, &hole))?; @@ -4465,12 +4487,10 @@ impl TrackRun { ) { match res? { Recv::Group(group) => { - self.owe_unreported(); // Never a second stream for a sequence this subscription already accounted // for (served, sent as a datagram, or dropped): a group re-created after its // DROP stays dropped, since the subscriber may settle before it arrives. A - // stale drop a lite-07 widening reversed is owed again (see - // `Self::owe_unreported`). + // stale drop a lite-07 update reopened is owed again (see `Self::owe`). if self .served .as_ref() @@ -4576,10 +4596,7 @@ impl TrackRun { if !self.emit_range { return Ok(false); } - let polled = self.track.poll_drop(waiter); - // A widening judged inside that poll may owe a stale drop again, before it is redone. - self.owe_unreported(); - let Poll::Ready((range, hole)) = polled else { + let Poll::Ready((range, hole)) = self.track.poll_drop(waiter) else { return Ok(false); }; match self.start { @@ -4633,13 +4650,12 @@ impl TrackRun { } } - /// Owe again what this subscription reported stale and a widening reversed: lite-07 - /// re-offers a group it still holds and drops again, under the new terms, one it cannot + /// Owe `reopened` again, what this subscription reported stale before an update: lite-07 + /// re-offers a group it still holds and drops again, under the new update, one it cannot /// serve. Earlier versions' drops are final, so their cursor takes nothing back. - fn owe_unreported(&mut self) { - let unreported = self.track.take_unreported(); + fn owe(&mut self, reopened: Vec>) { if let Some(served) = &mut self.served { - for run in unreported { + for run in reopened { served.remove(run); } } @@ -4687,13 +4703,9 @@ impl TrackRun { start: *groups.start(), end: *groups.end(), }), - hole => lite::SubscribeResponse::Drop(lite::SubscribeDrop::new( - groups, - &hole.error(), - hole.stale_budget(), - self.terms, - version, - )), + hole => { + lite::SubscribeResponse::Drop(lite::SubscribeDrop::new(groups, &hole.error(), self.update, version)) + } } } @@ -6297,7 +6309,7 @@ mod serve_group_test { min_delay: None, ranges: vec![lite::Range::span(None, 0, None, None)], order: Order::Descending, - terms: 0, + update: 0, }; run.update(update(Some(0))); assert_eq!(track.demand().wanted(), Some(0), "stale request is not a join"); @@ -6362,7 +6374,7 @@ mod serve_group_test { max_delay: Duration::ZERO, min_delay: None, ranges: vec![lite::Range::span(None, 0, None, None)], - terms: 0, + update: 0, }); { let mut poll = std::pin::pin!(kio::wait(|waiter| run.poll(&mut writer, waiter))); @@ -6478,7 +6490,7 @@ mod serve_group_test { max_delay: Duration::from_secs(3600), min_delay: None, ranges: vec![lite::Range::span(Some(5), 2, Some(7), Some(3))], - terms: 0, + update: 0, }); track @@ -6645,7 +6657,7 @@ mod serve_group_test { max_delay: Duration::from_secs(3600), min_delay: None, ranges: vec![lite::Range::span(None, 0, None, None)], - terms: 0, + update: 0, }); let second = track .append_datagram(Timestamp::from_millis(2).unwrap(), b"b".as_slice()) @@ -6849,56 +6861,74 @@ mod serve_group_test { drops } - /// A stale group already named in SUBSCRIBE_DROP stays dropped before lite-07 once - /// SUBSCRIBE_UPDATE widens the max age past it, since those subscribers settle a drop for - /// good. Lite-07 re-offers it, and only it: the groups served around it are never resent. + /// The SUBSCRIBE_DROP ranges a run loop's subscribe stream carried, once flushed, with the + /// update each names. + async fn drop_updates( + writer: &mut Writer, + log: &Log, + version: Version, + ) -> Vec<(u64, u64, u64)> { + kio::wait(|waiter| writer.poll_flush(waiter)).await.unwrap(); + let writes = log.writes.lock().unwrap().clone(); + let mut buf = writes.as_slice(); + let mut drops = Vec::new(); + while !buf.is_empty() { + let (msg, size) = lite::SubscribeResponse::decode_slice(buf, version).unwrap(); + buf = &buf[size..]; + if let lite::SubscribeResponse::Drop(drop) = msg { + drops.push((drop.start, drop.end, drop.update)); + } + } + drops + } + + /// A stale group already named in SUBSCRIBE_DROP stays dropped before lite-07 after any + /// SUBSCRIBE_UPDATE, since those subscribers settle a drop for good. Lite-07 reopens it on + /// every update: one that widens the max age past it serves it, and one that keeps the + /// budget drops it again under the new update's number. The groups served around it are + /// never resent. #[kio::test] - async fn a_widened_max_age_reoffers_a_reported_stale_drop_on_lite07_alone() { + async fn any_update_reoffers_a_reported_stale_drop_on_lite07_alone() { for version in [Version::Lite05, Version::Lite06, Version::Lite07] { - let mut track = track::Producer::new( - Arc::new(broadcast::Info::default()), - "test", - crate::track::Info::timed(), - ); - let subscriber = track.subscribe( - track::Subscription::default() - .with_delay(track::Delay::default().with_max(Duration::from_millis(1500))), - ); - let (mut run, mut writer, log) = lite_run(version, SinkSession::new(Log::default()), subscriber); - write_group(&mut track, 0, 0); - while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} - // Groups 1 and 2 reach 2s and 3s, at least 1.5s behind the 6s edge. - for (sequence, millis) in [(1, 1000), (2, 2000), (3, 3000), (4, 6000)] { - write_group(&mut track, sequence, millis); + for widen in [true, false] { + let mut track = track::Producer::new( + Arc::new(broadcast::Info::default()), + "test", + crate::track::Info::timed(), + ); + let budget = Duration::from_millis(1500); + let subscriber = track + .subscribe(track::Subscription::default().with_delay(track::Delay::default().with_max(budget))); + let (mut run, mut writer, log) = lite_run(version, SinkSession::new(Log::default()), subscriber); + write_group(&mut track, 0, 0); + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + // Groups 1 and 2 reach 2s and 3s, at least 1.5s behind the 6s edge. + for (sequence, millis) in [(1, 1000), (2, 2000), (3, 3000), (4, 6000)] { + write_group(&mut track, sequence, millis); + } + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + let case = format!("{version:?}, widen={widen}"); + assert_eq!(run.ctx.outcomes.opened.load(Ordering::Relaxed), 3, "{case}"); + assert_eq!(drop_updates(&mut writer, &log, version).await, [(1, 2, 0)], "{case}"); + + run.update(lite::SubscribeUpdate { + wanted_group: None, + priority: 0, + max_delay: if widen { Duration::from_secs(10) } else { budget }, + min_delay: None, + ranges: vec![lite::Range::span(None, 0, None, None)], + order: Order::Descending, + update: 1, + }); + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + let (opened, drops, why): (u64, &[(u64, u64, u64)], _) = match (version, widen) { + (Version::Lite07, true) => (5, &[(1, 2, 0)], "the stale groups are served, and nothing else"), + (Version::Lite07, false) => (3, &[(1, 2, 0), (1, 2, 1)], "still stale, so dropped again"), + _ => (3, &[(1, 2, 0)], "no stream and no drop again"), + }; + assert_eq!(run.ctx.outcomes.opened.load(Ordering::Relaxed), opened, "{case}: {why}"); + assert_eq!(drop_updates(&mut writer, &log, version).await, drops, "{case}: {why}"); } - while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} - assert_eq!(run.ctx.outcomes.opened.load(Ordering::Relaxed), 3, "{version:?}"); - assert_eq!(drops(&mut writer, &log, version).await, [(1, 2)], "{version:?}"); - - run.update(lite::SubscribeUpdate { - wanted_group: None, - priority: 0, - max_delay: Duration::from_secs(10), - min_delay: None, - ranges: vec![lite::Range::span(None, 0, None, None)], - order: Order::Descending, - terms: 1, - }); - while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} - let (opened, why) = match version { - Version::Lite07 => (5, "the reported stale groups are re-offered, and nothing else"), - _ => (3, "no stream for a dropped group"), - }; - assert_eq!( - run.ctx.outcomes.opened.load(Ordering::Relaxed), - opened, - "{version:?}: {why}" - ); - assert_eq!( - drops(&mut writer, &log, version).await, - [(1, 2)], - "{version:?}: no drop again" - ); } } @@ -7047,7 +7077,7 @@ mod serve_group_test { max_delay: Duration::from_secs(60), min_delay: None, ranges: vec![lite::Range::span(Some(0), 0, None, None)], - terms: 0, + update: 0, }); while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} assert_eq!(drops(&mut writer, &log, version).await, [(2, 2)]); @@ -7123,6 +7153,8 @@ mod serve_group_test { "test", crate::track::Info::timed(), ); + // A received copy, whose groups may land out of order. + track.received(); let subscriber = track.subscribe( track::Subscription::default().with_delay(track::Delay::default().with_max(Duration::from_secs(1))), ); @@ -7455,7 +7487,7 @@ mod serve_group_test { max_delay: Duration::from_secs(60), min_delay: None, ranges: vec![lite::Range::span(Some(0), 0, None, None)], - terms: 0, + update: 0, }); { let mut run = std::pin::pin!(kio::wait(|waiter| run.poll(&mut writer, waiter))); @@ -7464,8 +7496,10 @@ mod serve_group_test { run.await.unwrap(); } // Group 2 arrived before the subscription, but it is cached, so the lowered start - // serves it and the drops name only what is missing around it. - assert_eq!(drops(&mut writer, &log, version).await, [(0, 1), (3, 4), (6, 6)]); + // serves it and the drops name only what is missing around it. The publisher skipped + // 3 and 4 when it created group 5, so they go out once the lowered start reaches them, + // before the end names the rest. + assert_eq!(drops(&mut writer, &log, version).await, [(3, 4), (0, 1), (6, 6)]); } /// A lowered start restarts the age of the gaps it newly asks for, as the subscriber does, @@ -7497,7 +7531,7 @@ mod serve_group_test { max_delay: Duration::from_secs(1), min_delay: None, ranges: vec![lite::Range::span(Some(0), 0, None, None)], - terms: 0, + update: 0, }); { let mut run = std::pin::pin!(kio::wait(|waiter| run.poll(&mut writer, waiter))); @@ -7544,7 +7578,7 @@ mod serve_group_test { max_delay: Duration::from_secs(1), min_delay: None, ranges: vec![lite::Range::span(Some(start_group), 0, None, None)], - terms: 0, + update: 0, }; run.update(update(5)); { @@ -7591,7 +7625,7 @@ mod serve_group_test { max_delay: Duration::from_secs(10), min_delay: None, ranges: vec![lite::Range::span(Some(start_group), 0, None, None)], - terms: 0, + update: 0, }; run.update(update(5)); while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} @@ -7638,7 +7672,7 @@ mod serve_group_test { max_delay: Duration::from_secs(60), min_delay: None, ranges: vec![lite::Range::span(None, 0, None, None)], - terms: 0, + update: 0, }); write_group(&mut track, 5, 5); assert_eq!(settle(&mut run), 1, "served a group below the latest"); @@ -7669,7 +7703,7 @@ mod serve_group_test { max_delay: Duration::from_secs(60), min_delay: None, ranges: vec![lite::Range::span(None, 0, None, None)], - terms: 0, + update: 0, }); write_group(&mut track, 0, 0); @@ -7807,7 +7841,7 @@ mod serve_group_test { max_delay: Duration::from_secs(30), min_delay: None, ranges: vec![lite::Range::span(None, 0, None, None)], - terms: 0, + update: 0, }); relay.settle(); assert_eq!(relay.opened(), 1, "the finished group is the live edge"); @@ -7838,7 +7872,7 @@ mod serve_group_test { max_delay: Duration::from_secs(30), min_delay: None, ranges: vec![lite::Range::span(None, 0, None, None)], - terms: 0, + update: 0, }); relay.settle(); assert_eq!(relay.opened(), 3, "group 0 is cached and now in range"); @@ -7866,7 +7900,7 @@ mod serve_group_test { max_delay: Duration::ZERO, min_delay: None, ranges: vec![lite::Range::span(Some(7), 0, None, None)], - terms: 0, + update: 0, }; relay.run.update(update); relay.settle(); @@ -8003,9 +8037,8 @@ mod serve_group_test { assert!(futures::poll!(run.as_mut()).is_pending(), "group 1 is still opening"); assert_eq!( *log.writes.lock().unwrap(), - // The DROP carries OLD (0x34), judged under a zero budget (Max Age 1) and the - // SUBSCRIBE's terms. - [0, 4, 1, 0, 1, 1, 2, 5, 0, 0, 0x34, 1, 0, 1, 2, 2, 0], + // The DROP carries OLD (0x34) and the SUBSCRIBE's update, 0. + [0, 4, 1, 0, 1, 1, 2, 4, 0, 0, 0x34, 0, 1, 2, 2, 0], "the expired group is dropped and the end precedes stream credit" ); @@ -8017,7 +8050,7 @@ mod serve_group_test { assert!(futures::poll!(run.as_mut()).is_pending(), "the stream stays open"); assert_eq!( *log.writes.lock().unwrap(), - [0, 4, 1, 0, 1, 1, 2, 5, 0, 0, 0x34, 1, 0, 1, 2, 2, 0] + [0, 4, 1, 0, 1, 1, 2, 4, 0, 0, 0x34, 0, 1, 2, 2, 0] ); } } @@ -8706,7 +8739,7 @@ mod tests { max_delay: Duration::ZERO, min_delay: None, ranges: vec![lite::Range::span(None, 0, None, None)], - terms: 0, + update: 0, }; leave_request::(version, &subscribe, wait, close).await; @@ -8767,7 +8800,7 @@ mod tests { max_delay: Duration::ZERO, min_delay: None, ranges: vec![lite::Range::span(None, 0, None, None)], - terms: 0, + update: 0, } .encode(&mut crate::coding::Encoder::new(&mut script, version.into()), version) .unwrap(); @@ -8985,7 +9018,7 @@ mod tests { min_delay: None, // A relay asks from its cached group, so the SUBSCRIBE is not a live-edge join. ranges: vec![lite::Range::span(Some(1), 0, None, None)], - terms: 0, + update: 0, } .encode(&mut crate::coding::Encoder::new(&mut script, version.into()), version) .unwrap(); @@ -9009,7 +9042,7 @@ mod tests { max_delay: Duration::ZERO, min_delay: None, ranges: vec![lite::Range::span(Some(1), 0, None, None)], - terms: 0, + update: 0, }; session.push(&update(Some(2), 3).encode_bytes(version).unwrap()); session.push(&update(Some(2), 7).encode_bytes(version).unwrap()); @@ -9069,7 +9102,7 @@ mod tests { max_delay: Duration::from_secs(3600), min_delay: None, ranges: vec![lite::Range::span(None, 0, None, None)], - terms: 0, + update: 0, } } @@ -9109,7 +9142,7 @@ mod tests { max_delay: Duration::from_secs(3600), min_delay: None, ranges: vec![lite::Range::span(None, 0, None, None)], - terms: 0, + update: 0, } .encode(&mut crate::coding::Encoder::new(&mut script, version.into()), version) .unwrap(); @@ -9532,7 +9565,7 @@ mod tests { max_delay: Duration::ZERO, min_delay: None, ranges: vec![lite::Range::span(None, 0, None, None)], - terms: 0, + update: 0, }; let session = ScriptedSession::new(request.encode_bytes(version).unwrap().to_vec()); let log = session.log.clone(); @@ -9645,7 +9678,7 @@ mod tests { max_delay: Duration::ZERO, min_delay: None, ranges: vec![lite::Range::span(None, 0, None, None)], - terms: 0, + update: 0, }; let session = ScriptedSession::new(request.encode_bytes(version).unwrap().to_vec()); let log = session.log.clone(); @@ -9784,7 +9817,7 @@ mod tests { max_delay: Duration::ZERO, min_delay: None, ranges: vec![lite::Range::span(None, 0, None, None)], - terms: 0, + update: 0, }; let mut script = Vec::new(); msg.encode(&mut crate::coding::Encoder::new(&mut script, version.into()), version) diff --git a/rs/moq-net/src/lite/subscribe.rs b/rs/moq-net/src/lite/subscribe.rs index 14ca5804c8d..e7ad30bc354 100644 --- a/rs/moq-net/src/lite/subscribe.rs +++ b/rs/moq-net/src/lite/subscribe.rs @@ -30,10 +30,10 @@ pub struct Subscribe<'a> { /// Which pending group to send first. Lite07+ only; earlier wires decode as /// [`Order::Descending`] and cannot ask for anything else. pub order: Order, - /// The range terms id: every SUBSCRIBE_UPDATE carries a larger one, and a stale - /// [`SubscribeDrop`] echoes the one it was judged under. Lite07+ only; earlier wires + /// The number of the terms in effect: the SUBSCRIBE is 0 and every SUBSCRIBE_UPDATE + /// carries a larger one, which a [`SubscribeDrop`] echoes. Lite07+ only; earlier wires /// decode as 0 and drop it on encode. - pub terms: u64, + pub update: u64, } impl Subscribe<'_> { @@ -49,7 +49,7 @@ impl Subscribe<'_> { min_delay: self.min_delay, ranges: self.ranges, order: self.order, - terms: self.terms, + update: self.update, } } } @@ -147,7 +147,7 @@ impl<'a> Message<'a> for Subscribe<'a> { (order, max_delay, min_delay, decode_ranges(r, version)?) } }; - let terms = decode_terms(r, version)?; + let update = decode_update(r, version)?; Ok(Self { id, @@ -159,7 +159,7 @@ impl<'a> Message<'a> for Subscribe<'a> { min_delay, ranges, order, - terms, + update, }) } @@ -180,24 +180,24 @@ impl<'a> Message<'a> for Subscribe<'a> { encode_ranges(w, version, &self.ranges)?; } } - encode_terms(w, version, self.terms)?; + encode_update(w, version, self.update)?; Ok(()) } } -/// Decode the range `Terms` after the ranges. A wire without the field decodes as 0. -fn decode_terms(r: &mut Decoder<'_>, version: Version) -> Result { +/// Decode the `Update` number after the ranges. A wire without the field decodes as 0. +fn decode_update(r: &mut Decoder<'_>, version: Version) -> Result { match version.has_skip() { true => r.varint(), false => Ok(0), } } -/// Encode the range `Terms`, dropping it on a wire without the field. -fn encode_terms(w: &mut Encoder<'_>, version: Version, terms: u64) -> Result<(), EncodeError> { +/// Encode the `Update` number, dropping it on a wire without the field. +fn encode_update(w: &mut Encoder<'_>, version: Version, update: u64) -> Result<(), EncodeError> { match version.has_skip() { - true => w.varint(terms), + true => w.varint(update), false => Ok(()), } } @@ -676,8 +676,8 @@ pub struct SubscribeUpdate { pub ranges: Vec, /// See [`Subscribe::order`]. pub order: Order, - /// See [`Subscribe::terms`]: larger than the subscription's previous terms. - pub terms: u64, + /// See [`Subscribe::update`]: larger than the subscription's previous one. + pub update: u64, } impl<'a> Message<'a> for SubscribeUpdate { @@ -695,7 +695,7 @@ impl<'a> Message<'a> for SubscribeUpdate { let max_delay = std::time::Duration::from_millis(r.varint()?); let min_delay = decode_min_delay(r, version)?; let ranges = decode_ranges(r, version)?; - let terms = decode_terms(r, version)?; + let update = decode_update(r, version)?; let wanted_group = match version.has_refresh() { true => r.varint()?.checked_sub(1), @@ -709,7 +709,7 @@ impl<'a> Message<'a> for SubscribeUpdate { min_delay, ranges, order, - terms, + update, }) } @@ -726,7 +726,7 @@ impl<'a> Message<'a> for SubscribeUpdate { w.varint(u64::try_from(self.max_delay.as_millis()).map_err(|_| EncodeError::BoundsExceeded)?)?; encode_min_delay(w, version, self.min_delay)?; encode_ranges(w, version, &self.ranges)?; - encode_terms(w, version, self.terms)?; + encode_update(w, version, self.update)?; if version.has_refresh() { w.varint_opt(self.wanted_group)?; @@ -755,26 +755,17 @@ pub struct SubscribeDrop { /// application-specific code before, where 0 means simply unavailable. pub error: u64, - /// The budget a stale verdict was judged under, `None` for a drop that does not depend - /// on it. Lite07+ only, encoded as milliseconds plus one; earlier wires decode as `None`. - pub max_age: Option, - - /// The range terms id ([`Subscribe::terms`]) a stale verdict was judged under. Lite07+ - /// only; earlier wires decode as 0. - pub terms: u64, + /// The number of the SUBSCRIBE or SUBSCRIBE_UPDATE in effect when the groups were + /// given up ([`Subscribe::update`]): a stale (OLD) drop under a superseded one is no + /// verdict. Lite07+ only; earlier wires decode as 0. + pub update: u64, } #[cfg(feature = "async")] impl SubscribeDrop { - /// Name `groups` given up with `err`, as `version` carries it, judged under `max_age` - /// and `terms` when the verdict depends on the budget. - pub fn new( - groups: std::ops::RangeInclusive, - err: &StreamError, - max_age: Option, - terms: u64, - version: Version, - ) -> Self { + /// Name `groups` given up with `err` under the terms numbered `update`, as `version` + /// carries it. + pub fn new(groups: std::ops::RangeInclusive, err: &StreamError, update: u64, version: Version) -> Self { Self { start: *groups.start(), end: *groups.end(), @@ -783,8 +774,7 @@ impl SubscribeDrop { true => version.encode_stream_code(err).into(), false => 0, }, - max_age: max_age.filter(|_| version.has_skip()), - terms: if version.has_skip() { terms } else { 0 }, + update: if version.has_skip() { update } else { 0 }, } } @@ -816,16 +806,12 @@ impl<'a> Message<'a> for SubscribeDrop { return Err(DecodeError::InvalidValue); } let error = r.varint()?; - let (max_age, terms) = match version.has_skip() { - true => (r.varint_opt()?.map(std::time::Duration::from_millis), r.varint()?), - false => (None, 0), - }; + let update = decode_update(r, version)?; Ok(Self { start, end, error, - max_age, - terms, + update, }) } @@ -844,14 +830,7 @@ impl<'a> Message<'a> for SubscribeDrop { w.varint(self.start)?; w.varint(self.end)?; w.varint(self.error)?; - if version.has_skip() { - let max_age = self - .max_age - .map(|age| u64::try_from(age.as_millis()).map_err(|_| EncodeError::BoundsExceeded)) - .transpose()?; - w.varint_opt(max_age)?; - w.varint(self.terms)?; - } + encode_update(w, version, self.update)?; Ok(()) } @@ -997,8 +976,7 @@ mod test { start: 3, end: 1, error: 0, - max_age: None, - terms: 0, + update: 0, } .encode_bytes(version) .is_err() @@ -1013,8 +991,7 @@ mod test { start, end, error: 0, - max_age: None, - terms: 0, + update: 0, }; let data = msg.encode_bytes(version).unwrap(); let (decoded, _) = SubscribeDrop::decode_slice(&data, version).unwrap(); @@ -1135,14 +1112,13 @@ mod test { start: 1, end: 3, error: 7, - max_age: None, - terms: 0, + update: 0, }); let mut buf = Vec::new(); resp.encode(&mut Encoder::new(&mut buf, Version::Lite07.into()), Version::Lite07) .unwrap(); - // A drop no budget reverses has a Max Age of 0. - assert_eq!(buf, [2, 5, 1, 3, 7, 0, 0]); + // The drop names the SUBSCRIBE's update, 0. + assert_eq!(buf, [2, 4, 1, 3, 7, 0]); match crate::coding::decode_buf(&mut buf.as_slice(), Version::Lite07, |r, v| { SubscribeResponse::decode(r, v) }) @@ -1153,33 +1129,24 @@ mod test { } } - /// A stale drop names the budget, as milliseconds plus one so a zero budget is 1, and the - /// terms it was judged under; earlier versions carry neither. + /// A drop names the update it was given up under on lite-07; earlier versions carry none. #[test] - fn a_stale_drop_names_its_max_age_and_terms_on_lite07() { - for max_age in [std::time::Duration::ZERO, std::time::Duration::from_millis(300)] { - let msg = SubscribeDrop { - start: 4, - end: 4, - error: 0x34, - max_age: Some(max_age), - terms: 2, - }; - let buf = msg.encode_bytes(Version::Lite07).unwrap(); - let (decoded, _) = SubscribeDrop::decode_slice(&buf, Version::Lite07).unwrap(); - assert_eq!(decoded, msg); - } + fn a_drop_names_its_update_on_lite07() { let msg = SubscribeDrop { start: 4, end: 4, - error: 0, - max_age: Some(std::time::Duration::from_millis(300)), - terms: 2, + error: 0x34, + update: 2, }; + let buf = msg.encode_bytes(Version::Lite07).unwrap(); + assert_eq!(buf.as_ref(), [4, 4, 4, 0x34, 2]); + let (decoded, _) = SubscribeDrop::decode_slice(&buf, Version::Lite07).unwrap(); + assert_eq!(decoded, msg); + let msg = SubscribeDrop { error: 0, ..msg }; let buf = msg.encode_bytes(Version::Lite06).unwrap(); assert_eq!(buf.as_ref(), [3, 4, 4, 0]); let (decoded, _) = SubscribeDrop::decode_slice(&buf, Version::Lite06).unwrap(); - assert_eq!((decoded.max_age, decoded.terms), (None, 0)); + assert_eq!(decoded.update, 0); } /// SUBSCRIBE_SKIP is type 3 on lite-07 and unknown before it. @@ -1203,15 +1170,15 @@ mod test { )); } - /// SUBSCRIBE and SUBSCRIBE_UPDATE carry their range terms on lite-07 alone. + /// SUBSCRIBE and SUBSCRIBE_UPDATE carry their update number on lite-07 alone. #[test] - fn range_terms_roundtrip_only_on_lite07() { + fn update_number_roundtrips_only_on_lite07() { for version in [Version::Lite06, Version::Lite07] { let mut msg = subscribe_sample(); - msg.terms = 5; + msg.update = 5; let buf = encode(&msg, version).unwrap(); let decoded = crate::coding::decode_from(&mut buf.as_slice(), version, Subscribe::decode_msg).unwrap(); - assert_eq!(decoded.terms, if version.has_skip() { 5 } else { 0 }); + assert_eq!(decoded.update, if version.has_skip() { 5 } else { 0 }); } } @@ -1221,8 +1188,7 @@ mod test { start: 1, end: 3, error: 0, - max_age: None, - terms: 0, + update: 0, }); let mut buf = Vec::new(); resp.encode(&mut Encoder::new(&mut buf, Version::Lite05.into()), Version::Lite05) @@ -1243,8 +1209,7 @@ mod test { start: 1, end: 3, error: 0, - max_age: None, - terms: 0, + update: 0, }); let mut buf = Vec::new(); resp.encode(&mut Encoder::new(&mut buf, Version::Lite04.into()), Version::Lite04) @@ -1273,7 +1238,7 @@ mod test { start: absolute(7, 4), end: absolute(9, 3), }], - terms: 0, + update: 0, } } @@ -1452,7 +1417,7 @@ mod test { min_delay: None, ranges: msg.ranges.clone(), order: crate::track::Order::Ascending, - terms: 0, + update: 0, }; let buf = update.encode_bytes(Version::Lite07).unwrap(); let got = crate::coding::decode_buf(&mut buf.as_ref(), Version::Lite07, |r, v| SubscribeUpdate::decode(r, v)) @@ -1471,7 +1436,7 @@ mod test { msg.ranges = vec![Range::OLDEST]; let mut buf = encode(&msg, version).unwrap(); - // The list is last but for the terms: a count of one, then two anchors of type 0. + // The list is last but for the update number: a count of one, then two anchors of type 0. let at = buf.len() - 4; assert_eq!(&buf[at..], [1, 0, 0, 0]); buf[at] = 0; @@ -1548,7 +1513,7 @@ mod test { min_delay: None, ranges: vec![Range::OLDEST], order: crate::track::Order::Descending, - terms: 0, + update: 0, }; let buf = update.encode_bytes(version).unwrap(); let got = crate::coding::decode_buf(&mut buf.as_ref(), version, |r, v| SubscribeUpdate::decode(r, v)) @@ -1582,7 +1547,7 @@ mod test { min_delay, ranges: vec![Range::OLDEST], order: crate::track::Order::Descending, - terms: 0, + update: 0, }; let buf = update.encode_bytes(version).unwrap(); let got = crate::coding::decode_buf(&mut buf.as_ref(), version, |r, v| SubscribeUpdate::decode(r, v)) @@ -1611,7 +1576,7 @@ mod test { min_delay: None, ranges: vec![Range::OLDEST], order, - terms: 0, + update: 0, }; let mut buf = Vec::new(); update diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index 1438e94f073..71f70686dd4 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -19,7 +19,7 @@ use super::Version; use crate::tail::{self, Reading, Settle, Tail}; use kio::Lock; -use rangemap::RangeMap; +use rangemap::RangeSet; pub(super) struct SubscriberConfig { pub runtime: crate::time::Deadlines, @@ -2436,6 +2436,8 @@ mod tests { fn feed_fetch(sequence: u64) -> (track::Producer, kio::Pending, track::Fill) { let info = track::Info::default().with_timescale(None); let track = track::Producer::new(std::sync::Arc::new(crate::broadcast::Info::default()), "video", info); + // The relay's copy of the upstream feed, whose groups may land out of order. + track.received(); let dynamic = track.requests(); let fetching = track.consume().fetch_group(sequence, None); let req = dynamic.requested_fill().now_or_never().expect("queued").unwrap(); @@ -2466,8 +2468,8 @@ mod tests { unanswered: None, updated: false, _working: crate::session::Work::default().start(), - terms: 0, - stale: RangeMap::new(), + update: 0, + stale: RangeSet::new(), }) } @@ -2617,8 +2619,8 @@ mod tests { unanswered: None, updated: false, _working: crate::session::Work::default().start(), - terms: 0, - stale: RangeMap::new(), + update: 0, + stale: RangeSet::new(), }); if update { h.serve @@ -2659,6 +2661,8 @@ mod tests { ) -> (track::Producer, Vec>, Vec) { let info = track::Info::default().with_timescale(None); let track = track::Producer::new(Arc::new(crate::broadcast::Info::default()), "video", info); + // The relay's copy of the upstream feed, whose groups may land out of order. + track.received(); let dynamic = track.requests(); let mut callers = Vec::new(); let mut requests = Vec::new(); @@ -3080,8 +3084,8 @@ mod tests { unanswered: None, updated: false, _working: crate::session::Work::default().start(), - terms: 0, - stale: RangeMap::new(), + update: 0, + stale: RangeSet::new(), }; assert_eq!( sub.owed(None), @@ -4140,6 +4144,130 @@ mod tests { assert!(matches!(reader.recv_group().await, Err(Error::ProtocolViolation))); } + /// On lite-07 an OLD drop is a stale verdict judged under the budget this subscriber sent + /// with the update it names. Any SUBSCRIBE_UPDATE reopens every stale run in the tail, + /// whether or not it widens the budget, and an OLD drop naming a superseded update is no + /// verdict: still owed, and given up on no reader. Any other drop is final, whatever update + /// it names. + #[kio::test] + async fn an_update_reopens_every_stale_drop_and_supersedes_the_rest() { + let version = Version::Lite07; + let session = crate::lite::test_transport::ScriptedSession::new(Vec::new()); + let subscriber = Subscriber::new(SubscriberConfig { + runtime: crate::time::sim(), + session: crate::driver::Transport::new(session.clone()), + origin: origin::Config::new(crate::Hop::new(1).unwrap()).produce(), + recv_bandwidth: None, + version, + peer_setup: crate::lite::PeerSetup::new(false, crate::transport::Address::Setup), + peer_hop: None, + cost: None, + going_away: Default::default(), + auth: crate::auth::Link::new(false), + }); + let serve = TrackServe { + validity: Default::default(), + route: None, + subscriber, + path: Path::from_static("room"), + instance: None, + name: "video".to_string(), + }; + let broadcast = crate::broadcast::Info::new().produce(); + let request = broadcast.reserve_track("video").unwrap(); + let mut serving = ServeLoop::new(&serve, request, crate::track::Info::timed()); + let budget = Duration::from_millis(500); + let tail = kio::Producer::new(Tail::default()); + serving.sub = Sub::Active(SubStream { + order: crate::track::Order::Descending, + stream: Stream::open(&mut crate::driver::Transport::new(session.clone()), version) + .await + .unwrap(), + id: 0, + max_delay: budget, + min_delay: None, + start: None, + priority: 0, + requested: None, + ranges: vec![lite::Range::OLDEST], + tail: tail.clone(), + served: None, + started: false, + starts: Vec::new(), + end: None, + wanted: None, + unanswered: None, + updated: false, + _working: crate::session::Work::default().start(), + update: 0, + stale: RangeSet::new(), + }); + let mut reader = serving.serving.subscribe( + crate::track::Subscription::default() + .with_delay(crate::track::Delay::default().with_max(budget)) + .with_ranges([crate::track::Range::groups(0..)]), + ); + let waiter = kio::Waiter::noop(); + let code = |err: &crate::StreamError| u64::from(crate::coding::StreamCodes::encode_stream_code(&version, err)); + let drop = |start, end, err: &crate::StreamError, update| { + lite::SubscribeResponse::Drop(lite::SubscribeDrop { + start, + end, + error: code(err), + update, + }) + }; + let covers = |groups: std::ops::Range| tail.read().covers(groups); + + serving + .apply_response(&serve, drop(1, 2, &crate::StreamError::Old, 0)) + .unwrap(); + assert!(covers(1..3)); + assert_eq!( + reader.poll_drop(&waiter), + Poll::Ready((1..3, track::Hole::stale(budget))), + "judged under the budget sent with update 0" + ); + + // An update that keeps the budget and the ranges still reopens the stale run. + { + let Sub::Active(active) = &mut serving.sub else { + unreachable!("an active subscription"); + }; + buffer_update(active, &serving.serving, serve.subscriber.runtime.now()).unwrap(); + assert_eq!(active.update, 1); + assert!(active.stale.is_empty()); + } + assert!(!covers(1..3), "the publisher re-offers or drops it again"); + + // Still in flight from before the update: no verdict. + serving + .apply_response(&serve, drop(4, 4, &crate::StreamError::Old, 0)) + .unwrap(); + assert!(!covers(4..5), "still owed"); + assert!(reader.poll_drop(&waiter).is_pending(), "no reader settles on it"); + + // Any other drop is final, whatever update it names. + serving + .apply_response(&serve, drop(5, 5, &crate::StreamError::App(5), 0)) + .unwrap(); + assert!(covers(5..6)); + assert_eq!( + reader.poll_drop(&waiter), + Poll::Ready((5..6, track::Hole::drop(crate::StreamError::App(5)))) + ); + + // Under the current update, a stale drop is a verdict again. + serving + .apply_response(&serve, drop(6, 6, &crate::StreamError::Old, 1)) + .unwrap(); + assert!(covers(6..7)); + assert_eq!( + reader.poll_drop(&waiter), + Poll::Ready((6..7, track::Hole::stale(budget))) + ); + } + /// A SUBSCRIBE_DROP ends a group still arriving now, with its Error Code on lite-07. /// Earlier versions carry an application code there, so the same bytes read as /// OLD. @@ -4157,8 +4285,7 @@ mod tests { start: 2, end: 2, error: code.into(), - max_age: None, - terms: 0, + update: 0, }), ] { resp.encode( @@ -7538,12 +7665,12 @@ struct SubStream { /// Whether an update changed the ranges since the SUBSCRIBE, which the publisher may /// have read before its SUBSCRIBE_START, so the starts need not match the SUBSCRIBE. updated: bool, - /// The range terms id of the latest SUBSCRIBE or SUBSCRIBE_UPDATE sent: a stale drop - /// judged under an earlier one is no verdict on the current terms. - terms: u64, - /// The runs `tail` accounts for by a stale drop, with the max age each was judged under: - /// a widening past it reverses the verdict, so they are owed again. - stale: RangeMap, + /// The number of the latest SUBSCRIBE (0) or SUBSCRIBE_UPDATE sent: a stale drop given + /// up under an earlier one is no verdict on the current terms. + update: u64, + /// The runs `tail` accounts for by a stale drop under `update`: the next update reopens + /// them, since the publisher re-offers or drops them again under it. + stale: RangeSet, /// Holds the session busy for as long as the subscription is open. _working: crate::session::Working, } @@ -8350,31 +8477,24 @@ fn judged_end(ranges: &[lite::Range]) -> Option { .max() } -/// Buffer the current SUBSCRIBE parameters under new range terms. The caller flushes. +/// Buffer the current SUBSCRIBE parameters as the next update. The caller flushes. /// -/// A stale drop judged under earlier terms may still be in flight, and the update may have -/// moved the range it was judged against, so the runs `serving` holds as stale become -/// unknown again: a reader fills them by range rather than trust an old verdict. +/// A stale drop given up under an earlier update may still be in flight, and this one may +/// move the range or budget it was judged against, so the runs `serving` holds as stale +/// become unknown again: a reader fills them by range rather than trust an old verdict. /// -/// A lite-07 publisher re-offers or drops again what a widening reverses, so the tail owes -/// the stale runs judged under a narrower max age than this update's again, from `now`. +/// A lite-07 publisher re-offers every group it reported stale on any update, and drops +/// again under the new number what is still stale, so the tail owes every stale run again, +/// from `now`. fn buffer_update(active: &mut SubStream, serving: &track::Producer, now: crate::time::Instant) -> Result<(), Error> { - active.terms += 1; + active.update += 1; serving.supersede_stale(judged_end(&active.ranges)); - let reversed: Vec<_> = active - .stale - .iter() - .filter(|(_, judged)| **judged < active.max_delay) - .map(|(run, _)| run.clone()) - .collect(); + let reopened = std::mem::take(&mut active.stale); if let Ok(mut tail) = active.tail.write() { - for run in &reversed { + for run in reopened.iter() { tail.unaccount(run.clone(), now); } } - for run in reversed { - active.stale.remove(run); - } active.stream.writer.buffer(&lite::SubscribeUpdate { wanted_group: active.wanted, priority: active.priority, @@ -8382,7 +8502,7 @@ fn buffer_update(active: &mut SubStream, serving: &track::Producer, now: crate:: min_delay: active.min_delay, ranges: active.ranges.clone(), order: active.order, - terms: active.terms, + update: active.update, }) } @@ -8426,7 +8546,7 @@ impl Establish { min_delay: self.subscription.delay.min, ranges: self.ranges.clone(), order: self.subscription.order, - terms: 0, + update: 0, }; stream = stream.with_stats(self.serve.subscriber.origin.stats()); stream.writer.buffer(&lite::ControlType::Subscribe)?; @@ -8481,8 +8601,8 @@ impl Establish { end: None, wanted: None, updated: false, - terms: 0, - stale: RangeMap::new(), + update: 0, + stale: RangeSet::new(), _working: self.session.work().start(), } } @@ -9489,26 +9609,26 @@ impl ServeLoop { // one still arriving ends with the drop's error now. lite::SubscribeResponse::Drop(dropped) => { let groups = dropped.start..dropped.end.saturating_add(1); - // A stale verdict under terms an update has since replaced says nothing of the - // current ones: the groups are unknown, and a reader that wants one fills it by range. - let superseded = dropped.terms < active.terms; - // One the update widened past is served or dropped again, so it is still owed. - let reversed = superseded && dropped.max_age.is_some_and(|judged| judged < active.max_delay); - if !reversed && let Ok(mut tail) = active.tail.write() { - tail.account(groups.clone(), serve.subscriber.runtime.now()); - } - if let Some(judged) = dropped.max_age.filter(|_| !reversed) { - active.stale.insert(groups.clone(), judged); - } - match dropped.max_age { - Some(_) if superseded => {} - stale => self.serving.give_up( - groups, - track::Hole::Drop { - error: dropped.error(serve.subscriber.version), - stale, - }, - ), + let error = dropped.error(serve.subscriber.version); + // On lite-07 an OLD drop is a stale verdict, which the next update reopens; every + // other drop is final, whatever update it names. + let stale = serve.subscriber.version.has_skip() && error == crate::StreamError::Old; + // One given up under an update since replaced says nothing of the current terms: + // the publisher re-offers it or drops it again under the new number, so it is + // still owed, and a reader that wants it fills it by range meanwhile. + if !(stale && dropped.update < active.update) { + if let Ok(mut tail) = active.tail.write() { + tail.account(groups.clone(), serve.subscriber.runtime.now()); + } + let hole = match stale { + // Judged under the budget this subscriber sent with that update. + true => { + active.stale.insert(groups.clone()); + track::Hole::stale(active.max_delay) + } + false => track::Hole::drop(error), + }; + self.serving.give_up(groups, hole); } } // Never produced, so nothing is waited for and no reader ends. diff --git a/rs/moq-net/src/lite/version.rs b/rs/moq-net/src/lite/version.rs index 491779931d8..03b1f97b0fa 100644 --- a/rs/moq-net/src/lite/version.rs +++ b/rs/moq-net/src/lite/version.rs @@ -298,9 +298,9 @@ impl Version { } /// Whether holes split into SUBSCRIBE_SKIP, for sequences never produced, and - /// SUBSCRIBE_DROP, for groups that existed, whose drop names the `Max Age` and range - /// `Terms` a stale verdict was judged under; SUBSCRIBE and SUBSCRIBE_UPDATE carry those - /// `Terms`. Added in lite-07; earlier versions name every hole in SUBSCRIBE_DROP. + /// SUBSCRIBE_DROP, for groups that existed, which names the `Update` in effect when it was + /// given up; SUBSCRIBE and SUBSCRIBE_UPDATE carry that `Update` number. Added in lite-07; + /// earlier versions name every hole in SUBSCRIBE_DROP. pub(crate) fn has_skip(self) -> bool { !matches!( self, diff --git a/rs/moq-net/src/model/resume.rs b/rs/moq-net/src/model/resume.rs index 62abeab2fbb..8b749130fd5 100644 --- a/rs/moq-net/src/model/resume.rs +++ b/rs/moq-net/src/model/resume.rs @@ -1101,7 +1101,9 @@ struct Reader { out: Vec<(u64, Weak<()>)>, /// Runs the serving copy gave up that the reader has yet to report; see /// [`Reader::poll_drop`]. - drops: RangeMap, + /// Each with whether it is a received stale run the serving copy held, which new terms + /// can supersede. + drops: RangeMap, } impl Reader { @@ -1856,15 +1858,32 @@ impl Reader { /// so a run there waits until that copy no longer feeds. #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] fn poll_drop(&mut self, delivered: &Delivered, waiter: &kio::Waiter) -> Poll<(std::ops::Range, track::Hole)> { + let queued: Vec<_> = self + .drops + .iter() + .filter(|(_, (_, received))| *received) + .map(|(run, _)| run.clone()) + .collect(); let mut given_up = Vec::new(); + let mut superseded = Vec::new(); if let Poll::Ready(Some(sub)) = self.poll_serving(waiter) { - while let Poll::Ready(run) = sub.poll_drop(waiter) { - given_up.push(run); + // A received stale run queued here that new terms superseded since is unknown again: + // the copy no longer holds it, and its upstream re-offers or drops it again. + for run in queued { + superseded.extend(sub.stale_held(run.clone()).gaps(&run)); } + while let Poll::Ready((run, hole)) = sub.poll_drop(waiter) { + // A stale run the copy holds came from its upstream, which new terms may supersede. + let received = hole.stale_budget().is_some() && !sub.stale_held(run.clone()).is_empty(); + given_up.push((run, hole, received)); + } + } + for run in superseded { + self.drops.remove(run); } - for (run, hole) in given_up { + for (run, hole, received) in given_up { if !route_failed(&Error::Stream(hole.error())) { - self.drops.insert(run, hole); + self.drops.insert(run, (hole, received)); } } let covered: Vec<_> = self @@ -1892,7 +1911,7 @@ impl Reader { feeding.insert(from..until); } } - let next = self.drops.iter().find_map(|(run, err)| { + let next = self.drops.iter().find_map(|(run, (err, _))| { let run = run.start.max(start)..run.end.min(end.unwrap_or(u64::MAX)); let open = delivered .gaps(run) @@ -2597,6 +2616,13 @@ mod test { ) } + /// A route's copy received from a remote publisher, whose groups may land out of order. + fn received_copy() -> track::Producer { + let copy = copy(); + copy.received(); + copy + } + /// The logical track `routes` serves. fn logical(routes: &Producer) -> track::Producer { track::Request::new(Arc::new(broadcast::Info::default()), "video") @@ -3698,6 +3724,34 @@ mod test { assert_eq!(drain(&mut sub, false), Vec::::new()); } + /// A received stale run the reader held back is unknown again once new terms supersede it + /// on the serving copy: its upstream re-offers or drops it again, so it is not reported + /// meanwhile. + #[kio::test] + async fn a_held_back_stale_run_new_terms_supersede_is_not_reported() { + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + routes.serve(a.consume()); + let budget = Duration::from_secs(1); + let mut sub = subscribe(&logical, budget); + write(&a, 0); + assert_eq!(drain(&mut sub, false), [0]); + + let b = copy(); + routes.serve(b.consume()); + // The new route's upstream judged groups 1 and 2 stale; the old route may still deliver them. + b.give_up(1..3, track::Hole::stale(budget)); + assert_eq!(dropped(&mut sub), []); + write(&b, 3); + assert_eq!(drain(&mut sub, false), [3]); + + // The route's subscription updates before the old copy is past the budget. + b.supersede_stale(None); + kio::clock::sleep(budget).await; + assert_eq!(dropped(&mut sub), [], "no verdict under the new terms"); + } + /// A run a replaced route held back is dropped with that route: the route serving next /// may still deliver it. #[test] @@ -4632,7 +4686,7 @@ mod test { fn a_clean_end_waits_for_a_group_the_replaced_copy_still_carries() { let routes = Producer::new(); let logical = logical(&routes); - let a = copy(); + let a = received_copy(); routes.serve(a.consume()); let mut sub = subscribe(&logical, Duration::from_secs(1)); write(&a, 0); @@ -4660,7 +4714,7 @@ mod test { for ordered in [false, true] { let routes = Producer::new(); let logical = logical(&routes); - let a = copy(); + let a = received_copy(); let pool = a.broadcast().pool.clone(); let opened = a.consume().now(); let noop = kio::Waiter::noop(); @@ -4714,7 +4768,7 @@ mod test { fn a_read_waiting_past_a_drained_copy_wakes_when_the_route_changes() { let routes = Producer::new(); let logical = logical(&routes); - let a = copy(); + let a = received_copy(); routes.serve(a.consume()); let mut sub = subscribe(&logical, Duration::from_secs(1)); write(&a, 0); diff --git a/rs/moq-net/src/model/resume_model.rs b/rs/moq-net/src/model/resume_model.rs index 12ca4b57b74..71404241537 100644 --- a/rs/moq-net/src/model/resume_model.rs +++ b/rs/moq-net/src/model/resume_model.rs @@ -456,6 +456,8 @@ impl Model { let requests = self.origin.requests_route(PATH, route, None).unwrap(); let broadcast = broadcast::Info::new().produce(); let track = broadcast.create_track("video", track::Info::timed()).unwrap(); + // The route's live feed, received from its publisher: its groups land in any order. + track.received(); let upstream = broadcast.consume(); let serving = kio::clock::sim::spawn(async move { while let Ok(request) = kio::wait(|waiter| requests.poll_requested_broadcast(waiter)).await { diff --git a/rs/moq-net/src/model/skipped.rs b/rs/moq-net/src/model/skipped.rs index b39073a1bd2..e726c33773d 100644 --- a/rs/moq-net/src/model/skipped.rs +++ b/rs/moq-net/src/model/skipped.rs @@ -67,6 +67,19 @@ impl Skipped { widened } + /// The skipped runs within `groups`, clipped to them, lowest first; each stays until its + /// group is handed out or skipped again. + pub fn within(&self, groups: RangeInclusive) -> Vec> { + let Some(skipped) = &self.0 else { + return Vec::new(); + }; + skipped + .runs + .overlapping(&groups) + .map(|(run, _)| *run.start().max(groups.start())..=*run.end().min(groups.end())) + .collect() + } + /// Whether no run is recorded. pub fn is_empty(&self) -> bool { self.0.as_ref().is_none_or(|skipped| skipped.runs.is_empty()) diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index 8f6594344e6..b5027d9efd7 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -3425,6 +3425,10 @@ impl Producer { /// Create a new group with the given sequence number. /// + /// A gap it leaves above the last group is skipped at once: readers move past it without + /// waiting, and no reader ends. A group created in that gap later still reaches readers + /// that have not moved past it. + /// /// Readers and group demand reach it at once, but the track counts it as content (its /// live edge, or the successor that ages the group before it) only once its first frame /// opens or it finishes. @@ -3443,6 +3447,17 @@ impl Producer { if state.sealed || state.final_sequence().is_some_and(|fin| info.sequence >= fin) { return Err(Error::Closed); } + // What this track's own publisher passes over when it creates a group is skipped at + // once, never judged stale; a received copy's groups arrive in any order. A late group + // it still creates below wins over its skip, as a group stream wins over a drop. + // A run already given up keeps its reason. + if !state.received && received.is_none() && state.max_sequence.is_some() { + let next = state.next_sequence()?; + let gaps: Vec<_> = state.drops.gaps(&(next..info.sequence)).collect(); + for gap in gaps { + state.drop_groups(gap, Hole::Skip); + } + } let start = received.unwrap_or(0); let content = if received.is_some() { if state.holding_group(info.sequence, start).is_some() { @@ -3793,14 +3808,17 @@ impl Producer { /// Record that the publisher gave up `groups` with `err`: none will arrive, so a reader /// waiting on one moves past it, a group still arriving in them ends with `err`, the /// declared end does not wait for them, and a relay serving this track forwards the drop. - pub fn drop_groups(&self, groups: ops::Range, err: StreamError) { + pub(crate) fn drop_groups(&self, groups: ops::Range, err: StreamError) { self.give_up(groups, Hole::drop(err)); } - /// Record that the publisher never produced `groups`, such as skipped sequence numbers: - /// none will arrive, so a reader waiting on one moves past it, the declared end does not - /// wait for them, and a relay serving this track forwards the skip. Unlike - /// [`Self::drop_groups`], it ends no reader. + /// Record that the publisher never produced `groups`: none will arrive, so a reader + /// waiting on one moves past it, the declared end does not wait for them, and a relay + /// serving this track forwards the skip. Unlike a drop, it ends no reader. + /// + /// A gap below a created group is skipped already, since a publisher creates its groups + /// in order. This names the rest, such as sequences between the last group and an end + /// declared ahead of it that will never be produced. pub fn skip_groups(&self, groups: ops::Range) { self.give_up(groups, Hole::Skip); } @@ -6809,14 +6827,6 @@ struct Cursor { skipped: Skipped, /// The drops a wire publisher forwards, once it first polls for one. drops: Option>, - /// Whether a widening re-offers what this cursor reported stale: a wire whose drops a - /// widening reverses, unlike one whose drops are final. - reoffers_reported: bool, - /// The reported stale runs a widening took back since the owner last drained them. - unreported: Vec>, - /// The budget the reported stale runs were last checked against: only a change can - /// reverse one, so an unchanged budget costs nothing per poll. - reoffered_at: Option, /// What this cursor misses and asks fills for, once it first reads. misses: Option>, /// The heads a bounded window asked for, once it first finds a group missing one. @@ -6911,10 +6921,10 @@ struct DropCursor { /// Every sequence this cursor already reported. #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] reported: RangeSet, - /// The reported runs judged stale, with the budget each was judged under, for a wire - /// whose widening re-offers them; see [`Cursor::reoffers_reported`]. + /// The reported runs judged stale, for a wire whose update re-offers them; see + /// [`Subscriber::reoffer_stale`]. #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] - reported_stale: RangeMap, + reported_stale: RangeSet, } #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] @@ -6926,36 +6936,29 @@ impl DropCursor { pending: RangeMap::new(), scanned: None, reported: RangeSet::new(), - reported_stale: RangeMap::new(), + reported_stale: RangeSet::new(), } } - /// Take back the reported stale runs judged under a budget narrower than `budget`, and - /// withdraw those not reported yet: a widening reverses their verdict, so they are - /// judged again, and reported again if still given up. - fn widened(&mut self, budget: Duration) -> Vec> { - let pending: Vec<_> = self + /// Take back every stale run, reported or not: an update reopens their verdict, so they + /// are judged again, and reported again under the new update if still stale. + fn take_stale(&mut self) -> Vec> { + let mut stale: RangeSet = self .pending .iter() - .filter(|(_, pending)| pending.hole.stale_budget().is_some_and(|judged| judged < budget)) + .filter(|(_, pending)| pending.hole.stale_budget().is_some()) .map(|(run, _)| run.clone()) .collect(); - for run in pending { - self.pending.remove(run); + for run in stale.iter() { + self.pending.remove(run.clone()); } - let widened: Vec<_> = self - .reported_stale - .iter() - .filter(|(_, judged)| **judged < budget) - .map(|(run, _)| run.clone()) - .collect(); - // A wider budget changes the scan's key, so a received stale run it no longer settles is - // left out of the next scan. - for run in &widened { - self.reported_stale.remove(run.clone()); + for run in std::mem::take(&mut self.reported_stale).iter() { self.reported.remove(run.clone()); + stale.insert(run.clone()); } - widened + // A received stale run is folded in again by the next scan, if the track still holds it. + self.scanned = None; + stale.into_iter().collect() } /// Record a run this cursor's fills gave up on as a miss, as `hole` says; a reason @@ -7048,8 +7051,8 @@ impl DropCursor { } let hole = pending.hole; self.reported.insert(gap.clone()); - if let Some(budget) = hole.stale_budget() { - self.reported_stale.insert(gap.clone(), budget); + if hole.stale_budget().is_some() { + self.reported_stale.insert(gap.clone()); } return Some((gap, hole)); } @@ -7148,9 +7151,6 @@ impl Cursor { seek_pending: BTreeMap::new(), skipped: Skipped::default(), drops: None, - reoffers_reported: false, - unreported: Vec::new(), - reoffered_at: None, misses: None, heads: None, idle: None, @@ -7537,43 +7537,55 @@ impl Cursor { } /// Owe again the groups this cursor skipped as stale under a budget narrower than - /// `budget`: only a widening reverses a stale verdict. Costs the skipped runs, never - /// the cache, and nothing when the budget did not widen. + /// `budget`: only a widening reverses a stale verdict it has not reported. Costs the + /// skipped runs, never the cache, and nothing when the budget did not widen. fn reoffer(&mut self, budget: Duration) { - // A reported stale run reversed is judged again: a group still cached is owed below - // as a skipped one is, and a missing one is missed again. - if self.reoffers_reported - && self.reoffered_at.replace(budget) != Some(budget) - && let Some(drops) = &mut self.drops - { - for run in drops.widened(budget) { - if let Some(misses) = &mut self.misses { - misses.retry(run.clone()); - } - self.unreported.push(run); - } - } for run in self.skipped.widened(budget) { - self.filter - .written_off - .remove(*run.start()..run.end().saturating_add(1)); - // Judged again, so a drop not reported yet is withdrawn; a reported one stands. - if let Some(drops) = &mut self.drops { - let span = *run.start()..run.end().saturating_add(1); - let old: Vec<_> = drops - .pending - .overlapping(span.clone()) - .filter(|(_, pending)| pending.hole.stale_budget().is_some()) - .map(|(range, _)| range.start.max(span.start)..range.end.min(span.end)) - .collect(); - for range in old { - drops.pending.remove(range); + self.reowe(run); + } + } + + /// Owe again every run this cursor gave up as stale, reported or not: an update reopens + /// the verdict. A group still cached is owed as a widened skip is, a missing one is + /// missed again. Returns the runs taken back; whatever is still stale under the new + /// terms is given up and reported again. + fn reoffer_stale(&mut self) -> Vec> { + let Some(drops) = &mut self.drops else { + return Vec::new(); + }; + let runs = drops.take_stale(); + for run in &runs { + if let Some(misses) = &mut self.misses { + misses.retry(run.clone()); + } + if let Some(last) = run.end.checked_sub(1) { + for skipped in self.skipped.within(run.start..=last) { + self.reowe(skipped); } } - for owed in self.filter.admitted.overlapping(run.clone()) { - self.filter.owed.insert(owed); + } + runs + } + + /// Owe `run` of skipped groups again, judged anew, withdrawing a stale drop of it not + /// reported yet; a reported one stands. + fn reowe(&mut self, run: ops::RangeInclusive) { + let span = *run.start()..run.end().saturating_add(1); + self.filter.written_off.remove(span.clone()); + if let Some(drops) = &mut self.drops { + let old: Vec<_> = drops + .pending + .overlapping(span.clone()) + .filter(|(_, pending)| pending.hole.stale_budget().is_some()) + .map(|(range, _)| range.start.max(span.start)..range.end.min(span.end)) + .collect(); + for range in old { + drops.pending.remove(range); } } + for owed in self.filter.admitted.overlapping(run) { + self.filter.owed.insert(owed); + } } // A helper to automatically apply Dropped if the state is closed without an error. @@ -9272,24 +9284,14 @@ impl Subscriber { self } - /// Have a widening re-offer what this reader reported stale through - /// [`Self::poll_drop`], as a wire whose drops a widening reverses does: each run it - /// takes back is judged again, and [`Self::take_unreported`] hands it to the owner. A - /// front's reader reports only what its copies give up, so it keeps its reports. - #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] - pub(crate) fn reoffers_reported(mut self) -> Self { - if let Inner::Plain(cursor) = &mut self.inner { - cursor.reoffers_reported = true; - } - self - } - - /// The runs this reader reported stale that a widening took back since the last call, - /// which are owed again; see [`Self::reoffers_reported`]. + /// Re-offer what this reader gave up as stale through [`Self::poll_drop`], as a wire + /// whose every update reopens its stale drops does: each run is judged again under the + /// new terms. Returns the runs taken back, which are owed again. A front's reader + /// reports only what its copies give up, so it keeps its reports. #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] - pub(crate) fn take_unreported(&mut self) -> Vec> { + pub(crate) fn reoffer_stale(&mut self) -> Vec> { match &mut self.inner { - Inner::Plain(cursor) => std::mem::take(&mut cursor.unreported), + Inner::Plain(cursor) => cursor.reoffer_stale(), Inner::Resume(..) => Vec::new(), } } @@ -9419,6 +9421,24 @@ impl Subscriber { } } + /// The parts of `run` this subscriber's track still holds as a received stale run: one + /// new terms superseded is gone (see [`Producer::supersede_stale`]). + #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] + pub(crate) fn stale_held(&self, run: ops::Range) -> RangeSet { + match &self.inner { + Inner::Plain(cursor) => cursor + .state + .read() + .drops + .overlapping(&run) + .filter(|(_, hole)| hole.stale_budget().is_some()) + .map(|(held, _)| held.start.max(run.start)..held.end.min(run.end)) + .collect(), + // A front's reader checks its own routes' copies. + Inner::Resume(..) => RangeSet::new(), + } + } + /// The budget this subscriber judges staleness under: its max delay, clamped to the /// track's max age bound. #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] @@ -22032,6 +22052,8 @@ mod test { #[cfg(feature = "async")] fn a_reader_forgets_misses_behind_its_window() { let mut producer = track_producer("test", Info::timed()); + // A received copy, whose missing groups may still arrive. + producer.received(); let _dynamic = producer.requests(); let ascending = |groups| Subscription::default().with_range(groups).with_order(Order::Ascending); let mut subscriber = producer.subscribe(ascending(0..)); @@ -22132,6 +22154,8 @@ mod test { #[cfg(feature = "async")] fn an_end_ahead_of_the_edge_keeps_a_late_group() { let mut producer = track_producer("test", Info::timed()); + // A received copy, whose missing groups may still arrive. + producer.received(); let mut subscriber = producer.subscribe(replay().with_range(0..)); for sequence in [0, 1, 2, 4, 5] { write_group(&producer, sequence); @@ -23350,6 +23374,61 @@ mod test { ); } + /// A local publisher creating a group past the next sequence skips the gap at once: a + /// reader is told without waiting for any budget or grace, and no reader ends. A late + /// group the publisher still creates in the gap wins over the skip, as a group stream wins + /// over a drop, so a reader takes it once. + #[test] + #[cfg(feature = "async")] + fn a_local_gap_is_skipped_once_a_group_above_it_is_created() { + let producer = track_producer("test", Info::timed()); + let mut subscriber = producer.subscribe(replay().with_range(0..)); + let waiter = kio::Waiter::noop(); + write_group(&producer, 2); + write_group(&producer, 5); + assert_eq!(subscriber.poll_drop(&waiter), Poll::Ready((3..5, Hole::Skip))); + assert_eq!(drain_groups(&mut subscriber), [2, 5]); + + write_group(&producer, 4); + assert!( + !producer.state.read().drops.contains_key(&4), + "the group wins over its skip" + ); + assert_eq!(drain_groups(&mut subscriber), [4]); + let mut late = producer.subscribe(replay().with_range(0..)); + assert_eq!(drained_drops(&mut late), [(3, 4)]); + let mut read = drain_groups(&mut late); + read.sort(); + assert_eq!(read, [2, 4, 5]); + } + + /// Appending leaves no gap, and nothing below a track's first group is skipped. + #[test] + #[cfg(feature = "async")] + fn appending_and_a_first_group_skip_nothing() { + let producer = track_producer("test", Info::timed()); + let mut subscriber = producer.subscribe(replay().with_range(0..)); + write_group(&producer, 3); + producer.append_group().unwrap().finish().unwrap(); + assert!(producer.state.read().drops.is_empty()); + assert!(drained_drops(&mut subscriber).is_empty()); + } + + /// A received copy's groups land in any order, so a gap above its newest is still owed. + #[test] + #[cfg(feature = "async")] + fn a_received_gap_is_not_skipped() { + let producer = track_producer("test", Info::timed()); + producer.received(); + let mut subscriber = producer.subscribe(replay().with_range(0..)); + write_group(&producer, 2); + write_group(&producer, 5); + assert!(producer.state.read().drops.is_empty()); + assert!(drained_drops(&mut subscriber).is_empty()); + write_group(&producer, 3); + assert_eq!(drain_groups(&mut subscriber), [2, 5, 3]); + } + /// A received stale run superseded by new range terms is unknown again, and a group a /// widening re-offers wins over its drop. #[test] @@ -23794,6 +23873,9 @@ mod test { }]; for dropped in [false, true] { let producer = track_producer("test", Info::timed()); + // A received copy whose wire carries media time, so group 1 may still arrive. + producer.received(); + producer.anchored(); let mut zero = producer.create_group(group::Info { sequence: 0 }).unwrap(); zero.write_frame(at(0), bytes::Bytes::from_static(b"x")).unwrap(); let mut two = producer.create_group(group::Info { sequence: 2 }).unwrap(); @@ -24974,6 +25056,8 @@ mod test { #[cfg(feature = "async")] async fn a_bounded_read_ends_after_its_fill() { let (producer, _pool) = pooled_producer(1 << 40); + // A received copy, whose missing groups may still arrive. + producer.received(); let dynamic = producer.requests(); producer.create_group(0u64.into()).unwrap().finish().unwrap(); producer.create_group(2u64.into()).unwrap().finish().unwrap(); diff --git a/rs/moq-net/src/model/track/miss.rs b/rs/moq-net/src/model/track/miss.rs index 289b19e4109..4c6eb41cb62 100644 --- a/rs/moq-net/src/model/track/miss.rs +++ b/rs/moq-net/src/model/track/miss.rs @@ -1137,9 +1137,13 @@ impl Misses { } } - /// Judge `groups` again: a widening reversed the stale verdict they were given up with. + /// Judge `groups` again: an update reopened the stale verdict they were given up with. + /// They are no longer judged, and the next probe judges its whole span afresh, so it + /// asks a fill for them or gives them up again. pub(super) fn retry(&mut self, groups: ops::Range) { - self.given_up.remove(groups); + self.given_up.remove(groups.clone()); + self.judged.remove(groups); + self.at = None; } pub(super) fn give_up(&mut self, groups: ops::Range, given_up: &mut GivenUp) { @@ -1444,7 +1448,9 @@ mod tests { .unwrap(); let budget = Duration::from_secs(5); let _reader = track.subscribe(Subscription::default().with_delay(Delay::default().with_max(budget))); - for sequence in [0, 4] { + // Newest first, so the publisher skips nothing below it: group 2 is a live gap a fill + // may answer. + for sequence in [4, 0] { track .create_group(crate::group::Info { sequence }) .unwrap() @@ -1587,7 +1593,9 @@ mod tests { .unwrap(); let budget = Duration::from_secs(5); let _reader = track.subscribe(Subscription::default().with_delay(Delay::default().with_max(budget))); - for sequence in [0, 4] { + // Newest first, so the publisher skips nothing below it: group 2 is a live gap a fill + // may answer. + for sequence in [4, 0] { track .create_group(crate::group::Info { sequence }) .unwrap() @@ -1660,6 +1668,8 @@ mod tests { async fn unchanged_probes_and_late_arrivals_keep_other_gap_deadlines() { let broadcast = crate::broadcast::Info::default().produce(); let track = broadcast.create_track("sparse", crate::track::Info::default()).unwrap(); + // A received copy, whose live gaps are filled or waited for, not skipped. + track.received(); let budget = Duration::from_secs(5); let _reader = track.subscribe(Subscription::default().with_delay(Delay::default().with_max(budget))); for sequence in [0, 2, 4] { @@ -1760,6 +1770,31 @@ mod tests { "judged stale under the reader's budget" ); assert!(misses.deferred.runs.is_empty()); + + // An update reopens the verdict: retried, the gap is judged afresh, and a wider budget + // asks a fill for it. + misses.retry(3..4); + let wider = Drift { + budget: Duration::from_secs(3600), + bounded: false, + edge: state.drift_edge(&Servable::ALL), + coverage: None, + }; + given_up.clear(); + let asks = misses.judge( + &state, + (0, 0, None), + None, + None, + None, + |span| [span].into_iter().collect(), + &Delivered::default(), + &wider, + &mut given_up, + &waiter, + ); + assert!(asks.contains(&3), "a retried gap is asked again"); + assert!(given_up.is_empty()); } /// A grace change with nothing deferred has no deadline to move, so an unchanged probe @@ -1768,6 +1803,8 @@ mod tests { async fn a_grace_change_with_nothing_deferred_times_the_next_gap() { let broadcast = crate::broadcast::Info::default().produce(); let track = broadcast.create_track("grace", crate::track::Info::default()).unwrap(); + // A received copy, whose live gaps are filled or waited for, not skipped. + track.received(); let budget = Duration::from_secs(5); let _reader = track.subscribe(Subscription::default().with_delay(Delay::default().with_max(budget))); for sequence in 0..3 { @@ -2031,15 +2068,15 @@ mod tests { let mut track = broadcast.create_track("moved", crate::track::Info::default()).unwrap(); let budget = Duration::from_secs(5); let _reader = track.subscribe(Subscription::default().with_delay(Delay::default().with_max(budget))); - // A feed with holes, some of them under where it starts. - for sequence in 0..SEQUENCES { - if next(3) > 0 { - track - .create_group(crate::group::Info { sequence }) - .unwrap() - .finish() - .unwrap(); - } + // A feed with holes, some of them under where it starts, created newest first so the + // publisher skips none of them. + let sequences: Vec<_> = (0..SEQUENCES).filter(|_| next(3) > 0).collect(); + for &sequence in sequences.iter().rev() { + track + .create_group(crate::group::Info { sequence }) + .unwrap() + .finish() + .unwrap(); } if next(2) == 0 { track.start_at(next(SEQUENCES / 2)).unwrap(); From f0bc48ec5cde84c9567e8c010414de8ce90d38d0 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 16:43:27 -0800 Subject: [PATCH 13/68] fix(net): build drop_groups only for tests and fuzz benches Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/src/model/track.rs | 1 + 1 file changed, 1 insertion(+) diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index b5027d9efd7..dcb048c2d8e 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -3808,6 +3808,7 @@ impl Producer { /// Record that the publisher gave up `groups` with `err`: none will arrive, so a reader /// waiting on one moves past it, a group still arriving in them ends with `err`, the /// declared end does not wait for them, and a relay serving this track forwards the drop. + #[cfg(any(test, feature = "fuzz"))] pub(crate) fn drop_groups(&self, groups: ops::Range, err: StreamError) { self.give_up(groups, Hole::drop(err)); } From efcb809bafefd946b5165b609ee146570825855b Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 16:49:17 -0800 Subject: [PATCH 14/68] fix(net): drop expired streams again by run, not per sequence Co-Authored-By: Claude Opus 5.5 --- js/net/src/lite/publisher.ts | 20 +++++----- quest/m1/drop-set.md | 7 ++++ rs/moq-net/src/lite/publisher.rs | 63 ++++++++++++++++++++++++++++---- 3 files changed, 71 insertions(+), 19 deletions(-) diff --git a/js/net/src/lite/publisher.ts b/js/net/src/lite/publisher.ts index e090654184c..14e611c32f5 100644 --- a/js/net/src/lite/publisher.ts +++ b/js/net/src/lite/publisher.ts @@ -1342,8 +1342,8 @@ export class Publisher { // Every group this subscription started serving, until its stream finishes or resets. const groups = new Set>(); const drops = new Signal(0); - // Groups whose stream failed, with why, for the control stream to name. - const pendingDrops: { sequence: bigint; error: Error }[] = []; + // Groups whose stream failed, with why, for the control stream to name, as runs. + const pendingDrops: { start: bigint; end: bigint; error: Error }[] = []; // Lite-07 groups whose stream expired as stale, which every update drops again under its // number: the subscriber reopens each stale drop then, and the stream already reset. const expired: Run[] = []; @@ -1368,11 +1368,11 @@ export class Publisher { skipped.length = 0; } if (pendingDrops.length > 0) { - for (const { sequence, error } of pendingDrops.splice(0)) { + for (const { start, end, error } of pendingDrops.splice(0)) { // A stream expired under this subscription's budget is a stale verdict. if (hasSkip(this.version) && toStreamCode(error) === StreamCode.Old) - insertRun(expired, sequence, sequence + 1n); - const dropped = SubscribeDrop.given(sequence, sequence, error, this.version, updateNumber); + insertRun(expired, start, end); + const dropped = SubscribeDrop.given(start, end - 1n, error, this.version, updateNumber); if ( !(await controls.response(encodeSubscribeResponse(stream, { drop: dropped }, this.version))) ) @@ -1407,11 +1407,9 @@ export class Publisher { for (const run of reopened) served.splice(0, served.length, ...removeRun(served, run.start, run.end)); } - // A stream already reset as stale is not served again: dropped again. - for (const run of expired) { - for (let sequence = run.start; sequence < run.end; sequence++) - pendingDrops.push({ sequence, error: new StreamError(StreamCode.Old) }); - } + // A stream already reset as stale is not served again: dropped again, by run. + for (const run of expired) + pendingDrops.push({ ...run, error: new StreamError(StreamCode.Old) }); expired.length = 0; } console.debug(`subscribe update: broadcast=${broadcast} track=${track.name}`); @@ -1680,7 +1678,7 @@ export class Publisher { const task = opened.then(async (writer) => { const failed = writer instanceof Error ? writer : await this.#serveGroup(writer, options); if (emitRange && failed instanceof Error) { - pendingDrops.push({ sequence: group.sequence, error: failed }); + pendingDrops.push({ start: group.sequence, end: group.sequence + 1n, error: failed }); drops.update((revision) => revision + 1); } }); diff --git a/quest/m1/drop-set.md b/quest/m1/drop-set.md index 8021e4d1b08..1bc8f088a8f 100644 --- a/quest/m1/drop-set.md +++ b/quest/m1/drop-set.md @@ -155,6 +155,13 @@ Still to cover, beyond the memory work: copy with no media time asks a fill below the start even for a reader whose budget the publisher already exceeded. If the extra FETCH shows up in relay traffic, bound it with a run below the start carrying its budget. +- **Re-serve an expired stream on an update.** A lite-07 group whose stream + the publisher reset as OLD is dropped again under every later update + (`TrackRun::expired`, `expired` in the JS run loop), even when the new + terms would serve it: the cursor already handed it out (`Delivered` has no + removal) and its consumer latched the expiry. Once the disposition record + above records it as a stale run instead, an update re-offers it like any + other. - **A fill's stale verdict is final for everyone.** `TrackState::drop_unfed` (`rs/moq-net/src/model/track/fill.rs`) records what a fill's source gave up as a budget-independent drop, including OLD judged under that fill's own diff --git a/rs/moq-net/src/lite/publisher.rs b/rs/moq-net/src/lite/publisher.rs index ed671a8d4e5..718edd2932c 100644 --- a/rs/moq-net/src/lite/publisher.rs +++ b/rs/moq-net/src/lite/publisher.rs @@ -4122,6 +4122,8 @@ struct TrackRun { // Lite-07 groups whose stream expired as stale, which every update drops again under its // number: the subscriber reopens each stale drop then, and the stream already reset. expired: RangeSet, + // The expired runs an update owes a drop again, sent as ranges on the next poll. + redrop: RangeSet, end_sent: bool, // Lite-07 with every range bounded: once every group in its ranges is given up, it is // answered without a group to resolve its start, rather than left unanswered. @@ -4185,6 +4187,7 @@ impl TrackRun { skipped: RangeMap::new(), update: 0, expired: RangeSet::new(), + redrop: RangeSet::new(), end_sent: false, served, datagrams, @@ -4217,15 +4220,10 @@ impl TrackRun { if self.ctx.version.has_skip() { let reopened = self.track.reoffer_stale(); self.owe(reopened); - // A stream already reset as stale is not served again: dropped again below. + // A stream already reset as stale is not served again: dropped again, by run, on + // the next poll. let expired = std::mem::take(&mut self.expired); - let mut dropped = self.ctx.outcomes.dropped.lock().unwrap(); - dropped.extend( - expired - .iter() - .flat_map(|run| run.clone()) - .map(|sequence| (sequence, StreamError::Old)), - ); + self.redrop.extend(expired.iter().cloned()); } // Every update restates the wanted group. Past our largest it is a join, which // repeats the pending ask; at or below it the group already exists. @@ -4402,6 +4400,14 @@ impl TrackRun { } return Poll::Ready(Ok(ControlFlow::Continue(()))); } + if !self.redrop.is_empty() { + let hole = Hole::stale(self.track.budget()); + for run in std::mem::take(&mut self.redrop).iter() { + writer.buffer(&self.hole(run.start..=run.end - 1, &hole))?; + self.expired.insert(run.clone()); + } + return Poll::Ready(Ok(ControlFlow::Continue(()))); + } // Most wakes are a group machine's frame or the writer's credit. Nothing the track // side parked on fired then, so it would park again: skip it rather than lock the @@ -6932,6 +6938,47 @@ mod serve_group_test { } } + /// A group whose stream expired as stale is dropped again under every later update on + /// lite-07, by run, since the subscriber reopens each stale drop and the stream already + /// reset. Earlier versions' drops are final. + #[kio::test] + async fn an_update_drops_expired_streams_again_on_lite07_alone() { + for version in [Version::Lite06, Version::Lite07] { + let mut track = track::Producer::new( + Arc::new(broadcast::Info::default()), + "test", + crate::track::Info::timed(), + ); + let subscriber = track.subscribe(None); + let (mut run, mut writer, log) = lite_run(version, SinkSession::new(Log::default()), subscriber); + write_group(&mut track, 0, 0); + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + // Two group streams expired, as a blocked transport would leave them. + run.ctx + .outcomes + .dropped + .lock() + .unwrap() + .extend([(1, StreamError::Old), (2, StreamError::Old)]); + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + run.update(lite::SubscribeUpdate { + wanted_group: None, + priority: 1, + max_delay: Duration::from_secs(3600), + min_delay: None, + ranges: vec![lite::Range::span(None, 0, None, None)], + order: Order::Descending, + update: 1, + }); + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + let expected: &[(u64, u64, u64)] = match version { + Version::Lite07 => &[(1, 1, 0), (2, 2, 0), (1, 2, 1)], + _ => &[(1, 1, 0), (2, 2, 0)], + }; + assert_eq!(drop_updates(&mut writer, &log, version).await, expected, "{version:?}"); + } + } + /// Lite-05 and later name every sequence below the end that the subscription never /// got once the track ends: one skipped between groups, and one past the last. #[kio::test] From 47000a3d0fa8ea78f94e5ee64e4072a1f0144d4b Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 17:09:50 -0800 Subject: [PATCH 15/68] fix(net): re-offer a cached expired stream on a lite-07 update A group whose stream the publisher reset as stale is judged again under an update's new terms: one still cached, in range, and fresh is served again; the rest is dropped again under the new update, by run. Co-Authored-By: Claude Opus 5.5 --- js/net/src/internal.ts | 10 +++ js/net/src/lite/publisher.test.ts | 38 +++++++++ js/net/src/lite/publisher.ts | 41 +++++---- js/net/src/lite/widened_max_age.test.ts | 7 +- js/net/src/track.ts | 22 +++++ quest/m1/drop-set.md | 7 -- rs/moq-net/src/lite/publisher.rs | 105 +++++++++++++++++++++--- rs/moq-net/src/model/track.rs | 37 +++++++++ 8 files changed, 233 insertions(+), 34 deletions(-) diff --git a/js/net/src/internal.ts b/js/net/src/internal.ts index 5d3c47ab935..642e68da4fc 100644 --- a/js/net/src/internal.ts +++ b/js/net/src/internal.ts @@ -648,6 +648,13 @@ export const hooks: { * `track::Subscriber::reoffer_stale`. */ reofferStale: (subscriber: Subscriber) => Run[]; + /** + * Fresh reads of the groups in `[start, end)` a subscriber's track still caches, its range still + * admits, and its current budget no longer judges stale, each expiring against that budget, and + * the runs still given up: for a wire whose update re-offers a group whose stream it gave up as + * stale. Mirrors the Rust crate-private `track::Subscriber::reoffer_groups`. + */ + reofferGroups: (subscriber: Subscriber, start: bigint, end: bigint) => { groups: GroupConsumer[]; givenUp: Run[] }; /** * Whether a subscriber reads a copy received from a remote publisher, which cannot tell a group * its source never produced from one it never received. A resumed reader always does. @@ -874,6 +881,9 @@ export const hooks: { reofferStale: () => { throw new Error("track.ts not loaded"); }, + reofferGroups: () => { + throw new Error("track.ts not loaded"); + }, isReceived: () => { throw new Error("track.ts not loaded"); }, diff --git a/js/net/src/lite/publisher.test.ts b/js/net/src/lite/publisher.test.ts index 9de90901604..b117803cad8 100644 --- a/js/net/src/lite/publisher.test.ts +++ b/js/net/src/lite/publisher.test.ts @@ -1220,6 +1220,44 @@ test("lite draft-06: a group stream that never completes is reset once the group } }); +// Lite-07 reopens every stale drop on an update, including one for a stream the publisher reset +// as stale: a group still stale under the new terms is dropped again under the new update's +// number, so the subscriber settles at once. Mirrors the Rust +// `an_update_reoffers_a_cached_expired_stream_on_lite07`. +test("lite draft-07: an update drops an expired stream again while it is still stale", async () => { + const clock = new Manual(); + const installed = clock.install(); + const budget = Milli(1_000); + const sub = await servedSubscription({ version: Version.DRAFT_07, startGroup: 0n, maxAge: budget }); + try { + sub.serve(0n); + expect(await sub.servedSequence()).toBe(0n); + expect("start" in (await decodeSubscribeResponse(sub.client.reader, Version.DRAFT_07))).toBe(true); + + const reset = sub.stall(); + sub.serve(1n); + expect(await sub.servedHeader()).toBe(1n); + await flush(); + await clock.advance(Nano.fromMilli(Milli(budget * 10))); + sub.serve(2n); + expect(await sub.servedSequence()).toBe(2n); + await clock.advance(Nano.fromMilli(budget)); + expect(toStreamCode(fromTransport(await reset))).toBe(StreamCode.Old); + const first = await decodeSubscribeResponse(sub.client.reader, Version.DRAFT_07); + if (!("drop" in first)) throw new Error("expected SUBSCRIBE_DROP"); + expect([first.drop.start, first.drop.end, first.drop.update]).toEqual([1n, 1n, 0n]); + + // The track's max age still caps the budget, so group 1 is still stale. + await replayUpdate({ priority: 1, startGroup: 0n, update: 1n }).encode(sub.client.writer, Version.DRAFT_07); + const again = await decodeSubscribeResponse(sub.client.reader, Version.DRAFT_07); + if (!("drop" in again)) throw new Error("expected SUBSCRIBE_DROP"); + expect([again.drop.start, again.drop.end, again.drop.update]).toEqual([1n, 1n, 1n]); + } finally { + await sub.close(); + installed.close(); + } +}); + // An update that lowers the start below SUBSCRIBE_START asks for those groups too, so the // end sweep drops them, as the Rust publisher does. test("lite draft-06: an update lowering the start is swept at the end", async () => { diff --git a/js/net/src/lite/publisher.ts b/js/net/src/lite/publisher.ts index 14e611c32f5..fa5128b4051 100644 --- a/js/net/src/lite/publisher.ts +++ b/js/net/src/lite/publisher.ts @@ -1347,6 +1347,8 @@ export class Publisher { // Lite-07 groups whose stream expired as stale, which every update drops again under its // number: the subscriber reopens each stale drop then, and the stream already reset. const expired: Run[] = []; + // The expired groups an update re-offers, still cached and in range, served again in turn. + const reoffered: group.Consumer[] = []; // Cancels groups still queued for a stream slot. Only the subscriber leaving counts: // a track that ran out of groups still has to flush the ones already queued, and the @@ -1399,19 +1401,6 @@ export class Publisher { finished = false; const update = control.update; updateNumber = update.update; - // Lite-07 reopens every stale drop on any update: what was reported stale is - // offered again, and dropped again under this update's number if still stale. - if (hasSkip(this.version)) { - const reopened = hooks.reofferStale(track); - if (served) { - for (const run of reopened) - served.splice(0, served.length, ...removeRun(served, run.start, run.end)); - } - // A stream already reset as stale is not served again: dropped again, by run. - for (const run of expired) - pendingDrops.push({ ...run, error: new StreamError(StreamCode.Old) }); - expired.length = 0; - } console.debug(`subscribe update: broadcast=${broadcast} track=${track.name}`); // `cursorFloor` pins an unfloored pre-06 SUBSCRIBE to the latest // group, so an update that drops a named floor re-pins there too. On @@ -1440,6 +1429,26 @@ export class Publisher { bounds.endGroup = update.endGroup; bounds.endFrame = update.endFrame; moveCursor(); + // Lite-07 reopens every stale drop on any update, judged under its new terms + // (applied as the update was decoded): what was reported stale is offered + // again, and dropped again under this update's number if still stale. + if (hasSkip(this.version)) { + const owe = (start: bigint, end: bigint) => { + if (served) served.splice(0, served.length, ...removeRun(served, start, end)); + }; + for (const run of hooks.reofferStale(track)) owe(run.start, run.end); + // A group whose stream already reset as stale: one still cached, in range, and + // fresh under the new budget is served again; the rest is dropped again, by run. + for (const run of expired.splice(0)) { + const { groups, givenUp } = hooks.reofferGroups(track, run.start, run.end); + for (const group of groups) { + owe(group.sequence, group.sequence + 1n); + reoffered.push(group); + } + for (const gone of givenUp) + pendingDrops.push({ ...gone, error: new StreamError(StreamCode.Old) }); + } + } // A microtask is too short: decoding one framed update crosses several // awaits, each of which can requeue behind the serving continuation. A // task boundary usually lets the decoder finish whatever the transport @@ -1461,9 +1470,13 @@ export class Publisher { !startSent && hasLargest(this.version) ? newer(first?.largest, newer(hooks.bufferedLargest(track), hooks.peekLargest(track).largest)) : undefined; + // A group an update re-offers goes ahead of new ones. + const again = first ? undefined : reoffered.shift(); const recv = first ? ({ kind: "group", group: first.group } as const) - : hooks.tryRecvGroup(track, emitRange); + : again + ? ({ kind: "group", group: again } as const) + : hooks.tryRecvGroup(track, emitRange); first = undefined; switch (recv.kind) { case "error": diff --git a/js/net/src/lite/widened_max_age.test.ts b/js/net/src/lite/widened_max_age.test.ts index 5a221f36847..3519636a18b 100644 --- a/js/net/src/lite/widened_max_age.test.ts +++ b/js/net/src/lite/widened_max_age.test.ts @@ -2,7 +2,7 @@ // as stale, including those below where SUBSCRIBE_START resolved, and never one twice. import { expect, test } from "bun:test"; -import { Milli } from "@moq/clock"; +import { Milli, now } from "@moq/clock"; import { accept, connect } from "../connection/index.ts"; import { createMockTransportPair } from "../mock.ts"; import { Producer as OriginProducer } from "../origin.ts"; @@ -199,11 +199,12 @@ test("an update keeping the budget drops the stale groups again on lite-07", asy // Only the priority changes: groups 1 and 2 are still stale, so they are dropped again. reader.update({ priority: 5 }); - const before = performance.now(); + const before = now(); producer.finish(); producer.close(); expect(await reader.recvGroup()).toBeUndefined(); - expect(performance.now() - before).toBeLessThan(500); + // Well inside the 1.5s grace a missing re-drop would wait out, in nanoseconds. + expect(now() - before).toBeLessThan(500_000_000); reader.close(); remote.close(); diff --git a/js/net/src/track.ts b/js/net/src/track.ts index 3a8e1497ce3..d7e5d4b44c8 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -2408,6 +2408,27 @@ export class Subscriber { return taken; } + // Fresh reads of the cached groups in `[start, end)` this cursor admits and no longer judges + // stale, each expiring against the budget, and the runs still given up. A resumed reader holds + // no cache of its own, so it re-offers none. Mirrors the Rust `reoffer_groups`. + #reofferGroups(start: bigint, end: bigint): { groups: GroupConsumer[]; givenUp: Run[] } { + const groups: GroupConsumer[] = []; + const givenUp: Run[] = []; + const producer = this.#resume ? undefined : this.#state.producer; + const drift = this.#drift(); + const cap = this.#cursor.peek().end; + for (let sequence = start; sequence < end; sequence++) { + const admitted = (cap === undefined || sequence < cap) && this.#window.contains(sequence); + const group = + producer && admitted && !this.#staleAt(sequence, drift) + ? cachedMirror(producer, sequence, 0) + : undefined; + if (group) groups.push(this.#guard(group)); + else insertRun(givenUp, sequence, sequence + 1n); + } + return { groups, givenUp }; + } + private constructor(name: string, state: TrackState) { this.name = name; this.#state = state; @@ -2640,6 +2661,7 @@ export class Subscriber { subscriber.#reported = removeRun(subscriber.#reported, start, end); }; hooks.reofferStale = (subscriber) => subscriber.#reofferStale(); + hooks.reofferGroups = (subscriber, start, end) => subscriber.#reofferGroups(start, end); hooks.isReceived = (subscriber) => { if (subscriber.#resume) return true; const producer = subscriber.#state.producer; diff --git a/quest/m1/drop-set.md b/quest/m1/drop-set.md index 1bc8f088a8f..8021e4d1b08 100644 --- a/quest/m1/drop-set.md +++ b/quest/m1/drop-set.md @@ -155,13 +155,6 @@ Still to cover, beyond the memory work: copy with no media time asks a fill below the start even for a reader whose budget the publisher already exceeded. If the extra FETCH shows up in relay traffic, bound it with a run below the start carrying its budget. -- **Re-serve an expired stream on an update.** A lite-07 group whose stream - the publisher reset as OLD is dropped again under every later update - (`TrackRun::expired`, `expired` in the JS run loop), even when the new - terms would serve it: the cursor already handed it out (`Delivered` has no - removal) and its consumer latched the expiry. Once the disposition record - above records it as a stale run instead, an update re-offers it like any - other. - **A fill's stale verdict is final for everyone.** `TrackState::drop_unfed` (`rs/moq-net/src/model/track/fill.rs`) records what a fill's source gave up as a budget-independent drop, including OLD judged under that fill's own diff --git a/rs/moq-net/src/lite/publisher.rs b/rs/moq-net/src/lite/publisher.rs index 718edd2932c..edaa8265657 100644 --- a/rs/moq-net/src/lite/publisher.rs +++ b/rs/moq-net/src/lite/publisher.rs @@ -4124,6 +4124,9 @@ struct TrackRun { expired: RangeSet, // The expired runs an update owes a drop again, sent as ranges on the next poll. redrop: RangeSet, + // The expired groups an update re-offers, still cached and in range, each served again + // on its own stream as stream credit allows. + reoffered: std::collections::VecDeque, end_sent: bool, // Lite-07 with every range bounded: once every group in its ranges is given up, it is // answered without a group to resolve its start, rather than left unanswered. @@ -4188,6 +4191,7 @@ impl TrackRun { update: 0, expired: RangeSet::new(), redrop: RangeSet::new(), + reoffered: Default::default(), end_sent: false, served, datagrams, @@ -4215,16 +4219,6 @@ impl TrackRun { // notifies nothing it parked on. Idle::disarm(&mut self.idle); self.update = upd.update; - // Lite-07 reopens every stale drop on any update: what was reported stale is offered - // again, and dropped again under this update's number if still stale. - if self.ctx.version.has_skip() { - let reopened = self.track.reoffer_stale(); - self.owe(reopened); - // A stream already reset as stale is not served again: dropped again, by run, on - // the next poll. - let expired = std::mem::take(&mut self.expired); - self.redrop.extend(expired.iter().cloned()); - } // Every update restates the wanted group. Past our largest it is a join, which // repeats the pending ask; at or below it the group already exists. if upd @@ -4316,6 +4310,27 @@ impl TrackRun { } self.bounded = bounds.ends(self.ctx.version); self.frames = bounds.frames; + // Lite-07 reopens every stale drop on any update, judged under its new terms: what was + // reported stale is offered again, and dropped again under this update's number if + // still stale. + if self.ctx.version.has_skip() { + let reopened = self.track.reoffer_stale(); + self.owe(reopened); + // A group whose stream already reset as stale is judged again under the new terms: + // one still cached, in range, and fresh under the new budget is served again. The + // rest is dropped again, by run, on the next poll. + for run in std::mem::take(&mut self.expired).iter() { + let (groups, gone) = self.track.reoffer_groups(run.clone()); + self.owe( + groups + .iter() + .filter_map(|group| Some(group.sequence..group.sequence.checked_add(1)?)) + .collect(), + ); + self.reoffered.extend(groups); + self.redrop.extend(gone.iter().cloned()); + } + } } /// Apply [`Self::floor`] and [`Self::cap`] to the read cursor. @@ -4476,6 +4491,22 @@ impl TrackRun { return Poll::Ready(Ok(ControlFlow::Continue(()))); } + // A group an update re-offers takes the next stream credit, ahead of new ones. + if self.opening.is_none() + && let Some(group) = self.reoffered.pop_front() + { + let sequence = group.sequence; + match position_group(group, &self.frames) { + Some(group) => self.serve(group), + None => { + if let Some(end) = sequence.checked_add(1) { + self.drop_range(sequence..end, &Hole::drop(StreamError::NotFound), writer)?; + } + } + } + return Poll::Ready(Ok(ControlFlow::Continue(()))); + } + // One cursor drives the whole subscription: poll the cap-aware arrival-order // group and, when enabled, the next best-effort datagram. Groups are polled // first so a datagram burst can't starve them; datagrams flow whenever no @@ -6979,6 +7010,60 @@ mod serve_group_test { } } + /// A lite-07 group whose stream expired but that the track still caches is judged again + /// under an update: served again once the budget covers it, and dropped again under the + /// new update while it is still stale. + #[kio::test] + async fn an_update_reoffers_a_cached_expired_stream_on_lite07() { + for widen in [true, false] { + let mut track = track::Producer::new( + Arc::new(broadcast::Info::default()), + "test", + crate::track::Info::timed(), + ); + let budget = Duration::from_millis(500); + let subscriber = + track.subscribe(track::Subscription::default().with_delay(track::Delay::default().with_max(budget))); + let (mut run, mut writer, log) = lite07_run(SinkSession::new(Log::default()), subscriber); + write_group(&mut track, 0, 0); + write_group(&mut track, 1, 1000); + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + let opened = run.ctx.outcomes.opened.load(Ordering::Relaxed); + // Group 2 lands at 3s, so group 0's successor is 2s behind the edge, past the 500ms + // budget, and its stream expired. + write_group(&mut track, 2, 3000); + run.ctx.outcomes.dropped.lock().unwrap().push((0, StreamError::Old)); + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + assert_eq!(drop_updates(&mut writer, &log, Version::Lite07).await, [(0, 0, 0)]); + + run.update(lite::SubscribeUpdate { + wanted_group: None, + priority: 0, + max_delay: if widen { Duration::from_secs(10) } else { budget }, + min_delay: None, + ranges: vec![lite::Range::span(None, 0, None, None)], + order: Order::Descending, + update: 1, + }); + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + // Group 2 is served either way. + assert_eq!( + run.ctx.outcomes.opened.load(Ordering::Relaxed), + opened + 1 + u64::from(widen), + "widen={widen}: the cached group is offered again on its own stream once fresh" + ); + let drops: &[(u64, u64, u64)] = match widen { + true => &[(0, 0, 0)], + false => &[(0, 0, 0), (0, 0, 1)], + }; + assert_eq!( + drop_updates(&mut writer, &log, Version::Lite07).await, + drops, + "widen={widen}: still stale, it is dropped again under update 1" + ); + } + } + /// Lite-05 and later name every sequence below the end that the subscription never /// got once the track ends: one skipped between groups, and one past the last. #[kio::test] diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index dcb048c2d8e..fb352f9d7ae 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -9422,6 +9422,43 @@ impl Subscriber { } } + /// Fresh reads of the groups in `run` this subscriber's track still caches, its range + /// still admits, and its current budget no longer judges stale, each expiring against + /// that budget, and the runs still given up: no longer cached, or still stale. For a wire + /// whose update re-offers a group whose stream it gave up as stale. A front's reader holds + /// no cache of its own, so it re-offers none. + #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] + pub(crate) fn reoffer_groups(&self, run: ops::Range) -> (Vec, RangeSet) { + let mut given_up = RangeSet::new(); + given_up.insert(run.clone()); + let Inner::Plain(cursor) = &self.inner else { + return (Vec::new(), given_up); + }; + let budget = cursor.subscription.read().budget(); + let fresh: Vec<_> = { + let state = cursor.state.read(); + let budget = clamp_max_delay(budget, state.max_age_bound()); + let edge = state.drift_edge(&cursor.servable); + state + .lookup + .range(run) + .filter(|&(&sequence, slot)| { + slot.serves_end() && cursor.admits(sequence) && !state.is_stale(sequence, &edge, budget) + }) + .map(|(&sequence, slot)| (sequence, slot.serving().consume())) + .collect() + }; + let mut groups = Vec::new(); + for (sequence, group) in fresh { + if let Some(end) = sequence.checked_add(1) { + given_up.remove(sequence..end); + group.cache_refresh(); + groups.push(cursor.deliver(group)); + } + } + (groups, given_up) + } + /// The parts of `run` this subscriber's track still holds as a received stale run: one /// new terms superseded is gone (see [`Producer::supersede_stale`]). #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] From 5552dc2f01dc2f2a624e7ccc75c7eeed4a044fea Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 17:14:08 -0800 Subject: [PATCH 16/68] fix(net): re-offer expired streams through a front's serving copy A relay reading through a front re-offers an expired stream's group from the serving route's copy, and JS judges only the cached groups of an expired run rather than every sequence in it. Co-Authored-By: Claude Opus 5.5 --- js/net/src/internal.ts | 2 ++ js/net/src/resume.test.ts | 19 ++++++++++++++++ js/net/src/resume.ts | 12 ++++++++++ js/net/src/track.ts | 37 +++++++++++++++++++++---------- rs/moq-net/src/model/resume.rs | 40 ++++++++++++++++++++++++++++++++++ rs/moq-net/src/model/track.rs | 13 ++++++----- 6 files changed, 106 insertions(+), 17 deletions(-) diff --git a/js/net/src/internal.ts b/js/net/src/internal.ts index 642e68da4fc..8ab12ba4100 100644 --- a/js/net/src/internal.ts +++ b/js/net/src/internal.ts @@ -434,6 +434,8 @@ export interface ResumeSource { fork(options?: Subscription): Subscriber; /** A fresh reader for a group still cached by one of this reader's copies. */ cachedGroup(sequence: bigint, from?: number): GroupConsumer | undefined; + /** See {@link hooks.reofferGroups}, answered by the serving copy. */ + reofferGroups(start: bigint, end: bigint): { groups: GroupConsumer[]; givenUp: Run[] }; /** Let go of every copy. */ close(abort?: Error): void; } diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts index 0e4fc8b0780..2dfcbd846bf 100644 --- a/js/net/src/resume.test.ts +++ b/js/net/src/resume.test.ts @@ -3329,3 +3329,22 @@ test("a route whose IETF session closed before subscribing moves the track to th front.consumer.close(); b.close(); }); + +// A front's reader re-offers what its serving route's copy still caches and judges fresh, and gives +// up the rest, so a relay's publisher re-serves an expired stream through it. Mirrors Rust. +test("a resumed reader re-offers a group its serving copy still caches", () => { + const front = new Front(); + const a = new BroadcastProducer(); + const ta = copy(a); + front.serve(a.consume()); + const sub = front.consumer.track("seq").subscribe({ delay: { max: Milli(60_000) } }); + ta.write(0n); + expect(drain(sub)).toEqual([0n]); + const { groups, givenUp } = hooks.reofferGroups(sub, 0n, 3n); + expect(groups.map((group) => group.sequence)).toEqual([0n]); + expect(givenUp).toEqual([{ start: 1n, end: 3n }]); + for (const group of groups) group.close(); + sub.close(); + front.consumer.close(); + a.close(); +}); diff --git a/js/net/src/resume.ts b/js/net/src/resume.ts index c9c6c6bc8ad..c0f4d27d60f 100644 --- a/js/net/src/resume.ts +++ b/js/net/src/resume.ts @@ -595,6 +595,18 @@ class Reader implements ResumeSource { return holding; } + /** + * Fresh reads of the groups in `[start, end)` the serving copy still caches and judges fresh, + * handed out as any group of that copy is, and the runs still given up: all of them while no + * copy serves. Mirrors the Rust `Reader::reoffer_groups`. + */ + reofferGroups(start: bigint, end: bigint): { groups: GroupConsumer[]; givenUp: Run[] } { + const serving = this.#serving(); + if (!serving) return { groups: [], givenUp: [{ start, end }] }; + const { groups, givenUp } = hooks.reofferGroups(serving.sub, start, end); + return { groups: groups.map((group) => this.#handOut(serving, group)), givenUp }; + } + cachedGroup(sequence: bigint, from = 0): GroupConsumer | undefined { for (let index = this.#copies.length - 1; index >= 0; index--) { const copy = this.#copies[index]; diff --git a/js/net/src/track.ts b/js/net/src/track.ts index d7e5d4b44c8..8a1cafeae1b 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -741,6 +741,9 @@ type Start = { sequence: bigint; start: Timestamp }; let cachedMirror: (producer: Producer, sequence: bigint, from: number) => GroupConsumer | undefined; // Whether the producer still caches a group that has not failed; assigned in its static block. let cachedHolds: (producer: Producer, sequence: bigint) => boolean; +// The sequences in `[start, end)` the producer caches, unsorted, costing the smaller of the span +// and the cache; assigned in its static block. +let cachedIn: (producer: Producer, start: bigint, end: bigint) => bigint[]; // Mirror the cached groups in `[start, end)` that `offer` accepts into a sink again, cut ones // included as a fresh sink gets them, for a reader whose floor moved below them; assigned in // the producer's static block. @@ -993,6 +996,15 @@ export class Producer { const entry = producer.#cached.get(sequence); return entry !== undefined && !(entry.group.closed.peek() instanceof Error); }; + cachedIn = (producer, start, end) => { + if (end - start <= BigInt(producer.#cached.size)) { + const found: bigint[] = []; + for (let sequence = start; sequence < end; sequence++) + if (producer.#cached.has(sequence)) found.push(sequence); + return found; + } + return [...producer.#cached.keys()].filter((sequence) => sequence >= start && sequence < end); + }; cachedMirror = (producer, sequence, from) => { const entry = producer.#cached.get(sequence); if (!entry || entry.group.closed.peek() instanceof Error) return undefined; @@ -2409,22 +2421,25 @@ export class Subscriber { } // Fresh reads of the cached groups in `[start, end)` this cursor admits and no longer judges - // stale, each expiring against the budget, and the runs still given up. A resumed reader holds - // no cache of its own, so it re-offers none. Mirrors the Rust `reoffer_groups`. + // stale, each expiring against the budget, and the runs still given up. A resumed reader asks + // its serving route's copy. Mirrors the Rust `reoffer_groups`. #reofferGroups(start: bigint, end: bigint): { groups: GroupConsumer[]; givenUp: Run[] } { + if (this.#resume) return this.#resume.reofferGroups(start, end); const groups: GroupConsumer[] = []; - const givenUp: Run[] = []; - const producer = this.#resume ? undefined : this.#state.producer; + let givenUp: Run[] = [{ start, end }]; + const producer = this.#state.producer; + if (!producer) return { groups, givenUp }; const drift = this.#drift(); const cap = this.#cursor.peek().end; - for (let sequence = start; sequence < end; sequence++) { + // Only what the cache holds, so a long run of groups long gone costs nothing. + const cached = cachedIn(producer, start, end).sort((a, b) => (a < b ? -1 : a > b ? 1 : 0)); + for (const sequence of cached) { const admitted = (cap === undefined || sequence < cap) && this.#window.contains(sequence); - const group = - producer && admitted && !this.#staleAt(sequence, drift) - ? cachedMirror(producer, sequence, 0) - : undefined; - if (group) groups.push(this.#guard(group)); - else insertRun(givenUp, sequence, sequence + 1n); + if (!admitted || this.#staleAt(sequence, drift)) continue; + const group = cachedMirror(producer, sequence, 0); + if (!group) continue; + groups.push(this.#guard(group)); + givenUp = removeRun(givenUp, sequence, sequence + 1n); } return { groups, givenUp }; } diff --git a/rs/moq-net/src/model/resume.rs b/rs/moq-net/src/model/resume.rs index 8b749130fd5..d037462df38 100644 --- a/rs/moq-net/src/model/resume.rs +++ b/rs/moq-net/src/model/resume.rs @@ -894,6 +894,13 @@ impl Subscriber { reader.window.covered_by(from, cap, accounted) } + /// Fresh reads of the groups in `run` the serving route's copy still caches and judges + /// fresh, and the runs still given up; see [`track::Subscriber::reoffer_groups`]. + #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] + pub(super) fn reoffer_groups(&mut self, run: std::ops::Range) -> (Vec, RangeSet) { + self.reader().reoffer_groups(run) + } + /// Poll for sequences the serving copy gave up that no copy may still deliver; see /// [`track::Subscriber::poll_drop`]. #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] @@ -1851,6 +1858,23 @@ impl Reader { } } + /// Fresh reads of the groups in `run` the serving copy still caches and judges fresh, + /// handed out as any group of that copy is, and the runs still given up: all of `run` + /// while no copy serves. + #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] + fn reoffer_groups(&mut self, run: std::ops::Range) -> (Vec, RangeSet) { + let Poll::Ready(Some(sub)) = self.poll_serving(&kio::Waiter::noop()) else { + let mut given_up = RangeSet::new(); + given_up.insert(run); + return (Vec::new(), given_up); + }; + let (groups, given_up) = sub.reoffer_groups(run); + // The serving copy is the newest one. + let index = self.copies.len() - 1; + let groups = groups.into_iter().map(|group| self.hand_out(index, group)).collect(); + (groups, given_up) + } + /// The next run the serving copy gave up that `delivered` lacks and no other copy may /// still deliver. Only the serving copy's verdicts count, and a route failure is none: /// another route may still deliver the group, and a failed group it buffered continues @@ -3752,6 +3776,22 @@ mod test { assert_eq!(dropped(&mut sub), [], "no verdict under the new terms"); } + /// A front's reader re-offers what its serving route's copy still caches and judges fresh, + /// and gives up the rest, so a relay's publisher re-serves an expired stream through it. + #[test] + fn a_resumed_reader_reoffers_a_group_its_serving_copy_still_caches() { + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + routes.serve(a.consume()); + let mut sub = subscribe(&logical, HOUR); + write(&a, 0); + assert_eq!(drain(&mut sub, false), [0]); + let (groups, given_up) = sub.reoffer_groups(0..3); + assert_eq!(groups.iter().map(|group| group.sequence).collect::>(), [0]); + assert_eq!(given_up.iter().cloned().collect::>(), [1..3]); + } + /// A run a replaced route held back is dropped with that route: the route serving next /// may still deliver it. #[test] diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index fb352f9d7ae..c579b67b5de 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -9425,15 +9425,16 @@ impl Subscriber { /// Fresh reads of the groups in `run` this subscriber's track still caches, its range /// still admits, and its current budget no longer judges stale, each expiring against /// that budget, and the runs still given up: no longer cached, or still stale. For a wire - /// whose update re-offers a group whose stream it gave up as stale. A front's reader holds - /// no cache of its own, so it re-offers none. + /// whose update re-offers a group whose stream it gave up as stale. A front's reader asks + /// its serving route's copy. #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] - pub(crate) fn reoffer_groups(&self, run: ops::Range) -> (Vec, RangeSet) { + pub(crate) fn reoffer_groups(&mut self, run: ops::Range) -> (Vec, RangeSet) { + let cursor = match &mut self.inner { + Inner::Plain(cursor) => cursor, + Inner::Resume(resume, _) => return resume.reoffer_groups(run), + }; let mut given_up = RangeSet::new(); given_up.insert(run.clone()); - let Inner::Plain(cursor) = &self.inner else { - return (Vec::new(), given_up); - }; let budget = cursor.subscription.read().budget(); let fresh: Vec<_> = { let state = cursor.state.read(); From 85fe17fb370aaa32e6d0dcede7767632358678d9 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 17:22:55 -0800 Subject: [PATCH 17/68] test(net): compare the re-offered run as a set Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/src/model/resume.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rs/moq-net/src/model/resume.rs b/rs/moq-net/src/model/resume.rs index d037462df38..421a4f3cfea 100644 --- a/rs/moq-net/src/model/resume.rs +++ b/rs/moq-net/src/model/resume.rs @@ -3789,7 +3789,7 @@ mod test { assert_eq!(drain(&mut sub, false), [0]); let (groups, given_up) = sub.reoffer_groups(0..3); assert_eq!(groups.iter().map(|group| group.sequence).collect::>(), [0]); - assert_eq!(given_up.iter().cloned().collect::>(), [1..3]); + assert_eq!(given_up, std::iter::once(1..3).collect::>()); } /// A run a replaced route held back is dropped with that route: the route serving next From 228a8ce94c2280d2cf3ef5c3f67d4571ad950831 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 18:30:47 -0800 Subject: [PATCH 18/68] feat(net)!: SKIP only from an explicit skip_groups or append_group A gap between created groups tells moq-net nothing, since producers legitimately create groups out of order (claim_sequence, seek, late lower groups). So SKIP now comes only from the producer: the public `track::Producer::skip_groups`, or `append_group`, which means "the next group" and skips every sequence below it that never arrived and was not given up since the last append. `create_group` implies no skip, and the finish-time inference for never-arrived sequences is gone: an undeclared gap is judged stale or dropped NOT_FOUND. A skip is final: creating a group, appending one, or appending a datagram at a skipped sequence is refused with `Error::Skipped` (JS `Skipped`). Rust and JS mirror each other (`skipGroups`), and the lite draft says SKIP names only sequences the publisher declared it will never produce. Co-Authored-By: Claude Opus 5.5 --- drafts/draft-lcurley-moq-lite.md | 22 ++- js/net/src/error.ts | 19 ++ js/net/src/errors.ts | 1 + js/net/src/internal.ts | 8 - js/net/src/lite/publisher.test.ts | 64 ++++--- js/net/src/lite/publisher.ts | 11 +- js/net/src/lite/subscribe.ts | 6 +- js/net/src/lite/version.ts | 8 +- js/net/src/track.test.ts | 80 ++++++-- js/net/src/track.ts | 99 ++++++++-- quest/m1/drop-set.md | 59 ++++-- quest/m1/time-addressing/model.md | 19 +- rs/moq-archive/src/consumer/track.rs | 5 +- rs/moq-net/src/error.rs | 7 + rs/moq-net/src/lite/publisher.rs | 57 ++++-- rs/moq-net/src/lite/subscribe.rs | 6 +- rs/moq-net/src/lite/version.rs | 2 +- rs/moq-net/src/model/track.rs | 265 ++++++++++++++++++++------- rs/moq-net/src/model/track/hole.rs | 12 +- 19 files changed, 547 insertions(+), 203 deletions(-) diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index e55d2787232..4751d1f375a 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -335,7 +335,7 @@ Sent when resetting a stream (RESET_STREAM), or when refusing to receive one (ST | ------- | ------------- | ----------- | | 0x32 | GROUP_TOO_LARGE | The group grew past the publisher's cache budget and was aborted. | | ------- | ------------- | ----------- | -| 0x33 | NOT_FOUND | The requested group, track, or broadcast is not here. A sequence its publisher never produced is named by SUBSCRIBE_SKIP instead; a relay that cannot tell whether its source produced one names it with NOT_FOUND. | +| 0x33 | NOT_FOUND | The requested group, track, or broadcast is not here. A sequence its publisher declared it will never produce is named by SUBSCRIBE_SKIP instead; any other sequence a publisher never had, including a gap it never declared, is named with NOT_FOUND. | | ------- | ------------- | ----------- | | 0x34 | OLD | The group was superseded by a newer group and dropped. In SUBSCRIBE_DROP, the next SUBSCRIBE_UPDATE reopens it: the publisher re-offers the group or drops it again. | | ------- | ------------- | ----------- | @@ -1569,8 +1569,8 @@ The publisher SHOULD send SUBSCRIBE_FINISH as soon as the exclusive final group ## SUBSCRIBE_DROP {#subscribe-drop} A publisher names groups that existed but that it will not deliver with SUBSCRIBE_DROP on the Subscribe Stream: an aborted or reset group, one evicted or expired with no copy left to serve it, or one past the subscriber's budget (OLD). -Sequences the publisher never produced are named by [SUBSCRIBE_SKIP](#subscribe-skip) instead. -A relay cannot tell a sequence its source never produced from one it never received, so it names a sequence it never got in SUBSCRIBE_DROP with NOT_FOUND. +Sequences the publisher declared it will never produce are named by [SUBSCRIBE_SKIP](#subscribe-skip) instead. +A sequence the publisher never had but never declared skipped, such as a gap between the Groups it created or one a relay never received, might have existed, so it is named in SUBSCRIBE_DROP: with OLD when it is past the subscriber's budget, otherwise with NOT_FOUND. ~~~ SUBSCRIBE_DROP Message { @@ -1612,7 +1612,7 @@ Before moq-lite-07, SUBSCRIBE_DROP carries no `Update`, its Error Code is applic ## SUBSCRIBE_SKIP {#subscribe-skip} -A publisher names sequences it never produced with SUBSCRIBE_SKIP on the Subscribe Stream: skipped sequence numbers, or ones past a cut. +A publisher names sequences it declared it will never produce with SUBSCRIBE_SKIP on the Subscribe Stream. ~~~ SUBSCRIBE_SKIP Message { @@ -1623,13 +1623,17 @@ SUBSCRIBE_SKIP Message { } ~~~ -**Group Start** and **Group End** are the inclusive range of sequences that never held a group. +**Group Start** and **Group End** are the inclusive range of sequences that will never hold a group. `Group Start = Group End + 1` is an empty range; a larger Group Start is a PROTOCOL_VIOLATION. -A skip is final: no Group will ever exist at those sequences, whatever the terms. +A skip names only sequences the original publisher declared it will never produce, and it is final: no Group will ever exist at those sequences, whatever the terms. +A publisher MUST NOT later create a Group at a sequence it skipped, and a subscriber MUST NOT deliver a Group Stream for one. It carries no error and ends no reader, since there is nothing to read; a subscriber stops waiting for those sequences. -A publisher SHOULD name a gap it leaves in its own sequence numbers as soon as it creates the Group above it, rather than judge the gap stale later. -A relay forwards it unchanged; a relay that cannot tell whether its source produced a sequence uses SUBSCRIBE_DROP instead (see [SUBSCRIBE_DROP](#subscribe-drop)). + +A gap in the sequence numbers says nothing on its own: a publisher may create Groups out of order, or create again a sequence whose Group it aborted, so it never skips a sequence merely because a higher Group exists or the Track ended without it. +Only the application producing the Track knows a sequence will never exist, for example when it numbers Groups sparsely, or when it moves on to the next Group and so declares that it will not go back to fill the sequences below. +A publisher SHOULD send the skip as soon as it declares it. +A relay forwards a skip it received unchanged, and never skips a sequence on its own: one it never received is named in SUBSCRIBE_DROP (see [SUBSCRIBE_DROP](#subscribe-drop)). ## FETCH FETCH is sent by a subscriber to request a single group from a track. @@ -1955,7 +1959,7 @@ The `Message Length` describes the payload size on the wire. - Restored SUBSCRIBE_DROP (type 0x2), replacing Stream Count on SUBSCRIBE_FINISH. Every sequence is accounted for by a Group Stream, a datagram, or a drop; a publisher SHOULD send a drop as soon as it gives a sequence up. - SUBSCRIBE_DROP's Error Code is a stream error code, which ends the dropped groups' readers, including one still arriving. - Refuse a SUBSCRIBE_DROP whose Group Start is greater than Group End plus one on every version that carries it. -- Added SUBSCRIBE_SKIP (type 0x3) for sequences the publisher never produced, which ends no reader and is final. SUBSCRIBE_DROP now names only groups that existed. +- Added SUBSCRIBE_SKIP (type 0x3) for sequences the publisher declared it will never produce, which ends no reader and is final: a Group is never created at a skipped sequence. A gap in the sequence numbers is not a skip, since Groups may be created out of order; a sequence never had and never declared skipped is named in SUBSCRIBE_DROP. SUBSCRIBE_DROP no longer names declared skips. - Added `Update` to SUBSCRIBE, SUBSCRIBE_UPDATE, and SUBSCRIBE_DROP: the number of the terms in effect. A stale (OLD) drop holds only under the update it names, judged against the budget the subscriber sent with it; every later SUBSCRIBE_UPDATE reopens it, and the publisher re-offers the group or drops it again under the new `Update`. Any other drop is final. A Group Stream wins over its earlier drop. - SUBSCRIBE_FINISH declares the final boundary while lower groups can still be in flight, and no longer closes the Subscribe Stream: the publisher keeps reading SUBSCRIBE_UPDATE, so an update can reach an ended Track's history. - A covered subscription, whose every Range has an End and whose every Group was sent or dropped, also stays open for an update; the publisher never FINs first. diff --git a/js/net/src/error.ts b/js/net/src/error.ts index 5e215d0df38..cd36c735389 100644 --- a/js/net/src/error.ts +++ b/js/net/src/error.ts @@ -405,6 +405,25 @@ function localStreamCode(err: unknown): StreamCode { return StreamCode.Internal; } +/** + * A group was created at a sequence its track declared skipped. + * + * A skip says the sequence will never be produced, so it is final. See + * {@link TrackProducer.skipGroups}. Mirrors the Rust `Error::Skipped`. + * + * @public + */ +export class Skipped extends Error { + /** The sequence that was refused. */ + readonly sequence: bigint; + + constructor(sequence: bigint) { + super(`sequence ${sequence} was skipped`); + this.name = "Skipped"; + this.sequence = sequence; + } +} + /** * The transport refused a request stream or failed to grant stream credit before its deadline. * diff --git a/js/net/src/errors.ts b/js/net/src/errors.ts index b8421c2a0fb..a993fe7b8f3 100644 --- a/js/net/src/errors.ts +++ b/js/net/src/errors.ts @@ -12,6 +12,7 @@ export { ProtocolViolation, RefusedRedirect, Session, + Skipped, Stream, type StreamOptions, TimestampMismatch, diff --git a/js/net/src/internal.ts b/js/net/src/internal.ts index 8ab12ba4100..2b9b7f069bf 100644 --- a/js/net/src/internal.ts +++ b/js/net/src/internal.ts @@ -657,11 +657,6 @@ export const hooks: { * stale. Mirrors the Rust crate-private `track::Subscriber::reoffer_groups`. */ reofferGroups: (subscriber: Subscriber, start: bigint, end: bigint) => { groups: GroupConsumer[]; givenUp: Run[] }; - /** - * Whether a subscriber reads a copy received from a remote publisher, which cannot tell a group - * its source never produced from one it never received. A resumed reader always does. - */ - isReceived: (subscriber: Subscriber) => boolean; /** Whether this subscriber's budget gives up on `sequence`, judged by its successor; see {@link expireGroup}. */ staleAt: (subscriber: Subscriber, sequence: bigint) => Expiry; /** A group the subscriber's track still caches, as a fresh read handle. */ @@ -886,9 +881,6 @@ export const hooks: { reofferGroups: () => { throw new Error("track.ts not loaded"); }, - isReceived: () => { - throw new Error("track.ts not loaded"); - }, tryRecvDatagram: () => { throw new Error("track.ts not loaded"); }, diff --git a/js/net/src/lite/publisher.test.ts b/js/net/src/lite/publisher.test.ts index b117803cad8..a4687adcf0f 100644 --- a/js/net/src/lite/publisher.test.ts +++ b/js/net/src/lite/publisher.test.ts @@ -1031,36 +1031,42 @@ for (const version of [Version.DRAFT_06, Version.DRAFT_07]) { } }); - test(`lite ${version}: the end sweep names a group never produced ${version === Version.DRAFT_07 ? "in SUBSCRIBE_SKIP" : "in SUBSCRIBE_DROP"}`, async () => { - const sub = await servedSubscription({ version, startGroup: 5n }); - try { - sub.serve(5n); - expect(await sub.servedSequence()).toBe(5n); - await replayUpdate({ priority: 0, startGroup: 2n }).encode(sub.client.writer, version); - await flush(); - sub.track.finish(); - sub.track.close(); - const codes: number[] = []; - const skips: bigint[][] = []; - for (;;) { - const resp = await decodeSubscribeResponseMaybe(sub.client.reader, version); - if (!resp) break; - if ("drop" in resp) codes.push(resp.drop.error); - if ("skip" in resp) skips.push([resp.skip.start, resp.skip.end]); - // Lite-07 keeps the stream open past SUBSCRIBE_END, after the sweep's drops. - if ("end" in resp && keepsSubscribeOpen(version)) break; - } - // The track's own publisher never produced groups 2 through 4. - if (version === Version.DRAFT_07) { - expect(codes).toEqual([]); - expect(skips).toEqual([[2n, 4n]]); - } else { - expect(codes).toEqual([0]); + for (const declared of [false, true]) { + const named = declared && version === Version.DRAFT_07 ? "in SUBSCRIBE_SKIP" : "in SUBSCRIBE_DROP"; + test(`lite ${version}: the end sweep names a ${declared ? "declared skip" : "gap never declared"} ${named}`, async () => { + const sub = await servedSubscription({ version, startGroup: 5n }); + try { + sub.serve(5n); + expect(await sub.servedSequence()).toBe(5n); + await replayUpdate({ priority: 0, startGroup: 2n }).encode(sub.client.writer, version); + await flush(); + // Only a declared skip says groups 2 through 4 were never produced; a gap alone + // might have held them, so it is not found. + if (declared) sub.track.skipGroups(2n, 6n); + sub.track.finish(); + sub.track.close(); + const codes: number[] = []; + const skips: bigint[][] = []; + for (;;) { + const resp = await decodeSubscribeResponseMaybe(sub.client.reader, version); + if (!resp) break; + if ("drop" in resp) codes.push(resp.drop.error); + if ("skip" in resp) skips.push([resp.skip.start, resp.skip.end]); + // Lite-07 keeps the stream open past SUBSCRIBE_END, after the sweep's drops. + if ("end" in resp && keepsSubscribeOpen(version)) break; + } + if (version !== Version.DRAFT_07) { + expect([codes, skips]).toEqual([[0], []]); + } else if (declared) { + expect([codes, skips]).toEqual([[], [[2n, 4n]]]); + } else { + expect([codes, skips]).toEqual([[StreamCode.NotFound], []]); + } + } finally { + await sub.close(); } - } finally { - await sub.close(); - } - }); + }); + } } for (const version of [Version.DRAFT_06, Version.DRAFT_07]) { diff --git a/js/net/src/lite/publisher.ts b/js/net/src/lite/publisher.ts index fa5128b4051..f7db261d92c 100644 --- a/js/net/src/lite/publisher.ts +++ b/js/net/src/lite/publisher.ts @@ -31,7 +31,6 @@ import { type Run, removeRun, runGaps, - skipHole, } from "../internal.ts"; import type { Consumer as OriginConsumer } from "../origin.ts"; import * as Path from "../path.ts"; @@ -1589,11 +1588,11 @@ export class Publisher { // Name every sequence from the start to the end this subscription never // got. A cap below the end bounds what the subscriber is owed, as on its side. if (startSequence !== undefined) { - // The track's own publisher knows a group it never reported is one it never - // produced; a relay cannot tell what its source held. - const unserved: Hole = hooks.isReceived(track) - ? { error: new StreamError(StreamCode.NotFound, { message: "group not served" }) } - : skipHole(); + // A sequence the publisher declared skipped went out as a skip already; one + // it never declared might have existed, so it is not found. + const unserved: Hole = { + error: new StreamError(StreamCode.NotFound, { message: "group not served" }), + }; // Only the groups inside the ranges are owed. for (const run of hooks.servingRuns(track, startSequence, boundary())) { if (!(await drop(run.start, run.end, unserved))) return false; diff --git a/js/net/src/lite/subscribe.ts b/js/net/src/lite/subscribe.ts index 9a4ede30562..ffefe1b4e2b 100644 --- a/js/net/src/lite/subscribe.ts +++ b/js/net/src/lite/subscribe.ts @@ -874,7 +874,7 @@ export class SubscribeEnd { * no copy left, or stale. * * Draft-03+; publishers send it on Draft-05 and later. Before lite-07 it also names sequences - * never produced, which lite-07 names in {@link SubscribeSkip}. + * declared skipped, which lite-07 names in {@link SubscribeSkip}. */ export class SubscribeDrop { start: bigint; @@ -945,8 +945,8 @@ export class SubscribeDrop { } /** - * Names sequences the publisher never produced, such as skipped sequence numbers or ones past a - * cut: final, with no error, ending no reader. Lite-07+; earlier versions name them in a + * Names sequences the publisher declared it will never produce (see `Producer.skipGroups`): + * final, with no error, ending no reader. Lite-07+; earlier versions name them in a * {@link SubscribeDrop}. */ export class SubscribeSkip { diff --git a/js/net/src/lite/version.ts b/js/net/src/lite/version.ts index 41839bf47d5..21862c9a6d5 100644 --- a/js/net/src/lite/version.ts +++ b/js/net/src/lite/version.ts @@ -591,10 +591,10 @@ export function hasDropCode(version: Version): boolean { } /** - * Whether holes split into SUBSCRIBE_SKIP, for sequences never produced, and SUBSCRIBE_DROP, - * for groups that existed, whose drop names the `Max Age` and range `Terms` a stale verdict was - * judged under; SUBSCRIBE and SUBSCRIBE_UPDATE carry those `Terms`. Added in lite-07; earlier - * versions name every hole in SUBSCRIBE_DROP. + * Whether holes split into SUBSCRIBE_SKIP, for sequences declared never produced, and + * SUBSCRIBE_DROP, which names the `Update` in effect when it was given up; SUBSCRIBE and + * SUBSCRIBE_UPDATE carry that `Update` number. Added in lite-07; earlier versions name every + * hole in SUBSCRIBE_DROP. */ export function hasSkip(version: Version): boolean { // Explicitly list older versions so future versions keep the lite-07+ behavior. diff --git a/js/net/src/track.test.ts b/js/net/src/track.test.ts index c3ce14dae49..57944038963 100644 --- a/js/net/src/track.test.ts +++ b/js/net/src/track.test.ts @@ -1,7 +1,7 @@ import { expect, spyOn, test } from "bun:test"; import { Manual, Milli, Nano, sleep, Wall, yieldNow } from "@moq/clock"; import { race } from "@moq/signals"; -import { ProtocolViolation, StreamCode, StreamError, TimestampMismatch, TooFarBehind } from "./error.ts"; +import { ProtocolViolation, Skipped, StreamCode, StreamError, TimestampMismatch, TooFarBehind } from "./error.ts"; import { Decoder } from "./flate/decoder.ts"; import { Encoder } from "./flate/encoder.ts"; import type { Consumer as GroupConsumer, Producer as GroupProducer } from "./group.ts"; @@ -4853,24 +4853,37 @@ test("an errored end keeps its history", async () => { expect(() => polledOrdered.tryNextGroup()).toThrow(failed); }); -// A local publisher creates its groups in order, so the gap a created group leaves is skipped at -// once rather than waited for or judged stale; a reason already given keeps it. A received copy's -// groups arrive in any order, so its gap is not skipped. Mirrors Rust. -test("a local producer skips the gap a created group leaves at once", () => { +// Creating a group says nothing about the gap it leaves: groups may be created out of order. An +// append says the producer will not go back, so it skips every hole below it that was never +// produced, keeping a reason already given. A received copy's groups arrive in any order, so its +// gap is not skipped. Mirrors Rust. +test("an append skips the holes created groups left below it", () => { const producer = new TrackProducer("test"); producer.accept({}); const reader = producer.subscribe({ delay: { max: null } }); + const drain = () => { + const got: unknown[] = []; + for ( + let next = hooks.tryRecvGroup(reader, true); + next.kind !== "idle"; + next = hooks.tryRecvGroup(reader, true) + ) { + got.push(next.kind === "group" ? next.group.sequence : [next.start, next.end, next.skip === true]); + } + return got; + }; producer.createGroup(0n).finish(); - expect(hooks.tryRecvGroup(reader, true)).toMatchObject({ kind: "group", group: { sequence: 0n } }); hooks.dropTrackGroups(producer, 2n, 3n, { error: DROPPED }); producer.createGroup(4n).finish(); - expect(hooks.tryRecvGroup(reader, true)).toMatchObject({ kind: "drop", start: 1n, end: 2n, skip: true }); - expect(hooks.tryRecvGroup(reader, true)).toEqual({ kind: "drop", start: 2n, end: 3n, error: DROPPED }); - expect(hooks.tryRecvGroup(reader, true)).toMatchObject({ kind: "drop", start: 3n, end: 4n, skip: true }); - expect(hooks.tryRecvGroup(reader, true)).toMatchObject({ kind: "group", group: { sequence: 4n } }); - // A group appended next leaves no gap. + expect(drain()).toEqual([[2n, 3n, false], 0n, 4n]); + producer.appendGroup().finish(); - expect(hooks.tryRecvGroup(reader, true)).toMatchObject({ kind: "group", group: { sequence: 5n } }); + expect(drain()).toEqual([[1n, 2n, true], [3n, 4n, true], 5n]); + + // The next append only names what was left since this one. + producer.createGroup(7n).finish(); + producer.appendGroup().finish(); + expect(drain()).toEqual([[6n, 7n, true], 7n, 8n]); const copy = new TrackProducer("test"); hooks.receiveTrack(copy); @@ -4878,14 +4891,15 @@ test("a local producer skips the gap a created group leaves at once", () => { const late = copy.subscribe({ delay: { max: null } }); copy.createGroup(0n).finish(); copy.createGroup(2n).finish(); + copy.appendGroup().finish(); + expect(hooks.tryRecvGroup(late, true)).toMatchObject({ kind: "group" }); expect(hooks.tryRecvGroup(late, true)).toMatchObject({ kind: "group" }); expect(hooks.tryRecvGroup(late, true)).toMatchObject({ kind: "group" }); expect(hooks.tryRecvGroup(late, true)).toEqual({ kind: "idle" }); }); -// A late group the local publisher still creates in a skipped gap reaches a reader that has not -// moved past it, as a group stream wins over a drop. -test("a late local group wins over its skip for a reader that has not reported it", () => { +// A late group created out of order reaches a reader, since creating a group skips nothing. +test("a late local group reaches a reader, since creating out of order skips nothing", () => { const producer = new TrackProducer("test"); producer.accept({}); producer.createGroup(0n).finish(); @@ -4897,6 +4911,42 @@ test("a late local group wins over its skip for a reader that has not reported i expect(got).toEqual([0n, 1n, 2n]); }); +// A skip is final: creating at a sequence an append or skipGroups skipped throws, while a dropped +// sequence may be created again. skipGroups leaves cached and given-up sequences alone. Mirrors Rust. +test("creating a skipped sequence throws Skipped", () => { + const producer = new TrackProducer("test"); + producer.accept({}); + producer.createGroup(0n).finish(); + producer.createGroup(3n).finish(); + producer.appendGroup().finish(); + expect(() => producer.createGroup(1n)).toThrow(Skipped); + + producer.skipGroups(5n, 7n); + expect(() => producer.createGroup(6n)).toThrow(Skipped); + // The next sequence, 5, was skipped too. + expect(() => producer.appendGroup()).toThrow(Skipped); + // A dropped sequence may be created again. + hooks.dropTrackGroups(producer, 2n, 3n, { error: DROPPED }); + producer.createGroup(2n).finish(); + + const other = new TrackProducer("test"); + other.accept({}); + const reader = other.subscribe({ delay: { max: null } }); + other.createGroup(1n).finish(); + hooks.dropTrackGroups(other, 3n, 4n, { error: DROPPED }); + other.skipGroups(0n, 5n); + const drops: unknown[] = []; + for (let next = hooks.tryRecvGroup(reader, true); next.kind !== "idle"; next = hooks.tryRecvGroup(reader, true)) { + if (next.kind === "drop") drops.push([next.start, next.end, next.skip === true]); + } + expect(drops).toEqual([ + [0n, 1n, true], + [2n, 3n, true], + [3n, 4n, false], + [4n, 5n, true], + ]); +}); + // A received stale run is judged from where it meets the reader's span, as Rust clips it: a reader // asking only its suffix, up to where the upstream's range ended, is settled by that suffix. test("a received stale run settles the suffix a narrower range asks for", () => { diff --git a/js/net/src/track.ts b/js/net/src/track.ts index 8a1cafeae1b..e6c9d69f214 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -11,6 +11,7 @@ import { Delivered } from "./delivered.ts"; import { GroupTooLarge, ProtocolViolation, + Skipped, StreamCode, StreamError, TimestampMismatch, @@ -766,8 +767,6 @@ let startsOf: (producer: Producer) => readonly Start[]; // The exclusive end of the run of sequences from `sequence` the producer's feed can no longer // deliver, undefined while it may; assigned in its static block. let goneAt: (producer: Producer, sequence: bigint) => bigint | undefined; -// Whether the producer is fed by a wire subscription; assigned in its static block. -let received: (producer: Producer) => boolean; /** A reader's last contiguous run of groups: its start, and its exclusive end unless it reads past the live edge. */ type Coverage = { start: bigint; end?: bigint } | undefined; @@ -941,6 +940,13 @@ export class Producer { // later arrival opened it, as the session's tail does, so this holds only the gaps still // worth waiting for. #arrivals = new Ledger(); + // What this track's own publisher produced since its last append, which skips every + // sequence below it that never arrived and was not given up: the producer moved on. The + // arrivals cannot say so, since a gap folds into them once nobody waits for it. Only + // sequences at or above `#appended` are kept. Mirrors the Rust `produced`. + #produced: Run[] = []; + // One past the last append: every sequence below it arrived, was given up, or was skipped. + #appended?: bigint; constructor(name: string) { this.name = name; @@ -978,7 +984,6 @@ export class Producer { state.drops.set(drops.filter((run) => run.stale === undefined)); } }; - received = (producer) => producer.#subscribed; hooks.dropAnchors = (producer) => { for (const state of [producer.#state, ...producer.#sinks]) state.dropsAnchors.set(true); }; @@ -1549,6 +1554,7 @@ export class Producer { this.#cached.set(group.sequence, entry); this.#advance(group.sequence, true); this.#arrive(group.sequence); + this.#produce(group.sequence); this.#state.latest = max(this.#state.latest ?? 0n, group.sequence); for (const sink of this.#sinks) this.#mirror(entry, sink); // Follow the group's frames as they land, past the head `#advance` already named. @@ -1626,6 +1632,12 @@ export class Producer { /** * Append a new group with the next sequence number: one past this producer's own writes. * + * Appending says the producer will not go back, so it skips every sequence below the new + * group that it never produced and did not give up, from the lowest it produced since its + * last append, as {@link skipGroups} would. Those sequences are then final: creating one + * throws {@link Skipped}. Appending alone never leaves such a gap; one left by out-of-order + * {@link createGroup} calls is what it names. + * * The broadcast remembers nothing of an earlier producer of the name, so a replacement * starts at 0. Since a name always means the same content, a replacement continues past * its predecessor's groups with {@link createGroup}. @@ -1635,20 +1647,75 @@ export class Producer { * So does a first frame that starts before the group below it or after the one above it. */ appendGroup(): GroupProducer { - const group = new GroupProducer(this.#nextSequence()); + const sequence = this.#nextSequence(); + this.#refuseSkipped(sequence); + const group = new GroupProducer(sequence); this.#bind(group); this.#cap(group); this.#publish(group); + this.#settleAppend(sequence); return group; } + /** + * Declare that this producer will never produce `[start, end)`: none will arrive, so a reader + * waiting on one moves past it, the declared end does not wait for them, and a relay serving + * this track forwards the skip (lite-07 SUBSCRIBE_SKIP). Unlike a drop, it ends no reader. + * + * This and {@link appendGroup} are the only sources of a skip: a gap between created groups + * says nothing, since groups may be created out of order, so only the producer knows a + * sequence will never exist. A skip is final: creating a group at a skipped sequence throws + * {@link Skipped}. A sequence already produced or given up keeps what it is. + * Mirrors the Rust `skip_groups`. + */ + skipGroups(start: bigint, end: bigint): void { + if (this.#state.closed.peek() !== undefined) return; + const cached: Run[] = []; + for (const sequence of this.#cached.keys()) { + if (sequence >= start && sequence < end) insertRun(cached, sequence, sequence + 1n); + } + this.#skipGaps(cached, start, end); + } + + // Skip what of `[start, end)` neither `kept` nor a drop already names. + #skipGaps(kept: readonly Run[], start: bigint, end: bigint): void { + for (const gap of runGaps(kept, start, end)) { + for (const open of runGaps(this.#state.drops.peek(), gap.start, gap.end)) { + this.#drop(open.start, open.end, skipHole()); + } + } + } + + // Record a sequence this track's own publisher produced, for the next append to skip around. + #produce(sequence: bigint): void { + if (this.#subscribed || (this.#appended !== undefined && sequence < this.#appended)) return; + insertRun(this.#produced, sequence, sequence + 1n); + } + + // Settle an append at `sequence`: skip every sequence below it this track's own publisher + // never produced and did not give up, from the lowest it produced since its last append, + // since appending says it will not go back. A reason already given stands. + #settleAppend(sequence: bigint): void { + const from = this.#appended ?? this.#produced[0]?.start; + if (from !== undefined) this.#skipGaps(this.#produced, from, sequence); + this.#produced = []; + this.#appended = sequence + 1n; + } + + // Refuse a sequence the publisher declared skipped: a skip is final. + #refuseSkipped(sequence: bigint): void { + const run = this.#state.drops.peek().find((run) => run.start <= sequence && sequence < run.end); + if (run?.skip) throw new Skipped(sequence); + } + /** * Create a new group with the given sequence number. * - * A gap it leaves above the last group is skipped at once: readers move past it without - * waiting, and no reader ends. A group created in that gap later still reaches readers that - * have not moved past it. + * It says nothing about the sequences around it: groups may be created out of order, and an + * aborted or evicted sequence created again. A gap it leaves is skipped only once the + * producer says it will not fill it, with {@link skipGroups} or {@link appendGroup}. A + * sequence already skipped throws {@link Skipped}. * * Throws on a sequence that is still cached: a live duplicate would fan out to every * subscriber twice. An aborted incarnation is evicted so a fresh group can serve the @@ -1662,14 +1729,9 @@ export class Producer { #createGroup(sequence: bigint, start: number): GroupProducer { this.#writable(sequence); - // What this track's own publisher passes over when it creates a group is skipped at once, - // never judged stale; a received copy's groups arrive in any order. A late group it still - // creates below wins over its skip, as a group stream wins over a drop. Mirrors Rust. - // A run already given up keeps its reason. - if (!this.#subscribed && this.#received > 0n) { - for (const gap of runGaps(this.#state.drops.peek(), this.#received, sequence)) - this.#drop(gap.start, gap.end, skipHole()); - } + // A skip says the sequence will never be produced, so it is final. A drop is not: a + // publisher may re-create a group it cut or aborted. + this.#refuseSkipped(sequence); const existing = this.#cached.get(sequence); if (existing) { if (!(existing.group.closed.peek() instanceof Error)) { @@ -1704,6 +1766,7 @@ export class Producer { this.#advanceObject({ group: sequence, frame: 0 }); this.#advance(sequence, false); this.#arrive(sequence); + this.#produce(sequence); this.#finishLargest(sequence); for (const sink of this.#sinks) { sink.received.set(this.#received); @@ -1732,6 +1795,7 @@ export class Producer { */ appendDatagram(timestamp: Timestamp | undefined, payload: Uint8Array): bigint { const sequence = this.#nextSequence(); + this.#refuseSkipped(sequence); if (payload.byteLength > MAX_DATAGRAM_BYTES) throw new Error("datagram payload too large"); this.#checkDatagram(timestamp); @@ -2677,11 +2741,6 @@ export class Subscriber { }; hooks.reofferStale = (subscriber) => subscriber.#reofferStale(); hooks.reofferGroups = (subscriber, start, end) => subscriber.#reofferGroups(start, end); - hooks.isReceived = (subscriber) => { - if (subscriber.#resume) return true; - const producer = subscriber.#state.producer; - return producer !== undefined && received(producer); - }; hooks.tryRecvDatagram = (subscriber) => subscriber.#tryDatagram(); hooks.deliveredRuns = (subscriber) => subscriber.#delivered.runs; hooks.handedOut = (subscriber, sequence) => subscriber.#delivered.has(sequence); diff --git a/quest/m1/drop-set.md b/quest/m1/drop-set.md index 8021e4d1b08..63d73686970 100644 --- a/quest/m1/drop-set.md +++ b/quest/m1/drop-set.md @@ -51,8 +51,9 @@ group 2 under the open range and group 0 under a range capped before group conservative refill with bounded remembered ranges; treating every range update as invalidating in-flight drops. -Decided 2026-11-03 (maintainer, /quest-unblock), with `Max Age` dropped and -`Terms` renamed `Update` by the `/quest-complete` decisions below: lite-07 +Decided 2026-11-03 (maintainer, /quest-unblock), with `Max Age` dropped, +`Terms` renamed `Update`, and SKIP narrowed to declared sequences by the +`/quest-complete` decisions below: lite-07 splits its hole message by whether the group ever existed. `SUBSCRIBE_SKIP {Group Start, Group End}` names sequences the publisher never produced, such as skipped sequence numbers or ones past a cut: it has no error code, ends no reader, @@ -82,18 +83,50 @@ Decided 2026-11-03 (maintainer, /quest-complete), changing the design re-offers what the new terms cover and drops again, under the new Update, what is still stale. This replaces reopening only when the max delay widens past the drop's. A non-OLD drop is final, whatever its Update. -- **SKIP comes from the publisher's own group creation, eagerly:** when a - local (non-received) track producer creates a group above the next - sequence, the gap is recorded as `Hole::Skip` at once, not judged stale - until the track ends, and `append_group` never leaves a gap. A later - `create_group` below the high-water mark still works and wins over the skip, - as a Group Stream wins over a drop: in-tree callers re-create aborted - sequences (`claim_sequence`) and deliver late lower groups, and 85 moq-net - tests rely on it, so refusing it was not chosen. - **`track::Producer::drop_groups` is crate-private.** `skip_groups` stays - public for moq-archive, whose replay names the sequences it holds nothing - for up to a recorded end above its last group; a SKIP, matching the gaps - between stored groups that eager skipping already names. + public (see the next decision). + +Decided 2026-11-03 (maintainer, /quest-complete), replacing the "eager SKIP" +decision above and the PR's `## Waiting` item on out-of-order creation: + +- ✅ **SKIP comes only from an explicit, public `track::Producer::skip_groups`**, + and from `append_group` (below). Reason: producers legitimately create + groups out of order (`claim_sequence` re-creating an aborted sequence, + moq-mux and moq-c `seek`, `finish()`'s documented "older groups can still + arrive"), so a gap tells moq-net nothing; only the producer knows a + sequence will never exist. moq-archive's replay is the in-tree consumer of + `skip_groups`, for the sequences up to a recorded end it holds nothing + for; a producer that numbers sparsely would call it too. moq-mux's live + encoders use `append_group`. The relay forwards received SKIPs through the + crate-internal `give_up`. +- ✅ **Both inferences are removed:** the eager skip of the gap in + `create_group`, and the automatic SKIP at finish for never-arrived + sequences below the final one (`TrackState::missed` and the publisher's + `drop_unserved`). An undeclared gap is judged stale against the reader's + budget, or named with DROP NOT_FOUND, as before this PR. +- ✅ **SKIP is final:** creating a group (or appending a group or datagram) + at a skipped sequence is refused with `Error::Skipped` (JS `Skipped`). + Re-creating a dropped or aborted sequence still works. +- ✅ **`append_group` is an implicit skip** (refinement, same session): it + means "the next group", so the producer declares it won't go back, and it + skips every sequence below it that never arrived and wasn't already given + up, as if `skip_groups` were called on them; those are then final. Today + `append_group` leaves no gap of its own, so this only fires on holes earlier + `create_group` calls left below the head; under time addressing it names + the span between consecutive starts. `create_group` stays non-committal, + so out-of-order creation (`claim_sequence` retries, `seek`) keeps working. + There is still no inference at finish. +- ✅ **`skip_groups` stays public** and is documented as the explicit source + of SKIP; `drop_groups` stays crate-private. Rust and JS mirror each other + (`skipGroups`, `appendGroup`). +- ✅ **Draft:** SKIP names only sequences the publisher declared it will never + produce, and it is final. +- Not chosen: eager skip on `create_group` with a late group winning over + it (the shipped design, which inferred SKIP from a gap that out-of-order + creation makes meaningless); refusing every `create_group` below the last + group (broke `claim_sequence`, `seek` and 85 moq-net tests); keeping the + finish-time inference for a local producer (a gap at finish is no more a + declaration than one mid-stream). Decided 2026-10-27 (maintainer): this runs after the range count cost work, which finished in [moq-dev/moq#7542](https://github.com/moq-dev/moq/pull/7542), diff --git a/quest/m1/time-addressing/model.md b/quest/m1/time-addressing/model.md index a6f307c98c9..ed6f78e5885 100644 --- a/quest/m1/time-addressing/model.md +++ b/quest/m1/time-addressing/model.md @@ -63,13 +63,18 @@ a producer would need per-track memory of its last number. one FETCH per tick in the hole. Accepted: "This was a lingering problem. The entire point of (multiple) ranges is to avoid this issue." -- **Eager skips** (decided 2026-11-03, maintainer, /quest-complete): a local - producer's `create_group` skips the gap it leaves above its last group at - once, and `append_group` never leaves one - ([moq-dev/moq#7670](https://github.com/moq-dev/moq/pull/7670)). - `append_group(timestamp)` follows the same rule: every start between the - previous group's and the new one is skipped when the group is created, - never judged stale later. +- **Skips are declared, never inferred** (decided 2026-11-03, maintainer, + /quest-complete, replacing that day's "eager skips"; + [moq-dev/moq#7670](https://github.com/moq-dev/moq/pull/7670)). A gap tells + moq-net nothing, since groups may be created out of order. SKIP comes only + from the producer: an explicit `skip_groups`, or `append_group`, which + means "the next group" and so skips every key below it that never arrived + and was not given up, since the producer's last append. `create_group` + implies no skip. A skip is final: creating a group at a skipped key is + refused. Today `append_group` leaves no gap of its own, so its skip only + names holes earlier `create_group` calls left. `append_group(timestamp)` + keeps the rule, and there it names the span between consecutive starts: + every key between the previous group's start and the new one. Every current role of the sequence needs a replacement or a reason to drop it: diff --git a/rs/moq-archive/src/consumer/track.rs b/rs/moq-archive/src/consumer/track.rs index 4eb0ea60548..d3401565dbe 100644 --- a/rs/moq-archive/src/consumer/track.rs +++ b/rs/moq-archive/src/consumer/track.rs @@ -141,9 +141,8 @@ impl Live { }, None, )?; - // An empty group stores no object, so the replay never produces what the archive - // holds nothing for up to the end, as it skips the gaps between stored groups, and - // the end settles over it. + // An empty group stores no object, so the replay declares it never produces what + // the archive holds nothing for up to the end, and the end settles over it. let fin = at.group + u64::from(at.frame != 0); let from = self.next.unwrap_or(0); self.producer.skip_groups(from..fin); diff --git a/rs/moq-net/src/error.rs b/rs/moq-net/src/error.rs index 0f3e1cd2117..772c899e871 100644 --- a/rs/moq-net/src/error.rs +++ b/rs/moq-net/src/error.rs @@ -391,6 +391,11 @@ pub enum Error { #[error("duplicate")] Duplicate, + /// A group was created at a sequence its track declared skipped: a skip says the + /// sequence will never be produced, so it is final. See `track::Producer::skip_groups`. + #[error("sequence was skipped")] + Skipped, + /// Nobody is reading any more, so the producer stopped; or the session closed cleanly, /// by either side, as `Session::closed` reports it. Not a failure. #[error("cancelled")] @@ -657,6 +662,7 @@ impl From<&Error> for SessionError { | Error::UnexpectedMessage | Error::UnexpectedStream | Error::Duplicate + | Error::Skipped | Error::Decode(_) | Error::Encode(_) | Error::WrongSize @@ -744,6 +750,7 @@ impl From<&Error> for StreamError { | Error::UnknownAlpn(_) | Error::TooManyRequests | Error::Duplicate + | Error::Skipped | Error::GoawayTimeout | Error::ProtocolViolation | Error::UnexpectedMessage => Self::Session(SessionError::from(err)), diff --git a/rs/moq-net/src/lite/publisher.rs b/rs/moq-net/src/lite/publisher.rs index edaa8265657..8616acd314e 100644 --- a/rs/moq-net/src/lite/publisher.rs +++ b/rs/moq-net/src/lite/publisher.rs @@ -4747,8 +4747,8 @@ impl TrackRun { } /// Name every sequence from the start to the end that this subscription never got. The - /// track has ended, so none of them will be served now: groups it never produced (a - /// skipped sequence), and groups the cursor passed over (stale, or missing their head). + /// track has ended, so none of them will be served now: groups the cursor passed over + /// (stale, or missing their head), and gaps the publisher never declared skipped. /// /// Returns whether any drop was buffered. fn drop_unserved( @@ -4770,12 +4770,9 @@ impl TrackRun { // Only the groups the ranges name: one between disjoint ranges is not owed, and // naming it would keep a later update widening over it from delivering it. let end = self.track.subscription().end().map_or(fin, |end| end.min(fin)); - // The track's own publisher knows a group it never reported is one it never produced; - // a relay cannot tell what its source held. - let hole = match self.track.is_received() { - false => Hole::Skip, - true => Hole::drop(StreamError::NotFound), - }; + // A sequence the publisher declared skipped went out as a skip already; one it never + // declared might have existed, so it is not found. + let hole = Hole::drop(StreamError::NotFound); let mut dropped = false; for run in self.track.runs_in_range(start, end) { dropped |= self.drop_range(run, &hole, writer)?; @@ -6822,10 +6819,37 @@ mod serve_group_test { } } - /// A sequence the publisher never produced is accounted for by SKIP rather than a stream - /// count. + /// A gap the publisher never declared might have existed, so the end names it with a + /// DROP rather than a SKIP: here the reader passed it, so it is stale. + #[kio::test] + async fn lite07_end_drops_an_undeclared_gap() { + let mut track = track::Producer::new( + Arc::new(broadcast::Info::default()), + "test", + crate::track::Info::timed(), + ); + let subscriber = track.subscribe(None); + let (mut run, mut writer, log) = lite07_run(SinkSession::new(Log::default()), subscriber); + let mut run = std::pin::pin!(kio::wait(move |waiter| run.poll(&mut writer, waiter))); + + write_group(&mut track, 0, 0); + assert!(futures::poll!(run.as_mut()).is_pending()); + write_group(&mut track, 2, 2); + track.finish().unwrap(); + assert!(futures::poll!(run.as_mut()).is_pending(), "the stream stays open"); + + // SUBSCRIBE_START with the range starting at 0, a DROP for group 1 with OLD (0x34) + // under update 0, which the end gives up on at once, and SUBSCRIBE_FINISH at 3. + assert_eq!( + *log.writes.lock().unwrap(), + [0, 4, 1, 0, 1, 1, 2, 4, 1, 1, 0x34, 0, 1, 2, 3, 0] + ); + } + + /// A sequence the publisher declared it will never produce is accounted for by SKIP + /// rather than a stream count. #[kio::test] - async fn lite07_end_skips_sequences_never_produced() { + async fn lite07_skips_a_declared_skip() { let mut track = track::Producer::new( Arc::new(broadcast::Info::default()), "test", @@ -6838,11 +6862,12 @@ mod serve_group_test { write_group(&mut track, 0, 0); assert!(futures::poll!(run.as_mut()).is_pending()); write_group(&mut track, 2, 2); + track.skip_groups(1..2); track.finish().unwrap(); assert!(futures::poll!(run.as_mut()).is_pending(), "the stream stays open"); - // SUBSCRIBE_START with the range starting at 0, a SKIP for group 1, which the end - // gives up on at once, and SUBSCRIBE_FINISH at 3. + // SUBSCRIBE_START with the range starting at 0, a SKIP for group 1, and + // SUBSCRIBE_FINISH at 3. assert_eq!(*log.writes.lock().unwrap(), [0, 4, 1, 0, 1, 1, 3, 2, 1, 1, 1, 2, 3, 0]); } @@ -7628,10 +7653,8 @@ mod serve_group_test { run.await.unwrap(); } // Group 2 arrived before the subscription, but it is cached, so the lowered start - // serves it and the drops name only what is missing around it. The publisher skipped - // 3 and 4 when it created group 5, so they go out once the lowered start reaches them, - // before the end names the rest. - assert_eq!(drops(&mut writer, &log, version).await, [(3, 4), (0, 1), (6, 6)]); + // serves it and the drops name only what is missing around it. + assert_eq!(drops(&mut writer, &log, version).await, [(0, 1), (3, 4), (6, 6)]); } /// A lowered start restarts the age of the gaps it newly asks for, as the subscriber does, diff --git a/rs/moq-net/src/lite/subscribe.rs b/rs/moq-net/src/lite/subscribe.rs index e7ad30bc354..c604d389486 100644 --- a/rs/moq-net/src/lite/subscribe.rs +++ b/rs/moq-net/src/lite/subscribe.rs @@ -742,7 +742,7 @@ impl<'a> Message<'a> for SubscribeUpdate { /// `start = 5, end = 7` means groups 5, 6, and 7 were dropped. /// /// Lite03+; publishers send it on Lite05 and later. Before lite-07 it also names sequences -/// never produced, which lite-07 names in [`SubscribeSkip`]. +/// declared skipped, which lite-07 names in [`SubscribeSkip`]. #[derive(Clone, Debug, PartialEq, Eq)] pub struct SubscribeDrop { /// The first absolute group sequence in the dropped range. @@ -836,8 +836,8 @@ impl<'a> Message<'a> for SubscribeDrop { } } -/// Names sequences the publisher never produced, such as skipped sequence numbers or ones -/// past a cut: final, with no error, ending no reader. +/// Names sequences the publisher declared it will never produce (see +/// `track::Producer::skip_groups`): final, with no error, ending no reader. /// /// The range `[start, end]` is inclusive on both ends. Lite07+ only; earlier versions name /// them in a [`SubscribeDrop`]. diff --git a/rs/moq-net/src/lite/version.rs b/rs/moq-net/src/lite/version.rs index 03b1f97b0fa..78b329071eb 100644 --- a/rs/moq-net/src/lite/version.rs +++ b/rs/moq-net/src/lite/version.rs @@ -297,7 +297,7 @@ impl Version { ) } - /// Whether holes split into SUBSCRIBE_SKIP, for sequences never produced, and + /// Whether holes split into SUBSCRIBE_SKIP, for sequences declared never produced, and /// SUBSCRIBE_DROP, for groups that existed, which names the `Update` in effect when it was /// given up; SUBSCRIBE and SUBSCRIBE_UPDATE carry that `Update` number. Added in lite-07; /// earlier versions name every hole in SUBSCRIBE_DROP. diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index c579b67b5de..479ce81b1be 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -518,6 +518,14 @@ pub(crate) struct TrackState { // cannot erase a disposition a relay must forward, and as ranges so a wide one costs // one entry. drops: RangeMap, + // What this track's own publisher produced since its last `append_group`, which skips + // every sequence below it that never arrived and was not given up: the producer moved on. + // Arrivals cannot say so, since a gap folds into them once nobody waits for it. Only + // sequences at or above `appended` are kept, so it holds the runs created since then. + produced: RangeSet, + // One past the last `append_group`: every sequence below it arrived, was given up, or + // was skipped. + appended: Option, // Where the ranges a received copy's upstream judges stale runs against end: the // exclusive end of its current terms, `None` past the live edge or unknown. judged_end: Option, @@ -1153,21 +1161,49 @@ impl TrackState { self.change_drops(groups, Some(hole)); } - /// Why a run a reader's misses gave up will not come: never produced when this track's - /// own publisher ended without it ever arriving, whatever else judged it; otherwise stale - /// under `stale` when that is the verdict, or unknown, so simply not found. - fn missed(&self, run: &ops::Range, stale: Option) -> Hole { - let never = !self.received - && self.final_sequence().is_some_and(|fin| run.end <= fin) - && self.arrivals.accounted(run.clone()).next().is_none() - && self.lookup.range(run.clone()).next().is_none(); - match (never, stale) { - (true, _) => Hole::Skip, - (false, Some(budget)) => Hole::stale(budget), - (false, None) => Hole::drop(StreamError::NotFound), + /// Whether the publisher declared `sequence` skipped: never produced, which is final. + fn skipped(&self, sequence: u64) -> bool { + self.drops.get(&sequence).is_some_and(|hole| *hole == Hole::Skip) + } + + /// Skip what of `groups` is neither cached nor given up; see [`Producer::skip_groups`]. + fn skip_groups(&mut self, groups: ops::Range) { + let mut left = RangeSet::new(); + for gap in self.drops.gaps(&groups) { + left.insert(gap); + } + for (sequence, slot) in self.lookup.range(groups) { + if slot.visible + && let Some(end) = sequence.checked_add(1) + { + left.remove(*sequence..end); + } + } + for run in left.iter().cloned().collect::>() { + self.drop_groups(run, Hole::Skip); } } + /// Settle an append at `sequence`: skip every sequence below it that this track's own + /// publisher never produced and did not give up, from the lowest it produced since its + /// last append, since appending says it will not go back. A reason already given stands. + fn appended(&mut self, sequence: u64) { + if let Some(from) = self.appended.or(self.produced.iter().next().map(|run| run.start)) { + let mut left = RangeSet::new(); + for gap in self.drops.gaps(&(from..sequence)) { + left.insert(gap); + } + for run in self.produced.iter() { + left.remove(run.clone()); + } + for run in left.iter().cloned().collect::>() { + self.drop_groups(run, Hole::Skip); + } + } + self.produced.clear(); + self.appended = sequence.checked_add(1); + } + /// Forget every run held as stale; see [`Producer::supersede_stale`]. fn supersede_stale(&mut self) { let stale: Vec<_> = self @@ -2192,6 +2228,12 @@ impl TrackState { /// Record that the live feed delivered `sequence`, then fold the gaps that aged out. fn record_arrival(&mut self, sequence: u64, now: crate::time::Instant) { self.arrivals.arrive(sequence, now); + if !self.received + && self.appended.is_none_or(|appended| sequence >= appended) + && let Some(end) = sequence.checked_add(1) + { + self.produced.insert(sequence..end); + } // A feed still delivering keeps the gap above its last arrival open. if self.tail.is_some() { self.tail = Some(now); @@ -3425,9 +3467,10 @@ impl Producer { /// Create a new group with the given sequence number. /// - /// A gap it leaves above the last group is skipped at once: readers move past it without - /// waiting, and no reader ends. A group created in that gap later still reaches readers - /// that have not moved past it. + /// It says nothing about the sequences around it: groups may be created out of order, and + /// an aborted or evicted sequence created again. A gap it leaves is skipped only once the + /// producer says it will not fill it, with [`Self::skip_groups`] or [`Self::append_group`]. + /// A sequence already skipped is refused with [`Error::Skipped`]. /// /// Readers and group demand reach it at once, but the track counts it as content (its /// live edge, or the successor that ages the group before it) only once its first frame @@ -3447,16 +3490,10 @@ impl Producer { if state.sealed || state.final_sequence().is_some_and(|fin| info.sequence >= fin) { return Err(Error::Closed); } - // What this track's own publisher passes over when it creates a group is skipped at - // once, never judged stale; a received copy's groups arrive in any order. A late group - // it still creates below wins over its skip, as a group stream wins over a drop. - // A run already given up keeps its reason. - if !state.received && received.is_none() && state.max_sequence.is_some() { - let next = state.next_sequence()?; - let gaps: Vec<_> = state.drops.gaps(&(next..info.sequence)).collect(); - for gap in gaps { - state.drop_groups(gap, Hole::Skip); - } + // A skip says the sequence will never be produced, so it is final. A drop is not: a + // publisher may re-create a group it cut or aborted. + if state.skipped(info.sequence) { + return Err(Error::Skipped); } let start = received.unwrap_or(0); let content = if received.is_some() { @@ -3486,6 +3523,12 @@ impl Producer { /// Create a new group with the next sequence number: one past this producer's own writes. /// + /// Appending says the producer will not go back, so it skips every sequence below the new + /// group that it never produced and did not give up, from the lowest it produced since + /// its last append, as [`Self::skip_groups`] would. Those sequences are then final: creating + /// one is refused. Appending alone never leaves such a gap; one left by out-of-order + /// [`Self::create_group`] calls is what it names. + /// /// The broadcast remembers nothing of an earlier producer of the name, so a replacement /// starts at 0. Since a name always means the same content, a replacement continues past /// its predecessor's groups with [`Self::create_group`]. @@ -3497,6 +3540,9 @@ impl Producer { { return Err(Error::Closed); } + if state.skipped(sequence) { + return Err(Error::Skipped); + } let track = state.info.clone().unwrap(); @@ -3504,6 +3550,7 @@ impl Producer { group::Producer::new(group::Info { sequence }, track, state.cache.clone()).with_meter(self.stats.meter()); state.cap(&group.handle())?; state.commit_group(&group.handle()); + state.appended(sequence); let deferred = state.deferred(); drop(state); deferred.wake(); @@ -3545,6 +3592,9 @@ impl Producer { { return Err(Error::Closed); } + if state.skipped(sequence) { + return Err(Error::Skipped); + } state.advance(sequence, false); state.cache.finish_largest(sequence); // A datagram is its own group, so it compresses alone. @@ -3813,15 +3863,23 @@ impl Producer { self.give_up(groups, Hole::drop(err)); } - /// Record that the publisher never produced `groups`: none will arrive, so a reader + /// Declare that this producer will never produce `groups`: none will arrive, so a reader /// waiting on one moves past it, the declared end does not wait for them, and a relay - /// serving this track forwards the skip. Unlike a drop, it ends no reader. + /// serving this track forwards the skip (lite-07 SUBSCRIBE_SKIP). Unlike a drop, it ends + /// no reader. /// - /// A gap below a created group is skipped already, since a publisher creates its groups - /// in order. This names the rest, such as sequences between the last group and an end - /// declared ahead of it that will never be produced. + /// This and [`Self::append_group`] are the only sources of a skip: a gap between created + /// groups says nothing, since groups may be created out of order, so only the producer + /// knows a sequence will never exist. A skip is final: creating a group at a skipped + /// sequence is refused with [`Error::Skipped`]. A sequence already produced or given up + /// keeps what it is. pub fn skip_groups(&self, groups: ops::Range) { - self.give_up(groups, Hole::Skip); + if let Ok(mut state) = self.modify() { + state.skip_groups(groups); + let deferred = state.deferred(); + drop(state); + deferred.wake(); + } } /// Forget every run held as stale: the terms a received copy's upstream judged them @@ -7165,9 +7223,8 @@ impl Cursor { if self.drops.is_none() { let mut drops = Box::new(DropCursor::new(&self.state.read())); if let Some(misses) = &self.misses { - let state = self.state.read(); for run in misses.given_up().iter() { - drops.give_up(run.clone(), state.missed(run, None)); + drops.give_up(run.clone(), Hole::missed(None)); } } self.drops = Some(drops); @@ -7907,9 +7964,8 @@ impl Cursor { misses.expire(&self.state.read(), drift, &mut given_up, waiter); } if let Some(drops) = &mut self.drops { - let state = self.state.read(); for (run, stale) in given_up { - drops.give_up(run.clone(), state.missed(&run, stale)); + drops.give_up(run.clone(), Hole::missed(stale)); } } // A refusal taken in may move a time end walking history past it, widening what @@ -7927,9 +7983,8 @@ impl Cursor { let mut given_up = Vec::new(); let group = misses.next_filled(&self.state.read(), order, &mut given_up); if let Some(drops) = &mut self.drops { - let state = self.state.read(); for (run, stale) in given_up { - drops.give_up(run.clone(), state.missed(&run, stale)); + drops.give_up(run.clone(), Hole::missed(stale)); } } group @@ -9489,17 +9544,6 @@ impl Subscriber { } } - /// Whether this subscriber reads a copy received from a remote publisher, which cannot - /// tell a group its source never produced from one it never received. A front's reader - /// always reads its routes' copies. - #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] - pub(crate) fn is_received(&self) -> bool { - match &self.inner { - Inner::Plain(cursor) => cursor.state.read().received, - Inner::Resume(..) => true, - } - } - #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] pub(crate) fn latest_sequence(&self) -> Option { match &self.inner { @@ -23413,35 +23457,128 @@ mod test { ); } - /// A local publisher creating a group past the next sequence skips the gap at once: a - /// reader is told without waiting for any budget or grace, and no reader ends. A late - /// group the publisher still creates in the gap wins over the skip, as a group stream wins - /// over a drop, so a reader takes it once. + /// Creating a group past the next sequence says nothing about the gap: groups may be + /// created out of order, so a late one in it still reaches readers, and nothing is skipped. #[test] #[cfg(feature = "async")] - fn a_local_gap_is_skipped_once_a_group_above_it_is_created() { + fn creating_a_group_past_a_gap_skips_nothing() { let producer = track_producer("test", Info::timed()); let mut subscriber = producer.subscribe(replay().with_range(0..)); let waiter = kio::Waiter::noop(); write_group(&producer, 2); write_group(&producer, 5); - assert_eq!(subscriber.poll_drop(&waiter), Poll::Ready((3..5, Hole::Skip))); + assert!(producer.state.read().drops.is_empty()); + assert!(subscriber.poll_drop(&waiter).is_pending()); assert_eq!(drain_groups(&mut subscriber), [2, 5]); write_group(&producer, 4); - assert!( - !producer.state.read().drops.contains_key(&4), - "the group wins over its skip" - ); assert_eq!(drain_groups(&mut subscriber), [4]); - let mut late = producer.subscribe(replay().with_range(0..)); - assert_eq!(drained_drops(&mut late), [(3, 4)]); - let mut read = drain_groups(&mut late); - read.sort(); - assert_eq!(read, [2, 4, 5]); + assert!(producer.state.read().drops.is_empty()); + } + + /// Appending says the producer will not go back: every sequence below the new group it + /// never produced since its first group is skipped, and readers are told at once. + #[test] + #[cfg(feature = "async")] + fn appending_skips_the_holes_below_it() { + let producer = track_producer("test", Info::timed()); + let mut subscriber = producer.subscribe(replay().with_range(0..)); + let waiter = kio::Waiter::noop(); + write_group(&producer, 2); + write_group(&producer, 5); + write_group(&producer, 7); + producer.append_group().unwrap().finish().unwrap(); + assert_eq!(subscriber.poll_drop(&waiter), Poll::Ready((3..5, Hole::Skip))); + assert_eq!(subscriber.poll_drop(&waiter), Poll::Ready((6..7, Hole::Skip))); + assert_eq!(drain_groups(&mut subscriber), [2, 5, 7, 8]); + + // The next append only names what was left since this one. + write_group(&producer, 10); + producer.append_group().unwrap().finish().unwrap(); + assert_eq!(subscriber.poll_drop(&waiter), Poll::Ready((9..10, Hole::Skip))); + let skipped: Vec<_> = producer + .state + .read() + .drops + .iter() + .map(|(run, hole)| (run.clone(), hole.clone())) + .collect(); + assert_eq!(skipped, [(3..5, Hole::Skip), (6..7, Hole::Skip), (9..10, Hole::Skip)]); + } + + /// A sequence already given up keeps its reason when an append skips around it. + #[test] + #[cfg(feature = "async")] + fn appending_keeps_a_reason_already_given() { + let producer = track_producer("test", Info::timed()); + write_group(&producer, 0); + write_group(&producer, 4); + producer.give_up(1..2, Hole::drop(StreamError::Cancel)); + producer.append_group().unwrap().finish().unwrap(); + let drops: Vec<_> = producer + .state + .read() + .drops + .iter() + .map(|(run, hole)| (run.clone(), hole.clone())) + .collect(); + assert_eq!(drops, [(1..2, Hole::drop(StreamError::Cancel)), (2..4, Hole::Skip)]); + } + + /// A skip is final: creating a group at a sequence an append or `skip_groups` skipped is + /// refused, while a dropped one may be created again. + #[test] + #[cfg(feature = "async")] + fn creating_a_skipped_sequence_is_refused() { + let producer = track_producer("test", Info::timed()); + write_group(&producer, 0); + write_group(&producer, 3); + producer.append_group().unwrap().finish().unwrap(); + assert!(matches!( + producer.create_group(group::Info { sequence: 1 }), + Err(Error::Skipped) + )); + + producer.skip_groups(5..7); + assert!(matches!( + producer.create_group(group::Info { sequence: 6 }), + Err(Error::Skipped) + )); + // The next sequence, 5, was skipped too. + assert!(matches!(producer.append_group(), Err(Error::Skipped))); + // A dropped sequence may be created again. + producer.give_up(2..3, Hole::drop(StreamError::Cancel)); + write_group(&producer, 2); + } + + /// `skip_groups` leaves a cached group and a reason already given alone. + #[test] + #[cfg(feature = "async")] + fn skipping_leaves_produced_and_given_up_sequences() { + let producer = track_producer("test", Info::timed()); + write_group(&producer, 1); + producer.give_up(3..4, Hole::drop(StreamError::Cancel)); + producer.skip_groups(0..5); + let drops: Vec<_> = producer + .state + .read() + .drops + .iter() + .map(|(run, hole)| (run.clone(), hole.clone())) + .collect(); + assert_eq!( + drops, + [ + (0..1, Hole::Skip), + (2..3, Hole::Skip), + (3..4, Hole::drop(StreamError::Cancel)), + (4..5, Hole::Skip) + ] + ); } - /// Appending leaves no gap, and nothing below a track's first group is skipped. + /// Appending and a first group leave no gap, so they skip nothing; nor is anything below + /// a track's first group skipped. #[test] #[cfg(feature = "async")] fn appending_and_a_first_group_skip_nothing() { diff --git a/rs/moq-net/src/model/track/hole.rs b/rs/moq-net/src/model/track/hole.rs index bd010292144..52312784100 100644 --- a/rs/moq-net/src/model/track/hole.rs +++ b/rs/moq-net/src/model/track/hole.rs @@ -8,7 +8,8 @@ use crate::StreamError; /// earlier versions name both in SUBSCRIBE_DROP. #[derive(Clone, Debug, PartialEq, Eq)] pub(crate) enum Hole { - /// Never produced, such as a skipped sequence number: final, and it ends no reader. + /// Never produced, as the publisher declared (see `track::Producer::skip_groups`): final, + /// and it ends no reader. Never inferred from a gap, since groups may be created out of order. Skip, /// Produced but not delivered, ending any reader of it with `error`. Drop { @@ -33,6 +34,15 @@ impl Hole { } } + /// A run a reader's misses gave up: stale under `stale` when that is the verdict, + /// otherwise simply not found. Never a skip, which only the publisher declares. + pub fn missed(stale: Option) -> Self { + match stale { + Some(budget) => Self::stale(budget), + None => Self::drop(StreamError::NotFound), + } + } + /// The error a reader waiting on the sequence ends with: a sequence never produced is /// simply not found. pub fn error(&self) -> StreamError { From 67c68ea3e4d0e25ba3f89226466fa2e49ebe52ea Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 18:56:54 -0800 Subject: [PATCH 19/68] fix(net): re-judge pending redrops and front-backed stale reports on update A run one lite-07 update left to drop again is judged again under the next update instead of going out under terms that may serve it, and a front's reader takes back the stale runs it queued or its serving copy reported, so a relay re-offers them too (pullfrog review). JS mirrors both. Adds tests that a huge skip_groups or append gap costs one run and one SUBSCRIBE_SKIP in Rust and JS, and that an aborted sequence is created again while a skipped one is refused. Co-Authored-By: Claude Opus 5.5 --- js/net/src/internal.ts | 2 + js/net/src/lite/publisher.test.ts | 26 ++++++++++ js/net/src/lite/publisher.ts | 11 ++++ js/net/src/resume.test.ts | 19 +++++++ js/net/src/resume.ts | 14 ++++++ js/net/src/track.test.ts | 35 ++++++++++++- js/net/src/track.ts | 4 +- quest/m1/drop-set.md | 14 ++++++ rs/moq-net/src/lite/publisher.rs | 84 ++++++++++++++++++++++++++++++- rs/moq-net/src/model/resume.rs | 44 ++++++++++++++++ rs/moq-net/src/model/track.rs | 58 +++++++++++++++++++-- 11 files changed, 304 insertions(+), 7 deletions(-) diff --git a/js/net/src/internal.ts b/js/net/src/internal.ts index 2b9b7f069bf..fe43a8101a5 100644 --- a/js/net/src/internal.ts +++ b/js/net/src/internal.ts @@ -436,6 +436,8 @@ export interface ResumeSource { cachedGroup(sequence: bigint, from?: number): GroupConsumer | undefined; /** See {@link hooks.reofferGroups}, answered by the serving copy. */ reofferGroups(start: bigint, end: bigint): { groups: GroupConsumer[]; givenUp: Run[] }; + /** See {@link hooks.reofferStale}: the stale runs queued here and those the serving copy reported. */ + reofferStale(): Run[]; /** Let go of every copy. */ close(abort?: Error): void; } diff --git a/js/net/src/lite/publisher.test.ts b/js/net/src/lite/publisher.test.ts index a4687adcf0f..2a04baf8751 100644 --- a/js/net/src/lite/publisher.test.ts +++ b/js/net/src/lite/publisher.test.ts @@ -1069,6 +1069,32 @@ for (const version of [Version.DRAFT_06, Version.DRAFT_07]) { } } +// A huge declared skip goes out as one SUBSCRIBE_SKIP, at a cost independent of its length. +// Mirrors Rust. +test("lite-07 skips a huge run at once", async () => { + const FAR = 1n << 61n; + const sub = await servedSubscription({ version: Version.DRAFT_07, startGroup: 0n }); + try { + sub.serve(0n); + expect(await sub.servedSequence()).toBe(0n); + sub.track.skipGroups(1n, FAR); + sub.track.finish(); + sub.track.close(); + const codes: number[] = []; + const skips: bigint[][] = []; + for (;;) { + const resp = await decodeSubscribeResponseMaybe(sub.client.reader, Version.DRAFT_07); + if (!resp) break; + if ("drop" in resp) codes.push(resp.drop.error); + if ("skip" in resp) skips.push([resp.skip.start, resp.skip.end]); + if ("end" in resp) break; + } + expect([codes, skips]).toEqual([[], [[1n, FAR - 1n]]]); + } finally { + await sub.close(); + } +}); + for (const version of [Version.DRAFT_06, Version.DRAFT_07]) { for (const transport of [900, 1400]) { test(`lite ${version}: datagram bodies fit the wire and ${transport}-byte transport`, async () => { diff --git a/js/net/src/lite/publisher.ts b/js/net/src/lite/publisher.ts index f7db261d92c..b77b3190108 100644 --- a/js/net/src/lite/publisher.ts +++ b/js/net/src/lite/publisher.ts @@ -1436,6 +1436,17 @@ export class Publisher { if (served) served.splice(0, served.length, ...removeRun(served, start, end)); }; for (const run of hooks.reofferStale(track)) owe(run.start, run.end); + // A stale drop not sent yet, such as one an earlier update left to drop + // again, is judged again under these terms too. + for (const pending of pendingDrops.filter( + (d) => toStreamCode(d.error) === StreamCode.Old, + )) + insertRun(expired, pending.start, pending.end); + pendingDrops.splice( + 0, + pendingDrops.length, + ...pendingDrops.filter((d) => toStreamCode(d.error) !== StreamCode.Old), + ); // A group whose stream already reset as stale: one still cached, in range, and // fresh under the new budget is served again; the rest is dropped again, by run. for (const run of expired.splice(0)) { diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts index 2dfcbd846bf..500fbeda483 100644 --- a/js/net/src/resume.test.ts +++ b/js/net/src/resume.test.ts @@ -3348,3 +3348,22 @@ test("a resumed reader re-offers a group its serving copy still caches", () => { front.consumer.close(); a.close(); }); + +// A front's reader takes back the stale runs its serving route's copy reported, so a relay's +// publisher re-offers them under a new update rather than keep them reported. Mirrors Rust. +test("a resumed reader takes back its stale reports", () => { + const front = new Front(); + const a = new BroadcastProducer(); + const ta = copy(a); + front.serve(a.consume()); + const sub = front.consumer.track("seq").subscribe({ delay: { max: Milli(60_000) } }); + ta.write(0n); + expect(drain(sub)).toEqual([0n]); + hooks.dropTrackGroups(ta.producer, 1n, 3n, staleHole(60_000)); + expect(hooks.tryDrop(sub)).toMatchObject({ kind: "drop", start: 1n, end: 3n, stale: 60_000 }); + expect(hooks.reofferStale(sub)).toEqual([{ start: 1n, end: 3n }]); + expect(hooks.reofferStale(sub)).toEqual([]); + sub.close(); + front.consumer.close(); + a.close(); +}); diff --git a/js/net/src/resume.ts b/js/net/src/resume.ts index c0f4d27d60f..059e474958f 100644 --- a/js/net/src/resume.ts +++ b/js/net/src/resume.ts @@ -607,6 +607,20 @@ class Reader implements ResumeSource { return { groups: groups.map((group) => this.#handOut(serving, group)), givenUp }; } + /** + * Take back every stale run: those queued here, and those the serving copy reported, which + * its cursor owes again. Only the serving copy's verdicts count, so a replaced copy's are left + * alone. Mirrors the Rust `Reader::reoffer_stale`. + */ + reofferStale(): Run[] { + const taken: Run[] = []; + for (const run of this.#drops) if (run.stale !== undefined) insertRun(taken, run.start, run.end); + this.#drops = this.#drops.filter((run) => run.stale === undefined); + const serving = this.#serving(); + if (serving) for (const run of hooks.reofferStale(serving.sub)) insertRun(taken, run.start, run.end); + return taken; + } + cachedGroup(sequence: bigint, from = 0): GroupConsumer | undefined { for (let index = this.#copies.length - 1; index >= 0; index--) { const copy = this.#copies[index]; diff --git a/js/net/src/track.test.ts b/js/net/src/track.test.ts index 57944038963..79420e2d91d 100644 --- a/js/net/src/track.test.ts +++ b/js/net/src/track.test.ts @@ -4868,7 +4868,9 @@ test("an append skips the holes created groups left below it", () => { next.kind !== "idle"; next = hooks.tryRecvGroup(reader, true) ) { - got.push(next.kind === "group" ? next.group.sequence : [next.start, next.end, next.skip === true]); + if (next.kind === "group") got.push(next.group.sequence); + else if (next.kind === "drop") got.push([next.start, next.end, next.skip === true]); + else throw new Error(`unexpected ${next.kind}`); } return got; }; @@ -4963,3 +4965,34 @@ test("a received stale run settles the suffix a narrower range asks for", () => reader.close(); producer.close(); }); + +// Skips are kept and reported as runs: a huge skipGroups or append gap costs one run and one +// report, whatever its length. Mirrors Rust. +test("a huge skip costs one run", () => { + const FAR = 1n << 61n; + for (const append of [false, true]) { + const producer = new TrackProducer("test"); + producer.accept({}); + const reader = producer.subscribe({ delay: { max: null } }); + producer.createGroup(0n).finish(); + if (append) { + producer.createGroup(FAR).finish(); + producer.appendGroup().finish(); + } else { + producer.skipGroups(1n, FAR); + producer.createGroup(FAR).finish(); + } + const got: unknown[] = []; + for ( + let next = hooks.tryRecvGroup(reader, true); + next.kind !== "idle"; + next = hooks.tryRecvGroup(reader, true) + ) { + if (next.kind === "group") got.push(next.group.sequence); + else if (next.kind === "drop") got.push([next.start, next.end, next.skip === true]); + } + const groups = append ? [0n, FAR, FAR + 1n] : [0n, FAR]; + expect(got.filter((entry) => typeof entry === "bigint")).toEqual(groups); + expect(got.filter((entry) => typeof entry !== "bigint")).toEqual([[1n, FAR, true]]); + } +}); diff --git a/js/net/src/track.ts b/js/net/src/track.ts index e6c9d69f214..fba3e57e718 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -2463,9 +2463,9 @@ export class Subscriber { // Owe again every run this cursor gave up as stale, reported or not: an update reopens the // verdict. A group still cached is offered again as a widened skip is, and whatever is still // stale under the new terms is given up and reported again. Returns the runs taken back. A - // resumed reader reports only what its routes' copies give up, so it keeps its reports. + // resumed reader takes back what it queued and what its serving route's copy reported. #reofferStale(): Run[] { - if (this.#resume) return []; + if (this.#resume) return this.#resume.reofferStale(); const taken: Run[] = []; for (const run of this.#reportedStale) { this.#reported = removeRun(this.#reported, run.start, run.end); diff --git a/quest/m1/drop-set.md b/quest/m1/drop-set.md index 63d73686970..325fe4feffd 100644 --- a/quest/m1/drop-set.md +++ b/quest/m1/drop-set.md @@ -121,6 +121,20 @@ decision above and the PR's `## Waiting` item on out-of-order creation: (`skipGroups`, `appendGroup`). - ✅ **Draft:** SKIP names only sequences the publisher declared it will never produce, and it is final. +- ✅ **Run-length tracking** (refinement, same session): skipped and dropped + groups are kept as runs, never one entry per sequence, in Rust and JS: the + track's hole set, the implicit `append_group` skip, `skip_groups`, and what + the publisher reports. A huge `skip_groups` or append gap costs one run and + one SUBSCRIBE_SKIP (`a_huge_skip_costs_one_run`, + `lite07_skips_a_huge_run_at_once`, and their JS mirrors). +- ✅ **`create_group` refuses a skipped sequence and allows a dropped one** + (refinement, same session): a group stream wins over its drop, which fills + answering a miss and `claim_sequence` retries of an aborted sequence need. +- **A `create_group` gap error** (maintainer's idea, same session): not + implemented. No principled bound exists in the model: the cache and the + budget bound bytes and time, not sequence counts, a gap costs one run + whatever its length, and under time addressing every group leaves a gap + between starts. Left to the maintainer in the PR's `## Waiting`. - Not chosen: eager skip on `create_group` with a late group winning over it (the shipped design, which inferred SKIP from a gap that out-of-order creation makes meaningless); refusing every `create_group` below the last diff --git a/rs/moq-net/src/lite/publisher.rs b/rs/moq-net/src/lite/publisher.rs index 8616acd314e..71d32ab113f 100644 --- a/rs/moq-net/src/lite/publisher.rs +++ b/rs/moq-net/src/lite/publisher.rs @@ -4318,7 +4318,10 @@ impl TrackRun { self.owe(reopened); // A group whose stream already reset as stale is judged again under the new terms: // one still cached, in range, and fresh under the new budget is served again. The - // rest is dropped again, by run, on the next poll. + // rest is dropped again, by run, on the next poll. A run an earlier update left to + // drop again is judged again too, since these terms may serve it. + let pending = std::mem::take(&mut self.redrop); + self.expired.extend(pending.iter().cloned()); for run in std::mem::take(&mut self.expired).iter() { let (groups, gone) = self.track.reoffer_groups(run.clone()); self.owe( @@ -6871,6 +6874,41 @@ mod serve_group_test { assert_eq!(*log.writes.lock().unwrap(), [0, 4, 1, 0, 1, 1, 3, 2, 1, 1, 1, 2, 3, 0]); } + /// A huge declared skip goes out as one SUBSCRIBE_SKIP, at a cost independent of its + /// length, and the end settles over it. + #[kio::test] + async fn lite07_skips_a_huge_run_at_once() { + const FAR: u64 = 1 << 61; + let mut track = track::Producer::new( + Arc::new(broadcast::Info::default()), + "test", + crate::track::Info::timed(), + ); + let subscriber = track.subscribe(None); + let (mut run, mut writer, log) = lite07_run(SinkSession::new(Log::default()), subscriber); + write_group(&mut track, 0, 0); + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + track.skip_groups(1..FAR); + write_group(&mut track, FAR, 1); + track.finish().unwrap(); + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + kio::wait(|waiter| writer.poll_flush(waiter)).await.unwrap(); + + let writes = log.writes.lock().unwrap().clone(); + let mut buf = writes.as_slice(); + let (mut skips, mut drops) = (Vec::new(), 0); + while !buf.is_empty() { + let (msg, size) = lite::SubscribeResponse::decode_slice(buf, Version::Lite07).unwrap(); + buf = &buf[size..]; + match msg { + lite::SubscribeResponse::Skip(skip) => skips.push((skip.start, skip.end)), + lite::SubscribeResponse::Drop(_) => drops += 1, + _ => {} + } + } + assert_eq!((skips, drops), (vec![(1, FAR - 1)], 0)); + } + /// A relay that holds only the first group of a backlog answers with the largest its /// source reported, so a live joiner downstream starts at the newest group rather than /// the oldest one the budget serves. @@ -7089,6 +7127,50 @@ mod serve_group_test { } } + /// A run one update left to drop again is judged again under the next update that + /// arrives before it goes out, rather than dropped under terms that may serve it. + #[kio::test] + async fn a_pending_redrop_is_judged_again_under_a_later_update() { + let mut track = track::Producer::new( + Arc::new(broadcast::Info::default()), + "test", + crate::track::Info::timed(), + ); + let budget = Duration::from_millis(500); + let subscriber = + track.subscribe(track::Subscription::default().with_delay(track::Delay::default().with_max(budget))); + let (mut run, mut writer, log) = lite07_run(SinkSession::new(Log::default()), subscriber); + write_group(&mut track, 0, 0); + write_group(&mut track, 1, 1000); + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + let opened = run.ctx.outcomes.opened.load(Ordering::Relaxed); + write_group(&mut track, 2, 3000); + run.ctx.outcomes.dropped.lock().unwrap().push((0, StreamError::Old)); + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + assert_eq!(drop_updates(&mut writer, &log, Version::Lite07).await, [(0, 0, 0)]); + + // Update 1 keeps the budget, so group 0 is left to drop again; update 2 widens it + // before that drop goes out. + for (update, max_delay) in [(1, budget), (2, Duration::from_secs(10))] { + run.update(lite::SubscribeUpdate { + wanted_group: None, + priority: 0, + max_delay, + min_delay: None, + ranges: vec![lite::Range::span(None, 0, None, None)], + order: Order::Descending, + update, + }); + } + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + assert_eq!( + run.ctx.outcomes.opened.load(Ordering::Relaxed), + opened + 2, + "group 0 is served again under update 2, beside group 2" + ); + assert_eq!(drop_updates(&mut writer, &log, Version::Lite07).await, [(0, 0, 0)]); + } + /// Lite-05 and later name every sequence below the end that the subscription never /// got once the track ends: one skipped between groups, and one past the last. #[kio::test] diff --git a/rs/moq-net/src/model/resume.rs b/rs/moq-net/src/model/resume.rs index 421a4f3cfea..ad7edc815ff 100644 --- a/rs/moq-net/src/model/resume.rs +++ b/rs/moq-net/src/model/resume.rs @@ -901,6 +901,13 @@ impl Subscriber { self.reader().reoffer_groups(run) } + /// Take back every stale run this reader queued or the serving route's copy reported; + /// see [`track::Subscriber::reoffer_stale`]. + #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] + pub(super) fn reoffer_stale(&mut self) -> Vec> { + self.reader().reoffer_stale() + } + /// Poll for sequences the serving copy gave up that no copy may still deliver; see /// [`track::Subscriber::poll_drop`]. #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] @@ -1875,6 +1882,26 @@ impl Reader { (groups, given_up) } + /// Take back every stale run: those queued here, and those the serving copy reported, + /// which its cursor owes again. Only the serving copy's verdicts count (see + /// [`Self::poll_drop`]), so a replaced copy's are left alone. + #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] + fn reoffer_stale(&mut self) -> Vec> { + let mut runs: Vec<_> = self + .drops + .iter() + .filter(|(_, (hole, _))| hole.stale_budget().is_some()) + .map(|(run, _)| run.clone()) + .collect(); + for run in &runs { + self.drops.remove(run.clone()); + } + if let Poll::Ready(Some(sub)) = self.poll_serving(&kio::Waiter::noop()) { + runs.extend(sub.reoffer_stale()); + } + runs + } + /// The next run the serving copy gave up that `delivered` lacks and no other copy may /// still deliver. Only the serving copy's verdicts count, and a route failure is none: /// another route may still deliver the group, and a failed group it buffered continues @@ -3792,6 +3819,23 @@ mod test { assert_eq!(given_up, std::iter::once(1..3).collect::>()); } + /// A front's reader takes back the stale runs its serving route's copy reported, so a + /// relay's publisher re-offers them under a new update rather than keep them reported. + #[test] + fn a_resumed_reader_takes_back_its_stale_reports() { + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + routes.serve(a.consume()); + let mut sub = subscribe(&logical, HOUR); + write(&a, 0); + assert_eq!(drain(&mut sub, false), [0]); + a.give_up(1..3, track::Hole::stale(HOUR)); + assert_eq!(dropped(&mut sub), [(1..3, StreamError::Old)]); + assert_eq!(sub.reoffer_stale(), [1..3]); + assert_eq!(sub.reoffer_stale(), Vec::>::new()); + } + /// A run a replaced route held back is dropped with that route: the route serving next /// may still deliver it. #[test] diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index 479ce81b1be..a58a74292ac 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -9342,13 +9342,13 @@ impl Subscriber { /// Re-offer what this reader gave up as stale through [`Self::poll_drop`], as a wire /// whose every update reopens its stale drops does: each run is judged again under the - /// new terms. Returns the runs taken back, which are owed again. A front's reader - /// reports only what its copies give up, so it keeps its reports. + /// new terms. Returns the runs taken back, which are owed again. A front's reader takes + /// back what it queued and what its serving route's copy reported. #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] pub(crate) fn reoffer_stale(&mut self) -> Vec> { match &mut self.inner { Inner::Plain(cursor) => cursor.reoffer_stale(), - Inner::Resume(..) => Vec::new(), + Inner::Resume(resume, _) => resume.reoffer_stale(), } } @@ -23551,6 +23551,58 @@ mod test { write_group(&producer, 2); } + /// Skips are kept and reported as runs: a huge `skip_groups` or append gap costs one + /// entry and one report, whatever its length. + #[test] + #[cfg(feature = "async")] + fn a_huge_skip_costs_one_run() { + const FAR: u64 = 1 << 61; + let waiter = kio::Waiter::noop(); + // Presented just after group 0, since its sequence in milliseconds overflows. + let write_far = |producer: &Producer| { + let mut group = producer.create_group(group::Info { sequence: FAR }).unwrap(); + group + .write_frame(Timestamp::from_millis(1).unwrap(), &b"x"[..]) + .unwrap(); + group.finish().unwrap(); + }; + let producer = track_producer("test", Info::timed()); + let mut subscriber = producer.subscribe(replay().with_range(0..)); + write_group(&producer, 0); + producer.skip_groups(1..FAR); + assert_eq!(producer.state.read().drops.iter().count(), 1); + assert_eq!(subscriber.poll_drop(&waiter), Poll::Ready((1..FAR, Hole::Skip))); + write_far(&producer); + assert_eq!(drain_groups(&mut subscriber), [0, FAR]); + assert!(subscriber.poll_drop(&waiter).is_pending()); + + let producer = track_producer("test", Info::timed()); + let mut subscriber = producer.subscribe(replay().with_range(0..)); + write_group(&producer, 0); + write_far(&producer); + producer.append_group().unwrap().finish().unwrap(); + assert_eq!(producer.state.read().drops.iter().count(), 1); + assert_eq!(subscriber.poll_drop(&waiter), Poll::Ready((1..FAR, Hole::Skip))); + assert_eq!(drain_groups(&mut subscriber), [0, FAR, FAR + 1]); + } + + /// A dropped sequence may be created again, as a group stream wins over its drop, such as + /// one whose group was aborted: only a skip is final. + #[test] + #[cfg(feature = "async")] + fn an_aborted_sequence_is_created_again() { + let producer = track_producer("test", Info::timed()); + let mut subscriber = producer.subscribe(replay().with_range(0..)); + let mut group = producer.create_group(group::Info { sequence: 0 }).unwrap(); + group.abort(Error::Stream(StreamError::Cancel)).unwrap(); + write_group(&producer, 0); + producer.append_group().unwrap().finish().unwrap(); + assert!(producer.state.read().drops.is_empty()); + let mut read = drain_groups(&mut subscriber); + read.dedup(); + assert_eq!(read.last(), Some(&1)); + } + /// `skip_groups` leaves a cached group and a reason already given alone. #[test] #[cfg(feature = "async")] From 73b4f9f561ab89291f558f86f2ab1fed255ea713 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 18:59:45 -0800 Subject: [PATCH 20/68] fix(net): clippy and TypeDoc nits in the skip tests and docs Co-Authored-By: Claude Opus 5.5 --- js/net/src/error.ts | 4 ++-- rs/moq-net/src/model/resume.rs | 2 +- rs/moq-net/src/model/track.rs | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/js/net/src/error.ts b/js/net/src/error.ts index cd36c735389..de341eb02fc 100644 --- a/js/net/src/error.ts +++ b/js/net/src/error.ts @@ -408,8 +408,8 @@ function localStreamCode(err: unknown): StreamCode { /** * A group was created at a sequence its track declared skipped. * - * A skip says the sequence will never be produced, so it is final. See - * {@link TrackProducer.skipGroups}. Mirrors the Rust `Error::Skipped`. + * A skip says the sequence will never be produced, so it is final. See the track producer's + * `skipGroups`. Mirrors the Rust `Error::Skipped`. * * @public */ diff --git a/rs/moq-net/src/model/resume.rs b/rs/moq-net/src/model/resume.rs index ad7edc815ff..d6ba05f724e 100644 --- a/rs/moq-net/src/model/resume.rs +++ b/rs/moq-net/src/model/resume.rs @@ -3832,7 +3832,7 @@ mod test { assert_eq!(drain(&mut sub, false), [0]); a.give_up(1..3, track::Hole::stale(HOUR)); assert_eq!(dropped(&mut sub), [(1..3, StreamError::Old)]); - assert_eq!(sub.reoffer_stale(), [1..3]); + assert_eq!(sub.reoffer_stale(), vec![1..3]); assert_eq!(sub.reoffer_stale(), Vec::>::new()); } diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index a58a74292ac..7e3d4e518e5 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -23593,7 +23593,7 @@ mod test { fn an_aborted_sequence_is_created_again() { let producer = track_producer("test", Info::timed()); let mut subscriber = producer.subscribe(replay().with_range(0..)); - let mut group = producer.create_group(group::Info { sequence: 0 }).unwrap(); + let group = producer.create_group(group::Info { sequence: 0 }).unwrap(); group.abort(Error::Stream(StreamError::Cancel)).unwrap(); write_group(&producer, 0); producer.append_group().unwrap().finish().unwrap(); From 385f24e1671c57c62737af44a29ba4c01c5cba53 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 19:04:27 -0800 Subject: [PATCH 21/68] chore(js): record skipGroups and Skipped in the @moq/net API report Co-Authored-By: Claude Opus 5.5 --- js/net/api/index.api.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/js/net/api/index.api.md b/js/net/api/index.api.md index fd6dcb62248..b11647a052d 100644 --- a/js/net/api/index.api.md +++ b/js/net/api/index.api.md @@ -284,6 +284,10 @@ export namespace Error { }); readonly code: SessionCode; } + export class Skipped extends Error { + constructor(sequence: bigint); + readonly sequence: bigint; + } export class Stream extends Error { constructor(code: StreamCode, options?: Error.StreamOptions); readonly code: StreamCode; @@ -684,6 +688,7 @@ export namespace Track { largest(): Track.Position | undefined; readonly name: string; get priority(): number; + skipGroups(start: bigint, end: bigint): void; subscribe(options?: Track.Subscription): Track.Subscriber; get subscription(): Signals.Getter; writeFrame(frame: Group.Frame): void; From a87fa4029b3ccfe001c2d5445dc3603e966d933b Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 19:25:33 -0800 Subject: [PATCH 22/68] fix(net): report again what is stale at the max age bound on reopen A lite-07 update reopens every stale drop, but a group written off at the track's max age bound keeps no skip entry, so nothing offered it again or dropped it again and the subscriber waited for it. Whatever neither a skip nor a miss re-offers now goes back to the drop cursor as stale under the current budget and is reported again under the new update. JS mirrors it. Co-Authored-By: Claude Opus 5.5 --- js/net/src/track.test.ts | 32 ++++++++++++++++++++++ js/net/src/track.ts | 10 +++++++ rs/moq-net/src/lite/publisher.rs | 47 ++++++++++++++++++++++++++++++++ rs/moq-net/src/model/track.rs | 23 ++++++++++++++++ 4 files changed, 112 insertions(+) diff --git a/js/net/src/track.test.ts b/js/net/src/track.test.ts index 79420e2d91d..ed9c56b4c50 100644 --- a/js/net/src/track.test.ts +++ b/js/net/src/track.test.ts @@ -4996,3 +4996,35 @@ test("a huge skip costs one run", () => { expect(got.filter((entry) => typeof entry !== "bigint")).toEqual([[1n, FAR, true]]); } }); + +// A group stale at the track's max age bound is final under any budget, yet an update that reopens +// its stale drop must report it again rather than leave it owed: no skip offers it again. Mirrors +// Rust. +test("reopening stale drops reports again what is stale at the max age bound", () => { + const producer = new TrackProducer("test").accept({ timescale: Timescale.MILLI, maxAge: Milli(1500) }); + const reader = producer.subscribe({ delay: { max: Milli(1500) } }); + const drain = () => { + const got: unknown[] = []; + for ( + let next = hooks.tryRecvGroup(reader, true); + next.kind !== "idle"; + next = hooks.tryRecvGroup(reader, true) + ) { + if (next.kind === "group") got.push(next.group.sequence); + else if (next.kind === "drop") got.push([next.start, next.end, next.stale]); + } + return got; + }; + putGroup(producer, 0n, 0); + expect(drain()).toEqual([0n]); + for (const [sequence, millis] of [ + [1n, 1000], + [2n, 2000], + [3n, 3000], + [4n, 6000], + ] as const) + putGroup(producer, sequence, millis); + expect(drain()).toEqual([[1n, 2n, 1500], [2n, 3n, 1500], 3n, 4n]); + expect(hooks.reofferStale(reader)).toEqual([{ start: 1n, end: 3n }]); + expect(drain()).toEqual([[1n, 3n, 1500]]); +}); diff --git a/js/net/src/track.ts b/js/net/src/track.ts index fba3e57e718..8000bfff013 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -2479,6 +2479,16 @@ export class Subscriber { const sequences = [...this.#skipped.keys()].filter((sequence) => taken.some((run) => run.start <= sequence && sequence < run.end), ); + // What no skip offers again, such as a group written off at the track's max age bound, + // which no budget reverses, is still stale under the new terms: reported again. A + // received stale run is left to the scan above. Mirrors Rust. + let kept: Run[] = taken.map((run) => ({ ...run })); + for (const sequence of sequences) kept = removeRun(kept, sequence, sequence + 1n); + for (const dropped of this.#state.drops.peek()) kept = removeRun(kept, dropped.start, dropped.end); + if (kept.length > 0) { + const hole = staleHole(this.#drift().budget); + for (const run of kept) this.#pending = insertDrop(this.#pending, run.start, run.end, hole); + } const producer = this.#state.producer; if (sequences.length > 0 && producer && !this.#closed) reofferSequences(producer, this.#state, sequences); return taken; diff --git a/rs/moq-net/src/lite/publisher.rs b/rs/moq-net/src/lite/publisher.rs index 71d32ab113f..a29caac0222 100644 --- a/rs/moq-net/src/lite/publisher.rs +++ b/rs/moq-net/src/lite/publisher.rs @@ -7032,6 +7032,53 @@ mod serve_group_test { } } + /// A group stale at the track's max age bound is final under any budget, yet a lite-07 + /// update still reopens its drop at the subscriber, so it is dropped again under the new + /// update rather than left owed. + #[kio::test] + async fn an_update_drops_again_what_is_stale_at_the_bound() { + for widen in [false, true] { + let budget = Duration::from_millis(1500); + let mut track = track::Producer::new( + Arc::new(broadcast::Info::default()), + "test", + crate::track::Info::timed().with_max_age(budget), + ); + let subscriber = + track.subscribe(track::Subscription::default().with_delay(track::Delay::default().with_max(budget))); + let (mut run, mut writer, log) = lite07_run(SinkSession::new(Log::default()), subscriber); + write_group(&mut track, 0, 0); + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + for (sequence, millis) in [(1, 1000), (2, 2000), (3, 3000), (4, 6000)] { + write_group(&mut track, sequence, millis); + } + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + assert_eq!(drop_updates(&mut writer, &log, Version::Lite07).await, [(1, 2, 0)]); + let opened = run.ctx.outcomes.opened.load(Ordering::Relaxed); + + run.update(lite::SubscribeUpdate { + wanted_group: None, + priority: 5, + max_delay: if widen { Duration::from_secs(10) } else { budget }, + min_delay: None, + ranges: vec![lite::Range::span(None, 0, None, None)], + order: Order::Descending, + update: 1, + }); + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + assert_eq!( + run.ctx.outcomes.opened.load(Ordering::Relaxed), + opened, + "widen={widen}: the bound keeps them stale" + ); + assert_eq!( + drop_updates(&mut writer, &log, Version::Lite07).await, + [(1, 2, 0), (1, 2, 1)], + "widen={widen}: dropped again under update 1" + ); + } + } + /// A group whose stream expired as stale is dropped again under every later update on /// lite-07, by run, since the subscriber reopens each stale drop and the stream already /// reset. Earlier versions' drops are final. diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index 7e3d4e518e5..3c4a8ea7840 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -7612,16 +7612,39 @@ impl Cursor { return Vec::new(); }; let runs = drops.take_stale(); + // What neither a skip nor a miss offers again: a group written off at the track's max + // age bound, which no budget reverses. It is still stale under the new terms. + let mut kept = RangeSet::new(); for run in &runs { + kept.insert(run.clone()); if let Some(misses) = &mut self.misses { + for missed in misses.given_up().overlapping(run) { + kept.remove(missed.clone()); + } misses.retry(run.clone()); } if let Some(last) = run.end.checked_sub(1) { for skipped in self.skipped.within(run.start..=last) { + kept.remove(*skipped.start()..skipped.end().saturating_add(1)); self.reowe(skipped); } } } + let budget = { + let state = self.state.read(); + // A received stale run is folded in again by the next scan, if the track holds it. + for run in kept.clone().iter() { + for (dropped, _) in state.drops.overlapping(run) { + kept.remove(dropped.clone()); + } + } + clamp_max_delay(self.subscription.read().budget(), state.max_age_bound()) + }; + if let Some(drops) = &mut self.drops { + for run in kept.iter() { + drops.give_up(run.clone(), Hole::stale(budget)); + } + } runs } From 9b2f3c6984b1642e81dcd1c223a9757fc650adc2 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 19:51:32 -0800 Subject: [PATCH 23/68] docs(quest): record no create_group gap error for drop set Co-Authored-By: Claude Opus 5.5 --- quest/m1/drop-set.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/quest/m1/drop-set.md b/quest/m1/drop-set.md index 325fe4feffd..d95ca0d08e3 100644 --- a/quest/m1/drop-set.md +++ b/quest/m1/drop-set.md @@ -130,11 +130,13 @@ decision above and the PR's `## Waiting` item on out-of-order creation: - ✅ **`create_group` refuses a skipped sequence and allows a dropped one** (refinement, same session): a group stream wins over its drop, which fills answering a miss and `claim_sequence` retries of an aborted sequence need. -- **A `create_group` gap error** (maintainer's idea, same session): not - implemented. No principled bound exists in the model: the cache and the - budget bound bytes and time, not sequence counts, a gap costs one run - whatever its length, and under time addressing every group leaves a gap - between starts. Left to the maintainer in the PR's `## Waiting`. +- ✅ **No `create_group` gap error, for now** (maintainer, same session): + gaps stay legal, and `append_group` or `skip_groups` declares them. No + principled bound exists in the model: the cache and the budget bound bytes + and time, not sequence counts, a gap costs one run whatever its length, + and under time addressing every group leaves a gap between starts. Not + chosen: a debug log on a gap; a configurable size limit (an arbitrary + knob). - Not chosen: eager skip on `create_group` with a late group winning over it (the shipped design, which inferred SKIP from a gap that out-of-order creation makes meaningless); refusing every `create_group` below the last From 660ba2004f3a388abccea7a07779154266f5dd38 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 20:10:38 -0800 Subject: [PATCH 24/68] fix(net): keep forwarded OLD reports across route changes; SKIP replaces OLD A front's reader now records every stale run it reported, whichever copy gave it up, and `reoffer_stale` takes those back too. A route change since the report no longer leaves the run in the publisher's `served` set after a downstream update reopens it. A received SKIP, and `skip_groups` and `append_group`'s implicit skip, replace a stale (OLD) drop of the same sequences instead of leaving it for the next update to supersede. A final DROP keeps its verdict. The lite-07 subscriber also takes a skipped or finally dropped run out of its stale record, so the next update doesn't owe it again, and ignores an empty DROP or SKIP range rather than panic on it. JS mirrors both, and a received SKIP now goes through `skipGroups`, so it no longer overwrites a final drop or a cached group. Also fixes the main merge's JS type errors: the fill callbacks pass an Error where `#drop` now takes a Hole. Co-Authored-By: Claude Opus 5.5 --- drafts/draft-lcurley-moq-lite.md | 3 +- js/net/src/fill.test.ts | 2 +- js/net/src/lite/subscriber.test.ts | 63 +++++++++++++++++ js/net/src/lite/subscriber.ts | 16 +++-- js/net/src/resume.test.ts | 31 +++++++++ js/net/src/resume.ts | 18 +++-- js/net/src/track.test.ts | 5 +- js/net/src/track.ts | 18 ++--- quest/m1/drop-set.md | 14 +++- rs/moq-net/src/lite/subscriber.rs | 106 ++++++++++++++++++++++++++--- rs/moq-net/src/model/resume.rs | 49 +++++++++++-- rs/moq-net/src/model/track.rs | 40 +++++++---- 12 files changed, 315 insertions(+), 50 deletions(-) diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index 4751d1f375a..801e902ba7a 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -1598,7 +1598,7 @@ A stale drop whose `Update` a later update has replaced is no verdict, since the The next SUBSCRIBE_UPDATE reopens every stale drop the subscriber accounted for, which it owes again until the publisher serves the group or drops it again under the new `Update`, as described in [SUBSCRIBE_UPDATE](#subscribe-update). A relay that received a stale drop settles with it only a reader whose own budget is no wider than the one the relay sent with that update and whose Ranges cover the relay's own, which the verdict was judged against. -A drop with any other Error Code is final, whatever its `Update`. +A drop with any other Error Code is final, whatever its `Update`, and replaces a stale drop of the same sequences, so no update reopens them. A Group Stream for a sequence the subscriber already saw dropped, as after such a re-offer, wins over the drop: the subscriber delivers it. @@ -1629,6 +1629,7 @@ SUBSCRIBE_SKIP Message { A skip names only sequences the original publisher declared it will never produce, and it is final: no Group will ever exist at those sequences, whatever the terms. A publisher MUST NOT later create a Group at a sequence it skipped, and a subscriber MUST NOT deliver a Group Stream for one. It carries no error and ends no reader, since there is nothing to read; a subscriber stops waiting for those sequences. +A skip replaces a stale (OLD) drop of the same sequences, so no update reopens them; a sequence already named in a final drop keeps it. A gap in the sequence numbers says nothing on its own: a publisher may create Groups out of order, or create again a sequence whose Group it aborted, so it never skips a sequence merely because a higher Group exists or the Track ended without it. Only the application producing the Track knows a sequence will never exist, for example when it numbers Groups sparsely, or when it moves on to the next Group and so declares that it will not go back to fill the sequences below. diff --git a/js/net/src/fill.test.ts b/js/net/src/fill.test.ts index d1b17886121..c05f01a86a2 100644 --- a/js/net/src/fill.test.ts +++ b/js/net/src/fill.test.ts @@ -701,7 +701,7 @@ test("fill: a group the track gives up elsewhere answers the fill", async () => // An upstream SUBSCRIBE_DROP, say: the handler is no longer asked for it. const cut = new StreamError(StreamCode.Cancel, { message: "cut upstream" }); - hooks.dropTrackGroups(track, 2n, 3n, cut); + hooks.dropTrackGroups(track, 2n, 3n, { error: cut }); expect(fill.wanted()).toBeUndefined(); await expect(fetching).rejects.toBe(cut); diff --git a/js/net/src/lite/subscriber.test.ts b/js/net/src/lite/subscriber.test.ts index 2f9a0fe7851..838aed0055a 100644 --- a/js/net/src/lite/subscriber.test.ts +++ b/js/net/src/lite/subscriber.test.ts @@ -11,6 +11,7 @@ import { reason, SessionCode, SessionError, + Skipped, StreamCode, StreamError, TimestampMismatch, @@ -37,6 +38,7 @@ import { SubscribeEnd, SubscribeOk, type SubscribeResponse, + SubscribeSkip, SubscribeStart, } from "./subscribe.ts"; import { Minted, SUBSCRIBE_SETUP_TIMEOUT_MS, Subscriber } from "./subscriber.ts"; @@ -2563,6 +2565,67 @@ test("a lite-07 publisher FIN keeps a covered bounded subscription's copy", asyn } }); +// A SKIP, or any final drop, that follows an OLD drop of the same groups replaces it: an update +// reopens only stale verdicts, so the groups stay accounted for, and the skip stays final rather +// than being superseded with the OLD drop. An empty range names nothing. Mirrors Rust. +test("a skip or final drop replaces an earlier stale one on lite-07", async () => { + const version = Version.DRAFT_07; + const { quic, streams, opened } = fakeSession(); + const subscriber = new Subscriber(quic, version, HopSchema.parse(1n)); + let entry: { track: Producer; tail: { covers(start: bigint, end: bigint): boolean } } | undefined; + const internal = subscriber as unknown as { + runResponses: (stream: unknown, entry: NonNullable) => Promise; + }; + const runResponses = internal.runResponses.bind(internal); + const responses = spyOn(internal, "runResponses").mockImplementation((stream, captured) => { + entry = captured; + return runResponses(stream, captured); + }); + const broadcast = subscriber.consume(Path.from("room")); + const track = broadcast.track("video").subscribe({ delay: { max: Milli(500) } }); + try { + await opened(0); + await answerTrackInfo(streams[0], version); + const stream = await opened(1); + await stream.reading; + const chunks: Uint8Array[] = []; + const writer = new Writer( + new WritableStream({ write: (chunk) => void chunks.push(new Uint8Array(chunk)) }), + version, + ); + const old = (start: bigint, end: bigint): SubscribeResponse => ({ + drop: new SubscribeDrop({ start, end, error: StreamCode.Old }), + }); + for (const resp of [ + { start: new SubscribeStart([0n]) }, + old(1n, 2n), + old(4n, 4n), + { skip: new SubscribeSkip({ start: 1n, end: 2n }) }, + { drop: new SubscribeDrop({ start: 4n, end: 4n, error: StreamCode.NotFound }) }, + // Empty ranges. + old(6n, 5n), + { skip: new SubscribeSkip({ start: 6n, end: 5n }) }, + ] satisfies SubscribeResponse[]) { + await encodeSubscribeResponse(writer, resp, version); + } + for (const chunk of chunks) stream.inbound.enqueue(chunk); + await drainReactions(); + if (!entry) throw new Error("no subscription"); + + track.update({ priority: 1, delay: { max: Milli(500) } }); + await drainReactions(); + expect(entry.tail.covers(1n, 3n)).toBe(true); + expect(entry.tail.covers(4n, 5n)).toBe(true); + // Still skipped after the update. + expect(() => entry?.track.createGroup(1n)).toThrow(Skipped); + } finally { + responses.mockRestore(); + track.close(); + broadcast.close(); + subscriber.close(); + } +}); + // A widening reopens a covered copy, and its readers leaving while the publisher never takes the // new SUBSCRIBE gives the setup up: its stream is reset and the copy released, not held forever. test("a reopened lite-07 subscription ends once its readers leave", async () => { diff --git a/js/net/src/lite/subscriber.ts b/js/net/src/lite/subscriber.ts index 16b70bf1401..147570541e8 100644 --- a/js/net/src/lite/subscriber.ts +++ b/js/net/src/lite/subscriber.ts @@ -30,10 +30,10 @@ import { insertDrop, maxDelayOf, minDelayOf, + removeRun, scopeCaptures, scopeHead, scopeOverlaps, - skipHole, staleHole, } from "../internal.ts"; import type { Lister } from "../listing.ts"; @@ -1882,12 +1882,18 @@ export class Subscriber { entry.tail.account(start, end + 1n); // Judged under the budget this subscriber sent with that update. const hole = stale ? staleHole(entry.maxDelay ?? 0) : { error }; - if (stale) entry.stale = insertDrop(entry.stale ?? [], start, end + 1n, hole); + // A final drop is never reopened, even where it replaces a stale one. + entry.stale = stale + ? insertDrop(entry.stale ?? [], start, end + 1n, hole) + : removeRun(entry.stale ?? [], start, end + 1n); hooks.dropTrackGroups(entry.track, start, end + 1n, hole); } else if ("skip" in resp) { - // Never produced, so nothing is waited for and no reader ends. - entry.tail.account(resp.skip.start, resp.skip.end + 1n); - hooks.dropTrackGroups(entry.track, resp.skip.start, resp.skip.end + 1n, skipHole()); + // Never produced, so nothing is waited for and no reader ends. A skip is final: it + // replaces a stale drop of the same groups, which an update would otherwise reopen. + const { start, end } = resp.skip; + entry.tail.account(start, end + 1n); + entry.stale = removeRun(entry.stale ?? [], start, end + 1n); + entry.track.skipGroups(start, end + 1n); } } } diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts index 7e19bc98961..04ffb3ce4e1 100644 --- a/js/net/src/resume.test.ts +++ b/js/net/src/resume.test.ts @@ -3383,3 +3383,34 @@ test("a resumed reader takes back its stale reports", () => { front.consumer.close(); a.close(); }); + +// A stale run one route reported is still taken back once another serves: the update reopened it +// downstream, and the new route's copy knows nothing of it, so the groups it delivers there +// afterwards are owed rather than kept reported. Mirrors Rust. +test("a resumed reader takes back a replaced route's stale reports", async () => { + const front = new Front(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + const ta = copy(a); + const tb = copy(b); + front.serve(a.consume()); + const sub = front.consumer.track("seq").subscribe({ delay: { max: Milli(60_000) } }); + ta.write(0n); + expect(drain(sub)).toEqual([0n]); + hooks.dropTrackGroups(ta.producer, 1n, 3n, staleHole(60_000)); + expect(hooks.tryDrop(sub)).toMatchObject({ kind: "drop", start: 1n, end: 3n, stale: 60_000 }); + + front.serve(b.consume()); + await settle(); + expect(hooks.tryDrop(sub)).toBeUndefined(); + // Reopened, though the new route never reported it. + expect(hooks.reofferStale(sub)).toEqual([{ start: 1n, end: 3n }]); + expect(hooks.reofferStale(sub)).toEqual([]); + tb.write(1n); + tb.write(2n); + expect(drain(sub)).toEqual([1n, 2n]); + sub.close(); + front.consumer.close(); + a.close(); + b.close(); +}); diff --git a/js/net/src/resume.ts b/js/net/src/resume.ts index 059e474958f..e7f36b364ff 100644 --- a/js/net/src/resume.ts +++ b/js/net/src/resume.ts @@ -342,6 +342,11 @@ class Reader implements ResumeSource { #delivered = new Delivered(); /** Runs the serving copy gave up and the reader has yet to report; see {@link #tryDrop}. */ #drops: DropRun[] = []; + /** + * The stale runs reported since the last {@link reofferStale}, whichever copy gave them up: an + * update reopens each, and the copy that reported it may no longer serve. + */ + #reportedStale: Run[] = []; /** Whether groups were ever asked for; until then the reader keeps no replaced copy. */ #groupsRead = false; /** The groups handed out and still read, which other copies may back; see {@link #backs}. */ @@ -608,12 +613,16 @@ class Reader implements ResumeSource { } /** - * Take back every stale run: those queued here, and those the serving copy reported, which - * its cursor owes again. Only the serving copy's verdicts count, so a replaced copy's are left - * alone. Mirrors the Rust `Reader::reoffer_stale`. + * Take back every stale run: those queued here, those this reader reported, and those the + * serving copy reported, which its cursor owes again. What this reader reported counts + * whichever copy gave it up, since a route change since leaves the serving copy knowing + * nothing of it, though the update reopened it. Only the serving copy is judged again: a + * replaced copy may no longer deliver, so what it gave up is the new route's to deliver or + * give up. Mirrors the Rust `Reader::reoffer_stale`. */ reofferStale(): Run[] { - const taken: Run[] = []; + const taken = this.#reportedStale; + this.#reportedStale = []; for (const run of this.#drops) if (run.stale !== undefined) insertRun(taken, run.start, run.end); this.#drops = this.#drops.filter((run) => run.stale === undefined); const serving = this.#serving(); @@ -789,6 +798,7 @@ class Reader implements ResumeSource { if (open) { // Report once without marking it delivered: a late fill may still land. this.#drops = removeRun(this.#drops, open.start, open.end); + if (run.stale !== undefined) insertRun(this.#reportedStale, open.start, open.end); return { kind: "drop", ...open, error: run.error, skip: run.skip, stale: run.stale }; } } diff --git a/js/net/src/track.test.ts b/js/net/src/track.test.ts index ed9c56b4c50..f85bd7de140 100644 --- a/js/net/src/track.test.ts +++ b/js/net/src/track.test.ts @@ -4914,7 +4914,8 @@ test("a late local group reaches a reader, since creating out of order skips not }); // A skip is final: creating at a sequence an append or skipGroups skipped throws, while a dropped -// sequence may be created again. skipGroups leaves cached and given-up sequences alone. Mirrors Rust. +// sequence may be created again. skipGroups leaves cached sequences and final drops alone, and +// replaces a stale drop, which new terms would otherwise supersede. Mirrors Rust. test("creating a skipped sequence throws Skipped", () => { const producer = new TrackProducer("test"); producer.accept({}); @@ -4936,7 +4937,9 @@ test("creating a skipped sequence throws Skipped", () => { const reader = other.subscribe({ delay: { max: null } }); other.createGroup(1n).finish(); hooks.dropTrackGroups(other, 3n, 4n, { error: DROPPED }); + hooks.dropTrackGroups(other, 4n, 5n, staleHole(1000)); other.skipGroups(0n, 5n); + hooks.supersedeStale(other, undefined); const drops: unknown[] = []; for (let next = hooks.tryRecvGroup(reader, true); next.kind !== "idle"; next = hooks.tryRecvGroup(reader, true)) { if (next.kind === "drop") drops.push([next.start, next.end, next.skip === true]); diff --git a/js/net/src/track.ts b/js/net/src/track.ts index 838f309cb62..c79d2b0671c 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -978,7 +978,7 @@ export class Producer { if (info !== undefined && this.#state.info.peek() === undefined) this.accept(info); return this.#createGroup(sequence, 0); }, - drop: (start, end, error) => this.#drop(start, end, error), + drop: (start, end, error) => this.#drop(start, end, { error }), holds: (sequence) => cachedHolds(this, sequence), closed: () => this.#state.closed.peek() !== undefined, released: this.#state.closed, @@ -1720,8 +1720,8 @@ export class Producer { * This and {@link appendGroup} are the only sources of a skip: a gap between created groups * says nothing, since groups may be created out of order, so only the producer knows a * sequence will never exist. A skip is final: creating a group at a skipped sequence throws - * {@link Skipped}. A sequence already produced or given up keeps what it is. - * Mirrors the Rust `skip_groups`. + * {@link Skipped}. A sequence already produced or given up for good keeps what it is; a stale + * drop, which new terms may supersede, is replaced. Mirrors the Rust `skip_groups`. */ skipGroups(start: bigint, end: bigint): void { if (this.#state.closed.peek() !== undefined) return; @@ -1732,10 +1732,12 @@ export class Producer { this.#skipGaps(cached, start, end); } - // Skip what of `[start, end)` neither `kept` nor a drop already names. + // Skip what of `[start, end)` neither `kept` nor a final drop already names. A stale drop is + // replaced: new terms would supersede it, leaving the sequence owed, while a skip is final. #skipGaps(kept: readonly Run[], start: bigint, end: bigint): void { + const settled = this.#state.drops.peek().filter((run) => run.stale === undefined); for (const gap of runGaps(kept, start, end)) { - for (const open of runGaps(this.#state.drops.peek(), gap.start, gap.end)) { + for (const open of runGaps(settled, gap.start, gap.end)) { this.#drop(open.start, open.end, skipHole()); } } @@ -1748,8 +1750,8 @@ export class Producer { } // Settle an append at `sequence`: skip every sequence below it this track's own publisher - // never produced and did not give up, from the lowest it produced since its last append, - // since appending says it will not go back. A reason already given stands. + // never produced and did not give up for good, from the lowest it produced since its last + // append, since appending says it will not go back. A final reason already given stands. #settleAppend(sequence: bigint): void { const from = this.#appended ?? this.#produced[0]?.start; if (from !== undefined) this.#skipGaps(this.#produced, from, sequence); @@ -2751,7 +2753,7 @@ export class Subscriber { priority: () => this.#state.update.peek()?.priority ?? 0, maxDelay: () => this.#drift().budget, order: () => this.#state.update.peek()?.order ?? "descending", - refuse: (start, end, error) => this.#state.drops.update((drops) => insertDrop(drops, start, end, error)), + refuse: (start, end, error) => this.#state.drops.update((drops) => insertDrop(drops, start, end, { error })), }; return this.#state.asker; } diff --git a/quest/m1/drop-set.md b/quest/m1/drop-set.md index d95ca0d08e3..550a7ae61d0 100644 --- a/quest/m1/drop-set.md +++ b/quest/m1/drop-set.md @@ -97,8 +97,8 @@ decision above and the PR's `## Waiting` item on out-of-order creation: sequence will never exist. moq-archive's replay is the in-tree consumer of `skip_groups`, for the sequences up to a recorded end it holds nothing for; a producer that numbers sparsely would call it too. moq-mux's live - encoders use `append_group`. The relay forwards received SKIPs through the - crate-internal `give_up`. + encoders use `append_group`. The relay records a received SKIP with + `skip_groups` (below). - ✅ **Both inferences are removed:** the eager skip of the gap in `create_group`, and the automatic SKIP at finish for never-arrived sequences below the final one (`TrackState::missed` and the publisher's @@ -107,6 +107,16 @@ decision above and the PR's `## Waiting` item on out-of-order creation: - ✅ **SKIP is final:** creating a group (or appending a group or datagram) at a skipped sequence is refused with `Error::Skipped` (JS `Skipped`). Re-creating a dropped or aborted sequence still works. +- **A SKIP replaces a stale (OLD) drop, but not a final one** (decided in + `/quest-iterate` from review, 2026-11-04). OLD is the only verdict an + update reopens: a SKIP that left an earlier OLD in place let the next + update supersede it, so a relay waited on or refilled a sequence its + upstream had declared never produced. A final DROP already settles every + reader and no update reopens it, so the first final verdict stands. The + rule lives in the model, so `skip_groups` and `append_group`'s implicit + skip follow it too. A received SKIP or final DROP also takes its groups + out of the subscriber's stale record, so the next update doesn't owe them + again. - ✅ **`append_group` is an implicit skip** (refinement, same session): it means "the next group", so the producer declares it won't go back, and it skips every sequence below it that never arrived and wasn't already given diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index 71f70686dd4..1c51c0ded2e 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -4144,13 +4144,11 @@ mod tests { assert!(matches!(reader.recv_group().await, Err(Error::ProtocolViolation))); } - /// On lite-07 an OLD drop is a stale verdict judged under the budget this subscriber sent - /// with the update it names. Any SUBSCRIBE_UPDATE reopens every stale run in the tail, - /// whether or not it widens the budget, and an OLD drop naming a superseded update is no - /// verdict: still owed, and given up on no reader. Any other drop is final, whatever update - /// it names. - #[kio::test] - async fn an_update_reopens_every_stale_drop_and_supersedes_the_rest() { + /// A lite-07 subscription's serve loop with an active SUBSCRIBE under update 0 and the + /// max age `budget`, its broadcast, and the tail it accounts on. + async fn lite07_serving( + budget: Duration, + ) -> (TrackServe, ServeLoop, kio::Producer, crate::broadcast::Producer) { let version = Version::Lite07; let session = crate::lite::test_transport::ScriptedSession::new(Vec::new()); let subscriber = Subscriber::new(SubscriberConfig { @@ -4176,7 +4174,6 @@ mod tests { let broadcast = crate::broadcast::Info::new().produce(); let request = broadcast.reserve_track("video").unwrap(); let mut serving = ServeLoop::new(&serve, request, crate::track::Info::timed()); - let budget = Duration::from_millis(500); let tail = kio::Producer::new(Tail::default()); serving.sub = Sub::Active(SubStream { order: crate::track::Order::Descending, @@ -4202,6 +4199,19 @@ mod tests { update: 0, stale: RangeSet::new(), }); + (serve, serving, tail, broadcast) + } + + /// On lite-07 an OLD drop is a stale verdict judged under the budget this subscriber sent + /// with the update it names. Any SUBSCRIBE_UPDATE reopens every stale run in the tail, + /// whether or not it widens the budget, and an OLD drop naming a superseded update is no + /// verdict: still owed, and given up on no reader. Any other drop is final, whatever update + /// it names. + #[kio::test] + async fn an_update_reopens_every_stale_drop_and_supersedes_the_rest() { + let version = Version::Lite07; + let budget = Duration::from_millis(500); + let (serve, mut serving, tail, _broadcast) = lite07_serving(budget).await; let mut reader = serving.serving.subscribe( crate::track::Subscription::default() .with_delay(crate::track::Delay::default().with_max(budget)) @@ -4268,6 +4278,68 @@ mod tests { ); } + /// A SKIP, or any final drop, that follows an OLD drop of the same groups replaces it: an + /// update reopens only stale verdicts, so the groups stay accounted for, and the skip + /// stays final rather than being superseded with the OLD drop. An empty range names + /// nothing. + #[kio::test] + async fn a_skip_or_final_drop_replaces_an_earlier_stale_one() { + let version = Version::Lite07; + let budget = Duration::from_millis(500); + let (serve, mut serving, tail, _broadcast) = lite07_serving(budget).await; + let code = u64::from(crate::coding::StreamCodes::encode_stream_code( + &version, + &crate::StreamError::Old, + )); + let old = |start, end| { + lite::SubscribeResponse::Drop(lite::SubscribeDrop { + start, + end, + error: code, + update: 0, + }) + }; + let covers = |groups: std::ops::Range| tail.read().covers(groups); + + serving.apply_response(&serve, old(1, 2)).unwrap(); + serving.apply_response(&serve, old(4, 4)).unwrap(); + serving + .apply_response(&serve, lite::SubscribeResponse::Skip(lite::SubscribeSkip { start: 1, end: 2 })) + .unwrap(); + serving + .apply_response( + &serve, + lite::SubscribeResponse::Drop(lite::SubscribeDrop::new( + 4..=4, + &crate::StreamError::App(5), + 0, + version, + )), + ) + .unwrap(); + // Empty ranges. + serving.apply_response(&serve, old(6, 5)).unwrap(); + serving + .apply_response(&serve, lite::SubscribeResponse::Skip(lite::SubscribeSkip { start: 6, end: 5 })) + .unwrap(); + + { + let Sub::Active(active) = &mut serving.sub else { + unreachable!("an active subscription"); + }; + buffer_update(active, &serving.serving, serve.subscriber.runtime.now()).unwrap(); + } + assert!(covers(1..3), "the skip is final"); + assert!(covers(4..5), "the drop is final"); + assert!( + matches!( + serving.serving.create_group(crate::group::Info { sequence: 1 }), + Err(Error::Skipped) + ), + "still skipped after the update" + ); + } + /// A SUBSCRIBE_DROP ends a group still arriving now, with its Error Code on lite-07. /// Earlier versions carry an application code there, so the same bytes read as /// OLD. @@ -9609,6 +9681,10 @@ impl ServeLoop { // one still arriving ends with the drop's error now. lite::SubscribeResponse::Drop(dropped) => { let groups = dropped.start..dropped.end.saturating_add(1); + // An empty range (Group Start = Group End + 1) names nothing. + if groups.is_empty() { + return Ok(()); + } let error = dropped.error(serve.subscriber.version); // On lite-07 an OLD drop is a stale verdict, which the next update reopens; every // other drop is final, whatever update it names. @@ -9626,17 +9702,27 @@ impl ServeLoop { active.stale.insert(groups.clone()); track::Hole::stale(active.max_delay) } - false => track::Hole::drop(error), + // Final, so no update reopens it, even where it replaces a stale drop. + false => { + active.stale.remove(groups.clone()); + track::Hole::drop(error) + } }; self.serving.give_up(groups, hole); } } - // Never produced, so nothing is waited for and no reader ends. + // Never produced, so nothing is waited for and no reader ends. A skip is final: it + // replaces a stale drop of the same groups, which an update would otherwise reopen. lite::SubscribeResponse::Skip(skipped) => { let groups = skipped.start..skipped.end.saturating_add(1); + // An empty range (Group Start = Group End + 1) names nothing. + if groups.is_empty() { + return Ok(()); + } if let Ok(mut tail) = active.tail.write() { tail.account(groups.clone(), serve.subscriber.runtime.now()); } + active.stale.remove(groups.clone()); self.serving.skip_groups(groups); } // OK just resolves the range (the producer already orders diff --git a/rs/moq-net/src/model/resume.rs b/rs/moq-net/src/model/resume.rs index d6ba05f724e..10725ba9d0f 100644 --- a/rs/moq-net/src/model/resume.rs +++ b/rs/moq-net/src/model/resume.rs @@ -374,6 +374,7 @@ impl Consumer { groups_read: false, out: Vec::new(), drops: RangeMap::new(), + reported_stale: RangeSet::new(), declared: None, declared_error: None, refused: false, @@ -1118,6 +1119,9 @@ struct Reader { /// Each with whether it is a received stale run the serving copy held, which new terms /// can supersede. drops: RangeMap, + /// The stale runs reported since the last [`Reader::reoffer_stale`], whichever copy gave + /// them up: an update reopens each, and the copy that reported it may no longer serve. + reported_stale: RangeSet, } impl Reader { @@ -1882,24 +1886,29 @@ impl Reader { (groups, given_up) } - /// Take back every stale run: those queued here, and those the serving copy reported, - /// which its cursor owes again. Only the serving copy's verdicts count (see - /// [`Self::poll_drop`]), so a replaced copy's are left alone. + /// Take back every stale run: those queued here, those this reader reported, and those + /// the serving copy reported, which its cursor owes again. What this reader reported + /// counts whichever copy gave it up, since a route change since leaves the serving copy + /// knowing nothing of it, though the update reopened it. Only the serving copy is + /// judged again: a replaced copy may no longer deliver, so what it gave up is the new + /// route's to deliver or give up. #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] fn reoffer_stale(&mut self) -> Vec> { - let mut runs: Vec<_> = self + let mut runs = std::mem::take(&mut self.reported_stale); + let queued: Vec<_> = self .drops .iter() .filter(|(_, (hole, _))| hole.stale_budget().is_some()) .map(|(run, _)| run.clone()) .collect(); - for run in &runs { + for run in queued { self.drops.remove(run.clone()); + runs.insert(run); } if let Poll::Ready(Some(sub)) = self.poll_serving(&kio::Waiter::noop()) { runs.extend(sub.reoffer_stale()); } - runs + runs.into_iter().collect() } /// The next run the serving copy gave up that `delivered` lacks and no other copy may @@ -1976,6 +1985,9 @@ impl Reader { // Reported once. As on a plain cursor, the report is not final: a miss (a fill given // up) may still land, and is handed out if it does. self.drops.remove(open.clone()); + if err.stale_budget().is_some() { + self.reported_stale.insert(open.clone()); + } Poll::Ready((open, err)) } @@ -3836,6 +3848,31 @@ mod test { assert_eq!(sub.reoffer_stale(), Vec::>::new()); } + /// A stale run one route reported is still taken back once another serves: the update + /// reopened it downstream, and the new route's copy knows nothing of it, so the groups it + /// delivers there afterwards are owed rather than kept reported. + #[test] + fn a_resumed_reader_takes_back_a_replaced_routes_stale_reports() { + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + routes.serve(a.consume()); + let mut sub = subscribe(&logical, HOUR); + write(&a, 0); + assert_eq!(drain(&mut sub, false), [0]); + a.give_up(1..3, track::Hole::stale(HOUR)); + assert_eq!(dropped(&mut sub), [(1..3, StreamError::Old)]); + + let b = copy(); + routes.serve(b.consume()); + assert_eq!(dropped(&mut sub), []); + assert_eq!(sub.reoffer_stale(), vec![1..3], "reopened, though the new route never reported it"); + assert_eq!(sub.reoffer_stale(), Vec::>::new()); + write(&b, 1); + write(&b, 2); + assert_eq!(drain(&mut sub, false), [1, 2]); + } + /// A run a replaced route held back is dropped with that route: the route serving next /// may still deliver it. #[test] diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index dd9fa342c99..0b441119e93 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -1166,12 +1166,27 @@ impl TrackState { self.drops.get(&sequence).is_some_and(|hole| *hole == Hole::Skip) } - /// Skip what of `groups` is neither cached nor given up; see [`Producer::skip_groups`]. - fn skip_groups(&mut self, groups: ops::Range) { + /// What of `groups` no final verdict names: a gap, or a stale drop, which new terms may + /// supersede and so says nothing for good. + fn unsettled(&self, groups: ops::Range) -> RangeSet { let mut left = RangeSet::new(); - for gap in self.drops.gaps(&groups) { - left.insert(gap); + if groups.is_empty() { + return left; + } + left.insert(groups.clone()); + for (run, hole) in self.drops.overlapping(&groups) { + if hole.stale_budget().is_none() { + left.remove(run.clone()); + } } + left + } + + /// Skip what of `groups` is neither cached nor given up for good, replacing a stale drop: + /// a skip is final, and an update would otherwise supersede that drop and leave the + /// sequence owed. See [`Producer::skip_groups`]. + fn skip_groups(&mut self, groups: ops::Range) { + let mut left = self.unsettled(groups.clone()); for (sequence, slot) in self.lookup.range(groups) { if slot.visible && let Some(end) = sequence.checked_add(1) @@ -1185,14 +1200,12 @@ impl TrackState { } /// Settle an append at `sequence`: skip every sequence below it that this track's own - /// publisher never produced and did not give up, from the lowest it produced since its - /// last append, since appending says it will not go back. A reason already given stands. + /// publisher never produced and did not give up for good, from the lowest it produced + /// since its last append, since appending says it will not go back. A final reason + /// already given stands. fn appended(&mut self, sequence: u64) { if let Some(from) = self.appended.or(self.produced.iter().next().map(|run| run.start)) { - let mut left = RangeSet::new(); - for gap in self.drops.gaps(&(from..sequence)) { - left.insert(gap); - } + let mut left = self.unsettled(from..sequence); for run in self.produced.iter() { left.remove(run.clone()); } @@ -3872,7 +3885,7 @@ impl Producer { /// groups says nothing, since groups may be created out of order, so only the producer /// knows a sequence will never exist. A skip is final: creating a group at a skipped /// sequence is refused with [`Error::Skipped`]. A sequence already produced or given up - /// keeps what it is. + /// for good keeps what it is; a stale drop, which new terms may supersede, is replaced. pub fn skip_groups(&self, groups: ops::Range) { if let Ok(mut state) = self.modify() { state.skip_groups(groups); @@ -23701,14 +23714,17 @@ mod test { assert_eq!(read.last(), Some(&1)); } - /// `skip_groups` leaves a cached group and a reason already given alone. + /// `skip_groups` leaves a cached group and a final reason already given alone, and + /// replaces a stale drop, which new terms would otherwise supersede: the skip is final. #[test] #[cfg(feature = "async")] fn skipping_leaves_produced_and_given_up_sequences() { let producer = track_producer("test", Info::timed()); write_group(&producer, 1); producer.give_up(3..4, Hole::drop(StreamError::Cancel)); + producer.give_up(4..5, Hole::stale(Duration::from_secs(1))); producer.skip_groups(0..5); + producer.supersede_stale(None); let drops: Vec<_> = producer .state .read() From 464db629ef0f765dc02a7fbc5caccb4f04fc807a Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 20:14:38 -0800 Subject: [PATCH 25/68] style(net): fmt and a self-referencing test type Co-Authored-By: Claude Opus 5.5 --- js/net/src/lite/subscriber.test.ts | 5 +++-- js/net/src/track.ts | 3 ++- rs/moq-net/src/lite/subscriber.rs | 17 +++++++++-------- rs/moq-net/src/model/resume.rs | 6 +++++- 4 files changed, 19 insertions(+), 12 deletions(-) diff --git a/js/net/src/lite/subscriber.test.ts b/js/net/src/lite/subscriber.test.ts index 838aed0055a..50af3558cd3 100644 --- a/js/net/src/lite/subscriber.test.ts +++ b/js/net/src/lite/subscriber.test.ts @@ -2572,9 +2572,10 @@ test("a skip or final drop replaces an earlier stale one on lite-07", async () = const version = Version.DRAFT_07; const { quic, streams, opened } = fakeSession(); const subscriber = new Subscriber(quic, version, HopSchema.parse(1n)); - let entry: { track: Producer; tail: { covers(start: bigint, end: bigint): boolean } } | undefined; + type Entry = { track: Producer; tail: { covers(start: bigint, end: bigint): boolean } }; + let entry: Entry | undefined; const internal = subscriber as unknown as { - runResponses: (stream: unknown, entry: NonNullable) => Promise; + runResponses: (stream: unknown, entry: Entry) => Promise; }; const runResponses = internal.runResponses.bind(internal); const responses = spyOn(internal, "runResponses").mockImplementation((stream, captured) => { diff --git a/js/net/src/track.ts b/js/net/src/track.ts index c79d2b0671c..ea745dd9359 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -2753,7 +2753,8 @@ export class Subscriber { priority: () => this.#state.update.peek()?.priority ?? 0, maxDelay: () => this.#drift().budget, order: () => this.#state.update.peek()?.order ?? "descending", - refuse: (start, end, error) => this.#state.drops.update((drops) => insertDrop(drops, start, end, { error })), + refuse: (start, end, error) => + this.#state.drops.update((drops) => insertDrop(drops, start, end, { error })), }; return this.#state.asker; } diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index 1c51c0ded2e..855fcb95b0b 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -4304,23 +4304,24 @@ mod tests { serving.apply_response(&serve, old(1, 2)).unwrap(); serving.apply_response(&serve, old(4, 4)).unwrap(); serving - .apply_response(&serve, lite::SubscribeResponse::Skip(lite::SubscribeSkip { start: 1, end: 2 })) + .apply_response( + &serve, + lite::SubscribeResponse::Skip(lite::SubscribeSkip { start: 1, end: 2 }), + ) .unwrap(); serving .apply_response( &serve, - lite::SubscribeResponse::Drop(lite::SubscribeDrop::new( - 4..=4, - &crate::StreamError::App(5), - 0, - version, - )), + lite::SubscribeResponse::Drop(lite::SubscribeDrop::new(4..=4, &crate::StreamError::App(5), 0, version)), ) .unwrap(); // Empty ranges. serving.apply_response(&serve, old(6, 5)).unwrap(); serving - .apply_response(&serve, lite::SubscribeResponse::Skip(lite::SubscribeSkip { start: 6, end: 5 })) + .apply_response( + &serve, + lite::SubscribeResponse::Skip(lite::SubscribeSkip { start: 6, end: 5 }), + ) .unwrap(); { diff --git a/rs/moq-net/src/model/resume.rs b/rs/moq-net/src/model/resume.rs index 10725ba9d0f..3ed609f233b 100644 --- a/rs/moq-net/src/model/resume.rs +++ b/rs/moq-net/src/model/resume.rs @@ -3866,7 +3866,11 @@ mod test { let b = copy(); routes.serve(b.consume()); assert_eq!(dropped(&mut sub), []); - assert_eq!(sub.reoffer_stale(), vec![1..3], "reopened, though the new route never reported it"); + assert_eq!( + sub.reoffer_stale(), + vec![1..3], + "reopened, though the new route never reported it" + ); assert_eq!(sub.reoffer_stale(), Vec::>::new()); write(&b, 1); write(&b, 2); From 14fda0480becfd1ae65cebc8d4a8a384f0426780 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 20:38:19 -0800 Subject: [PATCH 26/68] fix(js): ask fills for stale runs a reader doesn't settle; skips reach the end `#askFills` excluded every drop from what it asks a fill for, so a received stale run that does not settle the reader (judged under other terms) left it idle. It now judges each drop with `settles`, from where it meets the asked run, as the Rust `Misses::unclaimed` does. A declared skip never advanced the live edge, so a track whose last sequences were skipped never reached its end. The producer's settlement, `#endLargest` and a reader's `#complete` now count sequences given up below the end as reached, mirroring the Rust `TrackState::reaches`. Co-Authored-By: Claude Opus 5.5 --- js/net/src/fill.test.ts | 23 ++++++++++++++++++++++- js/net/src/track.test.ts | 15 +++++++++++++++ js/net/src/track.ts | 35 ++++++++++++++++++++++++++++++----- 3 files changed, 67 insertions(+), 6 deletions(-) diff --git a/js/net/src/fill.test.ts b/js/net/src/fill.test.ts index c05f01a86a2..54e4d3863e4 100644 --- a/js/net/src/fill.test.ts +++ b/js/net/src/fill.test.ts @@ -4,7 +4,7 @@ import { Producer as BroadcastProducer } from "./broadcast.ts"; import { accept as acceptSession, connect as connectSession } from "./connection/index.ts"; import { NotFound, StreamCode, StreamError, toStreamCode } from "./error.ts"; import type { Consumer as GroupConsumer } from "./group.ts"; -import { hooks } from "./internal.ts"; +import { hooks, staleHole } from "./internal.ts"; import * as Lite from "./lite/index.ts"; import { createMockTransportPair } from "./mock.ts"; import { Producer as OriginProducer } from "./origin.ts"; @@ -710,6 +710,27 @@ test("fill: a group the track gives up elsewhere answers the fill", async () => broadcast.close(); }); +// A received stale run that does not settle a reader is unknown to it, so its fill is still asked: +// here the upstream judged it under an open range, while the reader is capped. Mirrors the Rust +// `Misses::unclaimed`. +test("fill: a stale run that does not settle the reader is still asked for", async () => { + const { broadcast, track } = resumed(2n); + const requests = track.requests(); + hooks.dropTrackGroups(track, 0n, 2n, staleHole(60_000)); + const reader = track.subscribe({ + ranges: [{ start: { absolute: 0n }, end: { absolute: 2n } }], + delay: { max: Milli(500) }, + }); + void drain(reader); + const fill = await Promise.race([requests.requestedFill(), yieldNow().then(() => undefined)]); + expect(fill && [fill.start, fill.end]).toEqual([0n, 2n]); + + fill?.close(); + reader.close(); + requests.close(); + broadcast.close(); +}); + test("fill: a cache-miss fetch of a dynamic track serves the one track the broadcast already has", async () => { const broadcast = new BroadcastProducer(); // Pulled first, so the broadcast serves tracks on demand. diff --git a/js/net/src/track.test.ts b/js/net/src/track.test.ts index f85bd7de140..fd8ce7db641 100644 --- a/js/net/src/track.test.ts +++ b/js/net/src/track.test.ts @@ -4913,6 +4913,21 @@ test("a late local group reaches a reader, since creating out of order skips not expect(got).toEqual([0n, 1n, 2n]); }); +// Declared skips up to the end settle it, as written groups would: an open-range reader ends after +// the groups below, and the track's outcome resolves. Mirrors the Rust `TrackState::reaches`. +test("declared skips up to the end settle the track", async () => { + const producer = new TrackProducer("test"); + producer.accept({}); + const reader = producer.subscribe({ delay: { max: null }, order: "ascending" }); + producer.createGroup(0n).finish(); + producer.finishAt({ group: 3n, frame: 0 }); + producer.skipGroups(1n, 3n); + expect((await reader.recvGroup())?.sequence).toBe(0n); + expect(await reader.recvGroup()).toBeUndefined(); + expect(await producer.closed).toBeNull(); + reader.close(); +}); + // A skip is final: creating at a sequence an append or skipGroups skipped throws, while a dropped // sequence may be created again. skipGroups leaves cached sequences and final drops alone, and // replaces a stale drop, which new terms would otherwise supersede. Mirrors Rust. diff --git a/js/net/src/track.ts b/js/net/src/track.ts index ea745dd9359..c8486c8e0e1 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -790,6 +790,14 @@ type Coverage = { start: bigint; end?: bigint } | undefined; * where the upstream's range ended: its own successor and edge are then no older than those the * verdict was judged against. Mirrors the Rust `settles`. */ +/** + * Whether the live edge `received` reached `final`: no sequence below it is left that was neither + * written nor given up, as a declared skip or a drop. Mirrors the Rust `TrackState::reaches`. + */ +function reaches(received: bigint, drops: readonly Run[], final: bigint): boolean { + return received >= final || runGaps(drops, received, final).length === 0; +} + function settles(run: DropRun, budget: number, coverage: Coverage, judgedEnd: bigint | undefined): boolean { if (run.stale === undefined) return true; if (budget > run.stale || coverage === undefined || coverage.start > run.start) return false; @@ -2029,7 +2037,7 @@ export class Producer { this.#settle(); if (this.#state.largestEnded.peek() || this.#largestOpen) return; const end = this.#state.end.peek(); - if (end === undefined || this.#received < finalSequence(end)) return; + if (end === undefined || !reaches(this.#received, this.#state.drops.peek(), finalSequence(end))) return; for (const state of [this.#state, ...this.#sinks]) state.largestEnded.set(true); } @@ -2109,7 +2117,7 @@ export class Producer { const end = this.#state.end.peek(); if (end === undefined) return false; const final = finalSequence(end); - if (this.#received < final || this.#owes(final)) return false; + if (!reaches(this.#received, this.#state.drops.peek(), final) || this.#owes(final)) return false; return this.#cache.every(({ group }) => group.sequence >= final || group.closed.peek() === null); } @@ -2777,9 +2785,22 @@ export class Subscriber { // A datagram the track still holds took its sequence, so no group is missed there, as // the Rust judge skips its track's datagram buffer; the Rust fetch names it // `NotFetchable`. - const gone = [...this.#state.drops.peek()]; - for (const sequence of heldDatagrams(producer)) insertRun(gone, sequence, sequence + 1n); + const datagrams: Run[] = []; + for (const sequence of heldDatagrams(producer)) insertRun(datagrams, sequence, sequence + 1n); + // A received stale run that does not settle this reader is unknown to it, so it is + // still asked for, as the Rust `Misses::unclaimed` judges it. + const budget = drift.budget; + const coverage = this.#coverage(); + const drops = this.#state.drops.peek(); for (const run of this.#wants(start, min(end, this.#state.received.peek()))) { + const gone = [...datagrams]; + for (const dropped of drops) { + if (dropped.end <= run.start || run.end <= dropped.start) continue; + // Judged from where it meets the asked run, as the reader would read it. + const clipped = { ...dropped, start: max(dropped.start, run.start) }; + if (settles(clipped, budget, coverage, this.#state.judgedEnd)) + insertRun(gone, clipped.start, dropped.end); + } for (const open of runGaps(gone, run.start, run.end)) { let next = open.start; for (let i = timelineIndex(timeline, open.start); i < timeline.length; i++) { @@ -3584,7 +3605,11 @@ export class Subscriber { // accounted for by the wire layer, and an abort still wins on the next read. #complete(): boolean { const end = this.#state.end.peek(); - return end !== undefined && this.#state.received.peek() >= finalSequence(end) && !this.#state.owed.peek(); + return ( + end !== undefined && + reaches(this.#state.received.peek(), this.#state.drops.peek(), finalSequence(end)) && + !this.#state.owed.peek() + ); } // Package-internal readiness half of recvGroup. Each registration fires at most once, and From d8d24d54d082ba8ae9b801a4fc0da4771388cf28 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 20:44:29 -0800 Subject: [PATCH 27/68] fix(js): settle the end when a drop or skip reaches it Recording a drop or a skip never re-evaluated the track's end, so skipping the last sequences below a declared end left `closed` unresolved and the largest stream open until some later write. `#drop` now settles the end as a write would. Co-Authored-By: Claude Opus 5.5 --- js/net/src/track.test.ts | 5 ++++- js/net/src/track.ts | 2 ++ 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/js/net/src/track.test.ts b/js/net/src/track.test.ts index fd8ce7db641..4dc4ed80a90 100644 --- a/js/net/src/track.test.ts +++ b/js/net/src/track.test.ts @@ -4920,11 +4920,14 @@ test("declared skips up to the end settle the track", async () => { producer.accept({}); const reader = producer.subscribe({ delay: { max: null }, order: "ascending" }); producer.createGroup(0n).finish(); + // Group 0's finish settles first, so only the skip can reach the end. + await yieldNow(); producer.finishAt({ group: 3n, frame: 0 }); + expect(producer.closed.peek()).toBeUndefined(); producer.skipGroups(1n, 3n); + expect(producer.closed.peek()).toBeNull(); expect((await reader.recvGroup())?.sequence).toBe(0n); expect(await reader.recvGroup()).toBeUndefined(); - expect(await producer.closed).toBeNull(); reader.close(); }); diff --git a/js/net/src/track.ts b/js/net/src/track.ts index c8486c8e0e1..ce9de9d74d6 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -1138,6 +1138,8 @@ export class Producer { // A pending tail owes nothing the publisher gave up. A stale verdict is not receipt: new // terms may supersede it, leaving those groups unknown again, as in Rust. if (hole.stale === undefined) this.#arrive(start, end); + // Giving up the last sequences below the end reaches it, as writing them would. + this.#endLargest(); } // A publisher may re-create a group it cut; the sequence is no longer given up. From f6a22818daee9f15a9dfd1633301c0c5909d529a Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 21:37:05 -0800 Subject: [PATCH 28/68] feat(net): a SKIP replaces every drop, final or stale (Rust) Maintainer decision: never produced is the most authoritative verdict, so a received SKIP, skip_groups, or append_group's implicit skip overwrites an earlier DROP of any code. A later drop leaves a skip in place, a cached group is never overwritten, and a relay forwards the skip over a drop it already forwarded. Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/src/lite/publisher.rs | 70 ++++++++- rs/moq-net/src/lite/subscriber.rs | 89 +++++++++-- rs/moq-net/src/model/resume.rs | 35 ++++- rs/moq-net/src/model/track.rs | 248 ++++++++++++++++++++---------- 4 files changed, 344 insertions(+), 98 deletions(-) diff --git a/rs/moq-net/src/lite/publisher.rs b/rs/moq-net/src/lite/publisher.rs index 744c3b0f9b5..a948138defa 100644 --- a/rs/moq-net/src/lite/publisher.rs +++ b/rs/moq-net/src/lite/publisher.rs @@ -4441,6 +4441,9 @@ struct TrackRun { // cannot tell when the subscriber stopped waiting for it. // `None` on other versions. served: Option>, + // Lite-07: the part of `served` named in SUBSCRIBE_DROP, never served. A skip replaces a + // drop, so it is sent over one of these, but never over a group served. + dropped: RangeSet, /// The one group waiting for stream credit. The rest of the backlog waits in the cache, /// which bills, ages, and evicts it, and the cursor's budget skips whatever went stale /// meanwhile: a machine per parked group would hold state for as long as the peer @@ -4494,6 +4497,7 @@ impl TrackRun { reoffered: Default::default(), end_sent: false, served, + dropped: RangeSet::new(), opening: None, children: kio::Tasks::new(), idle: None, @@ -4939,6 +4943,7 @@ impl TrackRun { fn owe(&mut self, reopened: Vec>) { if let Some(served) = &mut self.served { for run in reopened { + self.dropped.remove(run.clone()); served.remove(run); } } @@ -4968,9 +4973,27 @@ impl TrackRun { if range.is_empty() { return Ok(false); } - let gaps: Vec<_> = served.gaps(&range).collect(); + let mut gaps: RangeSet = served.gaps(&range).collect(); + if self.ctx.version.has_skip() { + match hole { + // A skip replaces a drop already sent, so the subscriber learns the groups + // were never produced; a reader the drop ended stays ended. + Hole::Skip => { + let replaced: Vec<_> = self + .dropped + .overlapping(&range) + .map(|run| run.start.max(range.start)..run.end.min(range.end)) + .collect(); + for run in replaced { + self.dropped.remove(run.clone()); + gaps.insert(run); + } + } + Hole::Drop { .. } => self.dropped.extend(gaps.iter().cloned()), + } + } served.extend(gaps.iter().cloned()); - for gap in &gaps { + for gap in gaps.iter() { let msg = self.hole(gap.start..=gap.end - 1, hole); writer.buffer(&msg)?; } @@ -6906,6 +6929,49 @@ mod serve_group_test { assert_eq!(*log.writes.lock().unwrap(), [0, 4, 1, 0, 1, 1, 3, 2, 1, 1, 1, 2, 3, 0]); } + /// A relay forwards a SKIP over a DROP it already forwarded for the same groups, since the + /// skip replaces it; lite-06, which names a skip as a drop, sends nothing more. + #[kio::test] + async fn a_skip_goes_out_over_a_forwarded_drop() { + for version in [Version::Lite07, Version::Lite06] { + let mut track = track::Producer::new( + Arc::new(broadcast::Info::default()), + "test", + crate::track::Info::timed(), + ); + track.received(); + let subscriber = track.subscribe(None); + let (mut run, mut writer, log) = lite_run(version, SinkSession::new(Log::default()), subscriber); + write_group(&mut track, 0, 0); + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + assert!(run.start.is_some(), "{version}: started"); + track.give_up(1..3, Hole::drop(StreamError::Cancel)); + write_group(&mut track, 3, 3); + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + track.skip_groups(0..4); + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + kio::wait(|waiter| writer.poll_flush(waiter)).await.unwrap(); + + let writes = log.writes.lock().unwrap().clone(); + let mut buf = writes.as_slice(); + let mut holes = Vec::new(); + while !buf.is_empty() { + let (msg, size) = lite::SubscribeResponse::decode_slice(buf, version).unwrap(); + buf = &buf[size..]; + match msg { + lite::SubscribeResponse::Skip(skip) => holes.push(("skip", skip.start, skip.end)), + lite::SubscribeResponse::Drop(drop) => holes.push(("drop", drop.start, drop.end)), + _ => {} + } + } + let expected = match version.has_skip() { + true => vec![("drop", 1, 2), ("skip", 1, 2)], + false => vec![("drop", 1, 2)], + }; + assert_eq!(holes, expected, "{version}"); + } + } + /// A huge declared skip goes out as one SUBSCRIBE_SKIP, at a cost independent of its /// length, and the end settles over it. #[kio::test] diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index 900ae615ffa..7d6a8f18ef1 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -4761,6 +4761,63 @@ mod tests { ); } + /// A SKIP replaces an earlier final DROP of the same groups too: "never produced" is the + /// most authoritative verdict. A later DROP leaves the skip in place, and a SKIP naming a + /// group that arrived keeps the group, without ending the session. + #[kio::test] + async fn a_skip_replaces_a_final_drop_but_not_a_cached_group() { + let version = Version::Lite07; + let (serve, mut serving, tail, _broadcast) = lite07_serving(Duration::from_millis(500)).await; + let drop = |start, end, error: &crate::StreamError| { + lite::SubscribeResponse::Drop(lite::SubscribeDrop::new(start..=end, error, 0, version)) + }; + let skip = |start, end| lite::SubscribeResponse::Skip(lite::SubscribeSkip { start, end }); + let mut group = serving + .serving + .create_group(crate::group::Info { sequence: 5 }) + .unwrap(); + group + .write_frame(crate::Timestamp::from_millis(5).unwrap(), &b"x"[..]) + .unwrap(); + group.finish().unwrap(); + + serving + .apply_response(&serve, drop(1, 2, &crate::StreamError::App(5))) + .unwrap(); + serving.apply_response(&serve, skip(1, 2)).unwrap(); + assert!( + matches!( + serving.serving.create_group(crate::group::Info { sequence: 2 }), + Err(Error::Skipped) + ), + "the skip replaced the final drop" + ); + serving + .apply_response(&serve, drop(2, 2, &crate::StreamError::NotFound)) + .unwrap(); + assert!( + matches!( + serving.serving.create_group(crate::group::Info { sequence: 2 }), + Err(Error::Skipped) + ), + "a later drop leaves the skip" + ); + + // Group 5 arrived, so the skip names only 4 and 6. + serving.apply_response(&serve, skip(4, 6)).unwrap(); + assert!(tail.read().covers(4..7)); + let mut reader = serving + .serving + .subscribe(crate::track::Subscription::default().with_ranges([crate::track::Range::groups(0..)])); + let waiter = kio::Waiter::noop(); + assert_eq!(reader.poll_drop(&waiter), Poll::Ready((1..3, track::Hole::Skip))); + assert_eq!(reader.poll_drop(&waiter), Poll::Ready((4..5, track::Hole::Skip))); + assert_eq!(reader.poll_drop(&waiter), Poll::Ready((6..7, track::Hole::Skip))); + assert!(reader.poll_drop(&waiter).is_pending()); + let group = reader.recv_group().await.unwrap().unwrap(); + assert_eq!(group.sequence, 5, "the group that arrived is kept"); + } + /// A SUBSCRIBE_DROP ends a group still arriving now, with its Error Code on lite-07. /// Earlier versions carry an application code there, so the same bytes read as /// OLD. @@ -10203,23 +10260,24 @@ impl ServeLoop { if let Ok(mut tail) = active.tail.write() { tail.account(groups.clone(), serve.subscriber.runtime.now()); } - let hole = match stale { - // Judged under the budget this subscriber sent with that update. + match stale { + // Judged under the budget this subscriber sent with that update. A + // sequence already skipped stays skipped, so no update reopens it. true => { - active.stale.insert(groups.clone()); - track::Hole::stale(active.max_delay) + let recorded = self.serving.give_up(groups, track::Hole::stale(active.max_delay)); + active.stale.extend(recorded); } // Final, so no update reopens it, even where it replaces a stale drop. false => { active.stale.remove(groups.clone()); - track::Hole::drop(error) + self.serving.give_up(groups, track::Hole::drop(error)); } - }; - self.serving.give_up(groups, hole); + } } } - // Never produced, so nothing is waited for and no reader ends. A skip is final: it - // replaces a stale drop of the same groups, which an update would otherwise reopen. + // Never produced, so nothing is waited for and no reader ends. A skip is final and + // replaces any drop of the same groups, stale or final, so no update reopens them; + // a reader a drop already ended stays ended. lite::SubscribeResponse::Skip(skipped) => { let groups = skipped.start..skipped.end.saturating_add(1); // An empty range (Group Start = Group End + 1) names nothing. @@ -10230,7 +10288,18 @@ impl ServeLoop { tail.account(groups.clone(), serve.subscriber.runtime.now()); } active.stale.remove(groups.clone()); - self.serving.skip_groups(groups); + // A group that arrived was produced after all, so the skip contradicts it: it + // keeps the group, and the session goes on, since the draft requires no more. + let cached = self.serving.skip_received(groups); + if let Some(first) = cached.iter().next() { + tracing::warn!( + track = %serve.name, + start = skipped.start, + end = skipped.end, + cached = first.start, + "protocol error: SUBSCRIBE_SKIP names a group that arrived; keeping the group" + ); + } } // OK just resolves the range (the producer already orders // groups). diff --git a/rs/moq-net/src/model/resume.rs b/rs/moq-net/src/model/resume.rs index 87de937031e..a95f19db22c 100644 --- a/rs/moq-net/src/model/resume.rs +++ b/rs/moq-net/src/model/resume.rs @@ -1839,8 +1839,11 @@ impl Reader { // Reported once. As on a plain cursor, the report is not final: a miss (a fill given // up) may still land, and is handed out if it does. self.drops.remove(open.clone()); - if err.stale_budget().is_some() { - self.reported_stale.insert(open.clone()); + match &err { + // Final, so no update reopens a stale report it replaced. + track::Hole::Skip => self.reported_stale.remove(open.clone()), + err if err.stale_budget().is_some() => self.reported_stale.insert(open.clone()), + track::Hole::Drop { .. } => {} } Poll::Ready((open, err)) } @@ -3509,6 +3512,34 @@ mod test { assert_eq!(dropped(&mut sub), []); } + /// A skip of a run the serving copy already dropped replaces the drop, so the resumed + /// reader reports it again, as a skip, and only once. + #[kio::test] + async fn a_resumed_reader_reports_a_skip_over_a_reported_drop() { + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + routes.serve(a.consume()); + let mut sub = subscribe(&logical, HOUR); + write(&a, 0); + assert_eq!(drain(&mut sub, false), [0]); + let holes = |sub: &mut track::Subscriber| { + let mut runs = Vec::new(); + while let Poll::Ready(run) = sub.poll_drop(&kio::Waiter::noop()) { + runs.push(run); + } + runs + }; + + a.drop_groups(1..3, StreamError::NotFound); + assert_eq!(holes(&mut sub), [(1..3, track::Hole::drop(StreamError::NotFound))]); + a.skip_groups(1..3); + assert_eq!(holes(&mut sub), [(1..3, track::Hole::Skip)]); + assert_eq!(holes(&mut sub), []); + write(&a, 3); + assert_eq!(drain(&mut sub, false), [3]); + } + /// A run the serving copy gave up waits while a copy it replaced may still deliver it /// under its bound, and goes out once that copy is past the reader's budget. #[kio::test] diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index 6c961ebdc54..de9330f1768 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -1133,13 +1133,27 @@ impl TrackState { } } - /// Record sequences the upstream publisher gave up, with why. - fn drop_groups(&mut self, groups: ops::Range, hole: Hole) { + /// Record sequences the upstream publisher gave up, with why. A skip is final and the + /// most authoritative verdict, so a drop never replaces one; see [`Self::skip_groups`]. + /// Returns the runs recorded. + fn drop_groups(&mut self, groups: ops::Range, hole: Hole) -> RangeSet { + let mut runs = RangeSet::new(); if groups.is_empty() { - return; + return runs; + } + runs.insert(groups.clone()); + if hole != Hole::Skip { + for (run, given) in self.drops.overlapping(&groups) { + if *given == Hole::Skip { + runs.remove(run.clone()); + } + } + } + self.cache.wakes().arrived(groups); + for run in runs.iter() { + self.change_drops(run.clone(), Some(hole.clone())); } - self.cache.wakes().arrived(groups.clone()); - self.change_drops(groups, Some(hole)); + runs } /// Whether the publisher declared `sequence` skipped: never produced, which is final. @@ -1147,51 +1161,51 @@ impl TrackState { self.drops.get(&sequence).is_some_and(|hole| *hole == Hole::Skip) } - /// What of `groups` no final verdict names: a gap, or a stale drop, which new terms may - /// supersede and so says nothing for good. - fn unsettled(&self, groups: ops::Range) -> RangeSet { - let mut left = RangeSet::new(); + /// Skip what of `groups` holds no cached group, replacing any drop of it, final or stale: + /// "never produced" is the most authoritative verdict on a sequence. A reader a drop + /// already ended stays ended, and a relay forwards the skip over the drop it reported. + /// A group that arrived is never overwritten, since it was produced after all. Returns + /// the cached sequences left alone. See [`Producer::skip_groups`]. + fn skip_groups(&mut self, groups: ops::Range) -> RangeSet { + let mut cached = RangeSet::new(); if groups.is_empty() { - return left; + return cached; } + let mut left = RangeSet::new(); left.insert(groups.clone()); - for (run, hole) in self.drops.overlapping(&groups) { - if hole.stale_budget().is_none() { - left.remove(run.clone()); + for (sequence, _) in self.lookup.range(groups.clone()) { + if let Some(end) = sequence.checked_add(1) { + left.remove(*sequence..end); + cached.insert(*sequence..end); } } - left - } - - /// Skip what of `groups` is neither cached nor given up for good, replacing a stale drop: - /// a skip is final, and an update would otherwise supersede that drop and leave the - /// sequence owed. See [`Producer::skip_groups`]. - fn skip_groups(&mut self, groups: ops::Range) { - let mut left = self.unsettled(groups.clone()); - for (sequence, slot) in self.lookup.range(groups) { - if slot.visible - && let Some(end) = sequence.checked_add(1) - { - left.remove(*sequence..end); + // Already skipped: nothing changes, so no reader rescans. + for (run, hole) in self.drops.overlapping(&groups) { + if *hole == Hole::Skip { + left.remove(run.clone()); } } for run in left.iter().cloned().collect::>() { self.drop_groups(run, Hole::Skip); } + cached } /// Settle an append at `sequence`: skip every sequence below it that this track's own - /// publisher never produced and did not give up for good, from the lowest it produced - /// since its last append, since appending says it will not go back. A final reason - /// already given stands. + /// publisher never produced, from the lowest it produced since its last append, since + /// appending says it will not go back. The skip replaces a drop given there, as + /// [`Self::skip_groups`] does. fn appended(&mut self, sequence: u64) { - if let Some(from) = self.appended.or(self.produced.iter().next().map(|run| run.start)) { - let mut left = self.unsettled(from..sequence); + if let Some(from) = self.appended.or(self.produced.iter().next().map(|run| run.start)) + && from < sequence + { + let mut left = RangeSet::new(); + left.insert(from..sequence); for run in self.produced.iter() { left.remove(run.clone()); } for run in left.iter().cloned().collect::>() { - self.drop_groups(run, Hole::Skip); + self.skip_groups(run); } } self.produced.clear(); @@ -3633,15 +3647,26 @@ impl Producer { /// This and [`Self::append_group`] are the only sources of a skip: a gap between created /// groups says nothing, since groups may be created out of order, so only the producer /// knows a sequence will never exist. A skip is final: creating a group at a skipped - /// sequence is refused with [`Error::Skipped`]. A sequence already produced or given up - /// for good keeps what it is; a stale drop, which new terms may supersede, is replaced. + /// sequence is refused with [`Error::Skipped`], and a later drop of it is ignored. + /// + /// A skip replaces any drop of the same sequences, whatever its error: a reader the drop + /// already ended stays ended, and every later reader and relay sees the skip. A group + /// already cached keeps what it is, since it was produced after all. pub fn skip_groups(&self, groups: ops::Range) { - if let Ok(mut state) = self.modify() { - state.skip_groups(groups); - let deferred = state.deferred(); - drop(state); - deferred.wake(); - } + self.skip_received(groups); + } + + /// [`Self::skip_groups`], returning the cached sequences it left alone: a skip naming a + /// group that arrived contradicts it, which a subscriber reports. + pub(crate) fn skip_received(&self, groups: ops::Range) -> RangeSet { + let Ok(mut state) = self.modify() else { + return RangeSet::new(); + }; + let cached = state.skip_groups(groups); + let deferred = state.deferred(); + drop(state); + deferred.wake(); + cached } /// Forget every run held as stale: the terms a received copy's upstream judged them @@ -3655,41 +3680,46 @@ impl Producer { } } - /// Record `groups` given up as `hole` describes. - pub(crate) fn give_up(&self, groups: ops::Range, hole: Hole) { - if groups.is_empty() { - return; - } + /// Record `groups` given up as `hole` describes, returning the runs recorded: a drop + /// leaves a skipped sequence skipped. + pub(crate) fn give_up(&self, groups: ops::Range, hole: Hole) -> RangeSet { let error = match &hole { - Hole::Drop { error, .. } => Some(error.clone()), - Hole::Skip => None, + Hole::Drop { error, .. } => error.clone(), + Hole::Skip => { + let mut runs = RangeSet::new(); + runs.insert(groups.clone()); + for cached in self.skip_received(groups).iter() { + runs.remove(cached.clone()); + } + return runs; + } }; - let open: Vec = match self.modify() { + if groups.is_empty() { + return RangeSet::new(); + } + let (runs, open): (_, Vec) = match self.modify() { Ok(mut state) => { - state.drop_groups(groups.clone(), hole); + let runs = state.drop_groups(groups.clone(), hole); // Only the live feed's copy: a drop judges the subscription, so a fetched head - // or fetch-only backfill may still complete. A skip names nothing produced. - match error { - None => Vec::new(), - Some(_) => state - .lookup - .range(groups) - .filter(|(_, slot)| slot.visible) - .map(|(_, slot)| &slot.group) - .filter(|group| !group.is_aborted()) - .cloned() - .collect(), - } + // or fetch-only backfill may still complete. + let open = state + .lookup + .range(groups) + .filter(|(_, slot)| slot.visible) + .map(|(_, slot)| &slot.group) + .filter(|group| !group.is_aborted()) + .cloned() + .collect(); + (runs, open) } - Err(_) => return, + Err(_) => return RangeSet::new(), }; // Settle the readers of a group still arriving now, rather than once its stream's // reset lands, with the track guard released. - if let Some(error) = error { - for group in open { - group.abort_open(Error::Stream(error.clone())); - } + for group in open { + group.abort_open(Error::Stream(error.clone())); } + runs } /// Readers get nothing from the cache until [`Self::set_live`]: the upstream @@ -6749,6 +6779,10 @@ struct DropCursor { /// [`Subscriber::reoffer_stale`]. #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] reported_stale: RangeSet, + /// The reported runs skipped. A skip replaces a drop, so a run reported dropped is + /// reported again once it is skipped, and only a skip already reported stops it. + #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] + reported_skip: RangeSet, } #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] @@ -6761,6 +6795,7 @@ impl DropCursor { scanned: None, reported: RangeSet::new(), reported_stale: RangeSet::new(), + reported_skip: RangeSet::new(), } } @@ -6869,14 +6904,25 @@ impl DropCursor { if clipped.end < range.end { self.pending.insert(clipped.end..range.end, pending.clone()); } - if let Some(gap) = self.reported.gaps(&clipped).next() { + // A skip goes out over a drop already reported, since it replaces it. + let reported = match pending.hole { + Hole::Skip => &self.reported_skip, + Hole::Drop { .. } => &self.reported, + }; + if let Some(gap) = reported.gaps(&clipped).next() { if gap.end < range.end { self.pending.insert(gap.end..range.end, pending.clone()); } let hole = pending.hole; self.reported.insert(gap.clone()); - if hole.stale_budget().is_some() { - self.reported_stale.insert(gap.clone()); + match &hole { + // Final, so no update reopens a stale report it replaced. + Hole::Skip => { + self.reported_skip.insert(gap.clone()); + self.reported_stale.remove(gap.clone()); + } + hole if hole.stale_budget().is_some() => self.reported_stale.insert(gap.clone()), + Hole::Drop { .. } => {} } return Some((gap, hole)); } @@ -22202,10 +22248,11 @@ mod test { assert_eq!(skipped, [(3..5, Hole::Skip), (6..7, Hole::Skip), (9..10, Hole::Skip)]); } - /// A sequence already given up keeps its reason when an append skips around it. + /// An append's skip replaces a drop below it, whatever its error: the sequence was never + /// produced, which is the most authoritative verdict. #[test] #[cfg(feature = "async")] - fn appending_keeps_a_reason_already_given() { + fn appending_replaces_a_drop_with_a_skip() { let producer = track_producer("test", Info::timed()); write_group(&producer, 0); write_group(&producer, 4); @@ -22218,7 +22265,7 @@ mod test { .iter() .map(|(run, hole)| (run.clone(), hole.clone())) .collect(); - assert_eq!(drops, [(1..2, Hole::drop(StreamError::Cancel)), (2..4, Hole::Skip)]); + assert_eq!(drops, [(1..4, Hole::Skip)]); } /// A skip is final: creating a group at a sequence an append or `skip_groups` skipped is @@ -22242,9 +22289,15 @@ mod test { )); // The next sequence, 5, was skipped too. assert!(matches!(producer.append_group(), Err(Error::Skipped))); + // A later drop leaves a skipped sequence skipped. + producer.give_up(1..2, Hole::drop(StreamError::Cancel)); + assert!(matches!( + producer.create_group(group::Info { sequence: 1 }), + Err(Error::Skipped) + )); // A dropped sequence may be created again. - producer.give_up(2..3, Hole::drop(StreamError::Cancel)); - write_group(&producer, 2); + producer.give_up(9..10, Hole::drop(StreamError::Cancel)); + write_group(&producer, 9); } /// Skips are kept and reported as runs: a huge `skip_groups` or append gap costs one @@ -22299,11 +22352,11 @@ mod test { assert_eq!(read.last(), Some(&1)); } - /// `skip_groups` leaves a cached group and a final reason already given alone, and - /// replaces a stale drop, which new terms would otherwise supersede: the skip is final. + /// `skip_groups` leaves a cached group alone, and replaces every drop, final or stale: the + /// skip is final, so new terms never supersede it. #[test] #[cfg(feature = "async")] - fn skipping_leaves_produced_and_given_up_sequences() { + fn skipping_replaces_every_drop_but_leaves_a_cached_group() { let producer = track_producer("test", Info::timed()); write_group(&producer, 1); producer.give_up(3..4, Hole::drop(StreamError::Cancel)); @@ -22317,15 +22370,42 @@ mod test { .iter() .map(|(run, hole)| (run.clone(), hole.clone())) .collect(); + assert_eq!(drops, [(0..1, Hole::Skip), (2..5, Hole::Skip)]); + } + + /// A skip after a drop already reported goes out again, replacing it, while a reader that + /// moved past the drop is not handed anything again; one already reported as a skip is + /// not reported twice, and a later drop leaves it skipped. + #[test] + #[cfg(feature = "async")] + fn a_skip_is_reported_over_a_reported_drop() { + let producer = track_producer("test", Info::timed()); + let mut subscriber = producer.subscribe(replay().with_range(0..)); + let waiter = kio::Waiter::noop(); + write_group(&producer, 0); + producer.give_up(1..3, Hole::drop(StreamError::Cancel)); + producer.give_up(4..5, Hole::drop(StreamError::NotFound)); + write_group(&producer, 5); assert_eq!( - drops, - [ - (0..1, Hole::Skip), - (2..3, Hole::Skip), - (3..4, Hole::drop(StreamError::Cancel)), - (4..5, Hole::Skip) - ] + subscriber.poll_drop(&waiter), + Poll::Ready((1..3, Hole::drop(StreamError::Cancel))) ); + assert_eq!( + subscriber.poll_drop(&waiter), + Poll::Ready((4..5, Hole::drop(StreamError::NotFound))) + ); + assert!(subscriber.poll_drop(&waiter).is_pending()); + assert_eq!(drain_groups(&mut subscriber), [0, 5]); + + // Group 0 arrived, so the skip leaves it; both drops and the gap between become one skip. + producer.skip_groups(0..5); + assert_eq!(subscriber.poll_drop(&waiter), Poll::Ready((1..5, Hole::Skip))); + assert!(subscriber.poll_drop(&waiter).is_pending()); + assert!(drain_groups(&mut subscriber).is_empty(), "nothing is delivered again"); + + producer.give_up(2..3, Hole::drop(StreamError::Cancel)); + assert!(subscriber.poll_drop(&waiter).is_pending(), "the skip stands"); + assert!(producer.state.read().skipped(2)); } /// Appending and a first group leave no gap, so they skip nothing; nor is anything below From 463bfc8ceea1026493744e28c439618f74c48748 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 21:42:41 -0800 Subject: [PATCH 29/68] feat(net): a SKIP replaces every drop in JS, the draft, and migration notes Mirror the Rust rule in js/net, document in lite-07 that a later SUBSCRIBE_SKIP overrides an earlier SUBSCRIBE_DROP without re-delivering, record the maintainer decision in the drop-set quest, and add the missing moq-net and @moq/net migration notes. Co-Authored-By: Claude Opus 5.5 --- drafts/draft-lcurley-moq-lite.md | 10 +++- js/net/MIGRATE.md | 6 +++ js/net/src/internal.ts | 13 ++++- js/net/src/lite/publisher.test.ts | 35 ++++++++++++++ js/net/src/lite/publisher.ts | 22 ++++++++- js/net/src/lite/subscriber.test.ts | 77 ++++++++++++++++++++++++++++++ js/net/src/lite/subscriber.ts | 21 +++++--- js/net/src/resume.test.ts | 33 +++++++++++++ js/net/src/resume.ts | 4 +- js/net/src/track.test.ts | 60 ++++++++++++++++++----- js/net/src/track.ts | 71 +++++++++++++++++++-------- quest/m1/drop-set.md | 23 +++++---- rs/moq-net/MIGRATE.md | 13 +++++ 13 files changed, 335 insertions(+), 53 deletions(-) diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index 16a7652f3af..1ac81905c93 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -1650,6 +1650,10 @@ A relay that received a stale drop settles with it only a reader whose own budge A drop with any other Error Code is final, whatever its `Update`, and replaces a stale drop of the same sequences, so no update reopens them. +A later [SUBSCRIBE_SKIP](#subscribe-skip) for the same sequences overrides a drop, whatever its Error Code: the subscriber records them as skipped from then on, and a relay forwards the skip even over a drop it already forwarded. +A subscriber that already ended readers on the drop does not deliver anything again: those readers stay ended. +A drop that follows a skip of the same sequences changes nothing. + A Group Stream for a sequence the subscriber already saw dropped, as after such a re-offer, wins over the drop: the subscriber delivers it. For every sequence in the subscription's range, the publisher MUST send a Group Stream or name it in SUBSCRIBE_SKIP or SUBSCRIBE_DROP. @@ -1678,7 +1682,9 @@ SUBSCRIBE_SKIP Message { A skip names only sequences the original publisher declared it will never produce, and it is final: no Group will ever exist at those sequences, whatever the terms. A publisher MUST NOT later create a Group at a sequence it skipped, and a subscriber MUST NOT deliver a Group Stream for one. It carries no error and ends no reader, since there is nothing to read; a subscriber stops waiting for those sequences. -A skip replaces a stale (OLD) drop of the same sequences, so no update reopens them; a sequence already named in a final drop keeps it. +A skip overrides an earlier SUBSCRIBE_DROP of the same sequences, whatever its Error Code, since never produced is the most authoritative account of a sequence: no update reopens them, and a relay forwards the skip downstream even over a drop it already forwarded. +A subscriber that already ended readers on that drop does not deliver anything again: those readers stay ended. +A skip never overrides a Group that arrived, since that Group was produced; a subscriber ignores a skip for such a sequence, which is a protocol error the publisher made, and MAY log it. A gap in the sequence numbers says nothing on its own: a publisher may create Groups out of order, or create again a sequence whose Group it aborted, so it never skips a sequence merely because a higher Group exists or the Track ended without it. Only the application producing the Track knows a sequence will never exist, for example when it numbers Groups sparsely, or when it moves on to the next Group and so declares that it will not go back to fill the sequences below. @@ -2009,7 +2015,7 @@ The `Message Length` describes the payload size on the wire. - Restored SUBSCRIBE_DROP (type 0x2), replacing Stream Count on SUBSCRIBE_FINISH. Every sequence is accounted for by a Group Stream or a drop; a publisher SHOULD send a drop as soon as it gives a sequence up. - SUBSCRIBE_DROP's Error Code is a stream error code, which ends the dropped groups' readers, including one still arriving. - Refuse a SUBSCRIBE_DROP whose Group Start is greater than Group End plus one on every version that carries it. -- Added SUBSCRIBE_SKIP (type 0x3) for sequences the publisher declared it will never produce, which ends no reader and is final: a Group is never created at a skipped sequence. A gap in the sequence numbers is not a skip, since Groups may be created out of order; a sequence never had and never declared skipped is named in SUBSCRIBE_DROP. SUBSCRIBE_DROP no longer names declared skips. +- Added SUBSCRIBE_SKIP (type 0x3) for sequences the publisher declared it will never produce, which ends no reader and is final: a Group is never created at a skipped sequence. A skip overrides an earlier drop of the same sequences, whatever its Error Code, without re-delivering to readers the drop already ended, and never overrides a Group that arrived. A gap in the sequence numbers is not a skip, since Groups may be created out of order; a sequence never had and never declared skipped is named in SUBSCRIBE_DROP. SUBSCRIBE_DROP no longer names declared skips. - Added `Update` to SUBSCRIBE, SUBSCRIBE_UPDATE, and SUBSCRIBE_DROP: the number of the terms in effect. A stale (OLD) drop holds only under the update it names, judged against the budget the subscriber sent with it; every later SUBSCRIBE_UPDATE reopens it, and the publisher re-offers the group or drops it again under the new `Update`. Any other drop is final. A Group Stream wins over its earlier drop. - SUBSCRIBE_FINISH declares the final boundary while lower groups can still be in flight, and no longer closes the Subscribe Stream: the publisher keeps reading SUBSCRIBE_UPDATE, so an update can reach an ended Track's history. - A covered subscription, whose every Range has an End and whose every Group was sent or dropped, also stays open for an update; the publisher never FINs first. diff --git a/js/net/MIGRATE.md b/js/net/MIGRATE.md index a2b0e96e2be..27109c5bcd1 100644 --- a/js/net/MIGRATE.md +++ b/js/net/MIGRATE.md @@ -4,6 +4,12 @@ How to update your code for each breaking change, newest first. ## Unreleased +### `Track.Producer.skipGroups` declares groups never produced ([#7670](https://github.com/moq-dev/moq/pull/7670)) + +`skipGroups(start, end)` says the track will never hold `[start, end)`: readers stop waiting for them, no reader ends, and a relay forwards a lite-07 SUBSCRIBE_SKIP. +A skip is final: `createGroup` and `appendGroup` at a skipped sequence throw `Skipped`, and a later drop leaves it skipped. It replaces an earlier drop of the same groups, whatever its error, but never a cached group. +`appendGroup` now skips the holes below it that earlier `createGroup` calls left since the last append; create every sequence you still mean to fill before appending. + ### A timestamp out of range at the track's timescale is a `TimestampMismatch` ([#7702](https://github.com/moq-dev/moq/pull/7702)) A frame or datagram whose timestamp leaves the safe range once converted to the track's timescale is refused at the write with `TimestampMismatch`, as Rust refuses it, rather than with a `RangeError` when it is encoded. diff --git a/js/net/src/internal.ts b/js/net/src/internal.ts index f9eb199732a..2f01b88577a 100644 --- a/js/net/src/internal.ts +++ b/js/net/src/internal.ts @@ -501,9 +501,15 @@ export const hooks: { * Record that the publisher gave up the groups `[start, end)` with `error`: none will * arrive, so a reader waiting on one moves past it, a group still arriving in them ends with * `error`, the declared end does not wait for them, and a relay serving this track forwards - * the drop. A skip ends no reader. Mirrors the Rust `give_up`; assigned by `track.ts`. + * the drop. A skip ends no reader, and replaces any drop; a drop leaves a skip in place. + * Returns the runs recorded. Mirrors the Rust `give_up`; assigned by `track.ts`. */ - dropTrackGroups: (producer: Producer, start: bigint, end: bigint, hole: Hole) => void; + dropTrackGroups: (producer: Producer, start: bigint, end: bigint, hole: Hole) => Run[]; + /** + * `Producer.skipGroups`, returning the cached sequences it left alone, which a skip naming a + * group that arrived contradicts. Mirrors the Rust `skip_received`; assigned by `track.ts`. + */ + skipTrackGroups: (producer: Producer, start: bigint, end: bigint) => Run[]; /** * Forget every run a received copy holds as stale: the terms its upstream judged them under * were replaced, so they are unknown again. The new terms' ranges end at `judgedEnd`, @@ -796,6 +802,9 @@ export const hooks: { dropTrackGroups: () => { throw new Error("track.ts not loaded"); }, + skipTrackGroups: () => { + throw new Error("track.ts not loaded"); + }, producerLargest: () => { throw new Error("track.ts not loaded"); }, diff --git a/js/net/src/lite/publisher.test.ts b/js/net/src/lite/publisher.test.ts index 2a04baf8751..a063a5d889e 100644 --- a/js/net/src/lite/publisher.test.ts +++ b/js/net/src/lite/publisher.test.ts @@ -1031,6 +1031,41 @@ for (const version of [Version.DRAFT_06, Version.DRAFT_07]) { } }); + // A skip replaces a drop, so a relay forwards it over a drop it already forwarded; lite-06, + // which names a skip as a drop, sends nothing more. Mirrors Rust. + test(`lite ${version}: a skip goes out over a forwarded drop`, async () => { + const sub = await servedSubscription({ version, startGroup: 0n, routed: true }); + try { + sub.serve(0n); + expect(await sub.servedSequence()).toBe(0n); + expect("start" in (await decodeSubscribeResponse(sub.client.reader, version))).toBe(true); + + hooks.dropTrackGroups(sub.track, 1n, 3n, { error: new StreamError(StreamCode(70), { message: "cut" }) }); + const resp = await decodeSubscribeResponse(sub.client.reader, version); + if (!("drop" in resp)) throw new Error("expected SUBSCRIBE_DROP"); + expect([resp.drop.start, resp.drop.end]).toEqual([1n, 2n]); + + sub.track.skipGroups(1n, 3n); + if (version === Version.DRAFT_07) { + const skip = await decodeSubscribeResponse(sub.client.reader, version); + expect(skip).toMatchObject({ skip: { start: 1n, end: 2n } }); + } + sub.track.finish(); + sub.track.close(); + const holes: unknown[] = []; + for (;;) { + const next = await decodeSubscribeResponseMaybe(sub.client.reader, version); + if (!next) break; + if ("drop" in next) holes.push(["drop", next.drop.start, next.drop.end]); + if ("skip" in next) holes.push(["skip", next.skip.start, next.skip.end]); + if ("end" in next && keepsSubscribeOpen(version)) break; + } + expect(holes).toEqual([]); + } finally { + await sub.close(); + } + }); + for (const declared of [false, true]) { const named = declared && version === Version.DRAFT_07 ? "in SUBSCRIBE_SKIP" : "in SUBSCRIBE_DROP"; test(`lite ${version}: the end sweep names a ${declared ? "declared skip" : "gap never declared"} ${named}`, async () => { diff --git a/js/net/src/lite/publisher.ts b/js/net/src/lite/publisher.ts index 75b433f53ed..b819b203973 100644 --- a/js/net/src/lite/publisher.ts +++ b/js/net/src/lite/publisher.ts @@ -1284,11 +1284,30 @@ export class Publisher { ? { skip: new SubscribeSkip({ start, end }) } : { drop: SubscribeDrop.given(start, end, hole.error, this.version, updateNumber) }; + // Lite-07: the part of `served` named in SUBSCRIBE_DROP, never served. A skip replaces a + // drop, so it is sent over one of these, but never over a group served. + let dropped: Run[] = []; + // Give up, as `hole` says, whatever part of `[start, end)` this subscription has not been // served or given up yet. const drop = async (start: bigint, end: bigint, hole: Hole): Promise => { if (!served) return true; - for (const gap of runGaps(served, start, end)) { + const gaps = runGaps(served, start, end); + if (hasSkip(this.version)) { + if (hole.skip) { + // A skip replaces a drop already sent, so the subscriber learns the groups were + // never produced; a reader the drop ended stays ended. + for (const run of dropped) { + const from = max(run.start, start); + const to = min(run.end, end); + if (from < to) insertRun(gaps, from, to); + } + dropped = removeRun(dropped, start, end); + } else { + for (const gap of gaps) insertRun(dropped, gap.start, gap.end); + } + } + for (const gap of gaps) { insertRun(served, gap.start, gap.end); const msg = holeMessage(gap.start, gap.end - 1n, hole); if (!(await controls.response(encodeSubscribeResponse(stream, msg, this.version)))) return false; @@ -1436,6 +1455,7 @@ export class Publisher { if (hasSkip(this.version)) { const owe = (start: bigint, end: bigint) => { if (served) served.splice(0, served.length, ...removeRun(served, start, end)); + dropped = removeRun(dropped, start, end); }; for (const run of hooks.reofferStale(track)) owe(run.start, run.end); // A stale drop not sent yet, such as one an earlier update left to drop diff --git a/js/net/src/lite/subscriber.test.ts b/js/net/src/lite/subscriber.test.ts index 50af3558cd3..12f51e3a865 100644 --- a/js/net/src/lite/subscriber.test.ts +++ b/js/net/src/lite/subscriber.test.ts @@ -2627,6 +2627,83 @@ test("a skip or final drop replaces an earlier stale one on lite-07", async () = } }); +// A SKIP replaces an earlier final DROP of the same groups too: "never produced" is the most +// authoritative verdict. A later DROP leaves the skip in place, and a SKIP naming a group that +// arrived keeps the group, logging a protocol error without ending the session. Mirrors Rust. +test("a skip replaces a final drop but not a cached group on lite-07", async () => { + const version = Version.DRAFT_07; + const { quic, streams, opened } = fakeSession(); + const subscriber = new Subscriber(quic, version, HopSchema.parse(1n)); + type Entry = { track: Producer; tail: { covers(start: bigint, end: bigint): boolean } }; + let entry: Entry | undefined; + const internal = subscriber as unknown as { + runResponses: (stream: unknown, entry: Entry) => Promise; + }; + const runResponses = internal.runResponses.bind(internal); + const responses = spyOn(internal, "runResponses").mockImplementation((stream, captured) => { + entry = captured; + return runResponses(stream, captured); + }); + const warn = spyOn(console, "warn").mockImplementation(() => {}); + const broadcast = subscriber.consume(Path.from("room")); + const track = broadcast.track("video").subscribe({ delay: { max: Milli(500) } }); + try { + await opened(0); + await answerTrackInfo(streams[0], version); + const stream = await opened(1); + await stream.reading; + const send = async (resps: SubscribeResponse[]) => { + const chunks: Uint8Array[] = []; + const writer = new Writer( + new WritableStream({ write: (chunk) => void chunks.push(new Uint8Array(chunk)) }), + version, + ); + for (const resp of resps) await encodeSubscribeResponse(writer, resp, version); + for (const chunk of chunks) stream.inbound.enqueue(chunk); + await drainReactions(); + }; + await send([ + { start: new SubscribeStart([0n]) }, + { drop: new SubscribeDrop({ start: 1n, end: 2n, error: StreamCode.NotFound }) }, + ]); + if (!entry) throw new Error("no subscription"); + entry.track.createGroup(5n).finish(); + + await send([ + { skip: new SubscribeSkip({ start: 1n, end: 2n }) }, + // A later drop leaves the skip in place. + { drop: new SubscribeDrop({ start: 2n, end: 2n, error: StreamCode.NotFound }) }, + // Group 5 arrived, so the skip names only 4 and 6. + { skip: new SubscribeSkip({ start: 4n, end: 6n }) }, + ]); + expect(() => entry?.track.createGroup(1n)).toThrow(Skipped); + expect(() => entry?.track.createGroup(2n)).toThrow(Skipped); + expect(entry.tail.covers(4n, 7n)).toBe(true); + expect(warn).toHaveBeenCalledTimes(1); + + const reader = entry.track.subscribe({ delay: { max: null } }); + const got: unknown[] = []; + for (let next = hooks.tryRecvGroup(reader, true); next.kind !== "idle"; next = hooks.tryRecvGroup(reader, true)) { + if (next.kind === "group") got.push(next.group.sequence); + else if (next.kind === "drop") got.push([next.start, next.end, next.skip === true]); + else break; + } + expect(got).toEqual([ + [1n, 3n, true], + [4n, 5n, true], + [6n, 7n, true], + 5n, + ]); + reader.close(); + } finally { + warn.mockRestore(); + responses.mockRestore(); + track.close(); + broadcast.close(); + subscriber.close(); + } +}); + // A widening reopens a covered copy, and its readers leaving while the publisher never takes the // new SUBSCRIBE gives the setup up: its stream is reset and the copy released, not held forever. test("a reopened lite-07 subscription ends once its readers leave", async () => { diff --git a/js/net/src/lite/subscriber.ts b/js/net/src/lite/subscriber.ts index 147570541e8..48652cc1694 100644 --- a/js/net/src/lite/subscriber.ts +++ b/js/net/src/lite/subscriber.ts @@ -1882,18 +1882,27 @@ export class Subscriber { entry.tail.account(start, end + 1n); // Judged under the budget this subscriber sent with that update. const hole = stale ? staleHole(entry.maxDelay ?? 0) : { error }; - // A final drop is never reopened, even where it replaces a stale one. + const recorded = hooks.dropTrackGroups(entry.track, start, end + 1n, hole); + // A final drop is never reopened, even where it replaces a stale one. A sequence + // already skipped stays skipped, so no update reopens it either. entry.stale = stale - ? insertDrop(entry.stale ?? [], start, end + 1n, hole) + ? recorded.reduce((runs, run) => insertDrop(runs, run.start, run.end, hole), entry.stale ?? []) : removeRun(entry.stale ?? [], start, end + 1n); - hooks.dropTrackGroups(entry.track, start, end + 1n, hole); } else if ("skip" in resp) { - // Never produced, so nothing is waited for and no reader ends. A skip is final: it - // replaces a stale drop of the same groups, which an update would otherwise reopen. + // Never produced, so nothing is waited for and no reader ends. A skip is final and + // replaces any drop of the same groups, stale or final, so no update reopens them; a + // reader a drop already ended stays ended. const { start, end } = resp.skip; entry.tail.account(start, end + 1n); entry.stale = removeRun(entry.stale ?? [], start, end + 1n); - entry.track.skipGroups(start, end + 1n); + // A group that arrived was produced after all, so the skip contradicts it: it keeps + // the group, and the session goes on, since the draft requires no more. + const [cached] = hooks.skipTrackGroups(entry.track, start, end + 1n); + if (cached) { + console.warn( + `protocol error: SUBSCRIBE_SKIP [${start}, ${end}] names group ${cached.start}, which arrived; keeping it`, + ); + } } } } diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts index 04ffb3ce4e1..9ae70f6ac24 100644 --- a/js/net/src/resume.test.ts +++ b/js/net/src/resume.test.ts @@ -1297,6 +1297,39 @@ test("a resumed reader reports a drop once and still serves a late group", async a.close(); }); +// A skip of a run the serving copy already dropped replaces the drop, so the resumed reader +// reports it again, as a skip, and only once. Mirrors Rust. +test("a resumed reader reports a skip over a reported drop", () => { + const front = new Front(); + const a = new BroadcastProducer(); + const ta = copy(a); + front.serve(a.consume()); + const sub = front.consumer.track("seq").subscribe({ delay: { max: Milli(60_000) } }); + const recv = () => { + const next = hooks.tryRecvGroup(sub, true); + if (next.kind === "group") { + next.group.close(); + return next.group.sequence; + } + if (next.kind === "drop") return [next.start, next.end, next.skip === true]; + return next.kind; + }; + ta.write(0n); + expect(recv()).toBe(0n); + hooks.dropTrackGroups(ta.producer, 1n, 3n, { error: new StreamError(StreamCode.NotFound) }); + expect(recv()).toEqual([1n, 3n, false]); + expect(recv()).toBe("idle"); + ta.producer.skipGroups(1n, 3n); + expect(recv()).toEqual([1n, 3n, true]); + expect(recv()).toBe("idle"); + ta.write(3n); + expect(recv()).toBe(3n); + + sub.close(); + front.consumer.close(); + a.close(); +}); + test("a separately polled drop does not deliver its run", () => { const front = new Front(); const source = new BroadcastProducer(); diff --git a/js/net/src/resume.ts b/js/net/src/resume.ts index e7f36b364ff..380d0c41d21 100644 --- a/js/net/src/resume.ts +++ b/js/net/src/resume.ts @@ -798,7 +798,9 @@ class Reader implements ResumeSource { if (open) { // Report once without marking it delivered: a late fill may still land. this.#drops = removeRun(this.#drops, open.start, open.end); - if (run.stale !== undefined) insertRun(this.#reportedStale, open.start, open.end); + // A skip is final, so no update reopens a stale report it replaced. + if (run.skip) this.#reportedStale = removeRun(this.#reportedStale, open.start, open.end); + else if (run.stale !== undefined) insertRun(this.#reportedStale, open.start, open.end); return { kind: "drop", ...open, error: run.error, skip: run.skip, stale: run.stale }; } } diff --git a/js/net/src/track.test.ts b/js/net/src/track.test.ts index 4dc4ed80a90..aee534d64c1 100644 --- a/js/net/src/track.test.ts +++ b/js/net/src/track.test.ts @@ -4855,8 +4855,8 @@ test("an errored end keeps its history", async () => { // Creating a group says nothing about the gap it leaves: groups may be created out of order. An // append says the producer will not go back, so it skips every hole below it that was never -// produced, keeping a reason already given. A received copy's groups arrive in any order, so its -// gap is not skipped. Mirrors Rust. +// produced, replacing a drop already given and reported. A received copy's groups arrive in any +// order, so its gap is not skipped. Mirrors Rust. test("an append skips the holes created groups left below it", () => { const producer = new TrackProducer("test"); producer.accept({}); @@ -4880,7 +4880,7 @@ test("an append skips the holes created groups left below it", () => { expect(drain()).toEqual([[2n, 3n, false], 0n, 4n]); producer.appendGroup().finish(); - expect(drain()).toEqual([[1n, 2n, true], [3n, 4n, true], 5n]); + expect(drain()).toEqual([[1n, 4n, true], 5n]); // The next append only names what was left since this one. producer.createGroup(7n).finish(); @@ -4931,9 +4931,9 @@ test("declared skips up to the end settle the track", async () => { reader.close(); }); -// A skip is final: creating at a sequence an append or skipGroups skipped throws, while a dropped -// sequence may be created again. skipGroups leaves cached sequences and final drops alone, and -// replaces a stale drop, which new terms would otherwise supersede. Mirrors Rust. +// A skip is final: creating at a sequence an append or skipGroups skipped throws, and a later drop +// leaves it skipped, while a dropped sequence may be created again. skipGroups leaves cached +// sequences alone and replaces every drop, final or stale. Mirrors Rust. test("creating a skipped sequence throws Skipped", () => { const producer = new TrackProducer("test"); producer.accept({}); @@ -4946,9 +4946,12 @@ test("creating a skipped sequence throws Skipped", () => { expect(() => producer.createGroup(6n)).toThrow(Skipped); // The next sequence, 5, was skipped too. expect(() => producer.appendGroup()).toThrow(Skipped); + // A later drop leaves a skipped sequence skipped. + expect(hooks.dropTrackGroups(producer, 1n, 2n, { error: DROPPED })).toEqual([]); + expect(() => producer.createGroup(1n)).toThrow(Skipped); // A dropped sequence may be created again. - hooks.dropTrackGroups(producer, 2n, 3n, { error: DROPPED }); - producer.createGroup(2n).finish(); + hooks.dropTrackGroups(producer, 9n, 10n, { error: DROPPED }); + producer.createGroup(9n).finish(); const other = new TrackProducer("test"); other.accept({}); @@ -4964,12 +4967,47 @@ test("creating a skipped sequence throws Skipped", () => { } expect(drops).toEqual([ [0n, 1n, true], - [2n, 3n, true], - [3n, 4n, false], - [4n, 5n, true], + [2n, 5n, true], ]); }); +// A skip after a drop already reported goes out again, replacing it, while a reader that moved past +// the drop is handed nothing again; a skip already reported is not reported twice, and a later drop +// leaves it skipped. Mirrors Rust. +test("a skip is reported over a reported drop", () => { + const producer = new TrackProducer("test"); + producer.accept({}); + const reader = producer.subscribe({ delay: { max: null } }); + const drain = () => { + const got: unknown[] = []; + for ( + let next = hooks.tryRecvGroup(reader, true); + next.kind !== "idle"; + next = hooks.tryRecvGroup(reader, true) + ) { + if (next.kind === "group") got.push(next.group.sequence); + else if (next.kind === "drop") got.push([next.start, next.end, next.skip === true]); + else throw new Error(`unexpected ${next.kind}`); + } + return got; + }; + producer.createGroup(0n).finish(); + hooks.dropTrackGroups(producer, 1n, 3n, { error: DROPPED }); + hooks.dropTrackGroups(producer, 4n, 5n, { error: DROPPED }); + producer.createGroup(5n).finish(); + expect(drain()).toEqual([[1n, 3n, false], [4n, 5n, false], 0n, 5n]); + + // Group 0 arrived, so the skip leaves it; both drops and the gap between become one skip. + expect(hooks.skipTrackGroups(producer, 0n, 5n)).toEqual([{ start: 0n, end: 1n }]); + expect(drain()).toEqual([[1n, 5n, true]]); + + expect(hooks.dropTrackGroups(producer, 2n, 3n, { error: DROPPED })).toEqual([]); + expect(drain()).toEqual([]); + expect(() => producer.createGroup(2n)).toThrow(Skipped); + reader.close(); + producer.close(); +}); + // A received stale run is judged from where it meets the reader's span, as Rust clips it: a reader // asking only its suffix, up to where the upstream's range ended, is settled by that suffix. test("a received stale run settles the suffix a narrower range asks for", () => { diff --git a/js/net/src/track.ts b/js/net/src/track.ts index ce9de9d74d6..eaceb250a20 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -1016,6 +1016,7 @@ export class Producer { hooks.trackReleased = (producer) => producer.#state.closed; hooks.trackFinished = (producer) => producer.#state.end.peek() !== undefined; hooks.dropTrackGroups = (producer, start, end, hole) => producer.#drop(start, end, hole, true); + hooks.skipTrackGroups = (producer, start, end) => producer.#skipReceived(start, end); hooks.supersedeStale = (producer, judgedEnd) => { for (const state of [producer.#state, ...producer.#sinks]) { state.judgedEnd = judgedEnd; @@ -1118,15 +1119,19 @@ export class Producer { return undefined; } - // Record `[start, end)` given up as `hole` says. An upstream drop (`open`) also ends the - // groups still arriving in it with its error, so their readers settle now rather than once - // the stream's reset lands; one that finished here stays whole. A skip names nothing produced. - #drop(start: bigint, end: bigint, hole: Hole, open = false): void { - if (start >= end) return; + // Record `[start, end)` given up as `hole` says, returning the runs recorded. A skip is final + // and the most authoritative verdict, so a drop never replaces one. An upstream drop (`open`) + // also ends the groups still arriving in it with its error, so their readers settle now + // rather than once the stream's reset lands; one that finished here stays whole. A skip names + // nothing produced. + #drop(start: bigint, end: bigint, hole: Hole, open = false): Run[] { + if (start >= end) return []; + const skipped = hole.skip ? [] : this.#state.drops.peek().filter((run) => run.skip); + const runs = runGaps(skipped, start, end); // A fill no longer owes what the track gave up. this.#fills.dropped(start, end); for (const state of [this.#state, ...this.#sinks]) { - state.drops.update((drops) => insertDrop(drops, start, end, hole)); + state.drops.update((drops) => runs.reduce((drops, run) => insertDrop(drops, run.start, run.end, hole), drops)); } if (open && !hole.skip) { for (const [sequence, entry] of this.#cached) { @@ -1140,6 +1145,7 @@ export class Producer { if (hole.stale === undefined) this.#arrive(start, end); // Giving up the last sequences below the end reaches it, as writing them would. this.#endLargest(); + return runs; } // A publisher may re-create a group it cut; the sequence is no longer given up. @@ -1697,8 +1703,8 @@ export class Producer { * Append a new group with the next sequence number: one past this producer's own writes. * * Appending says the producer will not go back, so it skips every sequence below the new - * group that it never produced and did not give up, from the lowest it produced since its - * last append, as {@link skipGroups} would. Those sequences are then final: creating one + * group that it never produced, from the lowest it produced since its last append, as + * {@link skipGroups} would, replacing any drop there. Those sequences are then final: creating one * throws {@link Skipped}. Appending alone never leaves such a gap; one left by out-of-order * {@link createGroup} calls is what it names. * @@ -1730,24 +1736,33 @@ export class Producer { * This and {@link appendGroup} are the only sources of a skip: a gap between created groups * says nothing, since groups may be created out of order, so only the producer knows a * sequence will never exist. A skip is final: creating a group at a skipped sequence throws - * {@link Skipped}. A sequence already produced or given up for good keeps what it is; a stale - * drop, which new terms may supersede, is replaced. Mirrors the Rust `skip_groups`. + * {@link Skipped}, and a later drop of it is ignored. + * + * A skip replaces any drop of the same sequences, whatever its error: a reader the drop + * already ended stays ended, and every later reader and relay sees the skip. A group already + * cached keeps what it is, since it was produced after all. Mirrors the Rust `skip_groups`. */ skipGroups(start: bigint, end: bigint): void { - if (this.#state.closed.peek() !== undefined) return; + this.#skipReceived(start, end); + } + + // `skipGroups`, returning the cached sequences it left alone: a skip naming a group that + // arrived contradicts it, which a subscriber reports. + #skipReceived(start: bigint, end: bigint): Run[] { + if (this.#state.closed.peek() !== undefined) return []; const cached: Run[] = []; for (const sequence of this.#cached.keys()) { if (sequence >= start && sequence < end) insertRun(cached, sequence, sequence + 1n); } this.#skipGaps(cached, start, end); + return cached; } - // Skip what of `[start, end)` neither `kept` nor a final drop already names. A stale drop is - // replaced: new terms would supersede it, leaving the sequence owed, while a skip is final. + // Skip what of `[start, end)` neither `kept` nor an earlier skip names, replacing any drop. #skipGaps(kept: readonly Run[], start: bigint, end: bigint): void { - const settled = this.#state.drops.peek().filter((run) => run.stale === undefined); + const skipped = this.#state.drops.peek().filter((run) => run.skip); for (const gap of runGaps(kept, start, end)) { - for (const open of runGaps(settled, gap.start, gap.end)) { + for (const open of runGaps(skipped, gap.start, gap.end)) { this.#drop(open.start, open.end, skipHole()); } } @@ -1760,11 +1775,17 @@ export class Producer { } // Settle an append at `sequence`: skip every sequence below it this track's own publisher - // never produced and did not give up for good, from the lowest it produced since its last - // append, since appending says it will not go back. A final reason already given stands. + // never produced, from the lowest it produced since its last append, since appending says it + // will not go back. The skip replaces a drop given there, as `skipGroups` does. #settleAppend(sequence: bigint): void { const from = this.#appended ?? this.#produced[0]?.start; - if (from !== undefined) this.#skipGaps(this.#produced, from, sequence); + if (from !== undefined) { + const kept = [...this.#produced]; + for (const cached of this.#cached.keys()) { + if (cached >= from && cached < sequence) insertRun(kept, cached, cached + 1n); + } + this.#skipGaps(kept, from, sequence); + } this.#produced = []; this.#appended = sequence + 1n; } @@ -2225,6 +2246,9 @@ export class Subscriber { // The reported runs judged stale, for a wire whose update re-offers them; see // `hooks.reofferStale`. #reportedStale: Run[] = []; + // The reported runs skipped: a skip replaces a drop, so a run reported dropped is reported + // again once it is skipped, and only a skip already reported stops it. + #reportedSkip: Run[] = []; // The group the frame-level helpers are currently draining, acquired through the // sequence cursor so frame reads and {@link Ordered.nextGroup} share one floor. #frameGroup?: GroupConsumer; @@ -3517,14 +3541,19 @@ export class Subscriber { return this.#parked(groups) ? { kind: "boundary" } : { kind: "done" }; } - // The first part of `[from, to)` this cursor has not reported yet, now reported, with why. + // The first part of `[from, to)` this cursor has not reported yet, now reported, with why. A + // skip goes out over a drop already reported, since it replaces it. #report(from: bigint, to: bigint, hole: Hole): (Recv & { kind: "drop" }) | undefined { const { start, end } = this.#cursor.peek(); for (const run of this.#window.runs(max(from, start), min(to, end ?? to))) { - const [gap] = runGaps(this.#reported, run.start, run.end); + const [gap] = runGaps(hole.skip ? this.#reportedSkip : this.#reported, run.start, run.end); if (!gap) continue; insertRun(this.#reported, gap.start, gap.end); - if (hole.stale !== undefined) insertRun(this.#reportedStale, gap.start, gap.end); + if (hole.skip) { + // Final, so no update reopens a stale report it replaced. + insertRun(this.#reportedSkip, gap.start, gap.end); + this.#reportedStale = removeRun(this.#reportedStale, gap.start, gap.end); + } else if (hole.stale !== undefined) insertRun(this.#reportedStale, gap.start, gap.end); return { kind: "drop", ...gap, error: hole.error, skip: hole.skip, stale: hole.stale }; } return undefined; diff --git a/quest/m1/drop-set.md b/quest/m1/drop-set.md index 550a7ae61d0..62255a664e4 100644 --- a/quest/m1/drop-set.md +++ b/quest/m1/drop-set.md @@ -107,16 +107,21 @@ decision above and the PR's `## Waiting` item on out-of-order creation: - ✅ **SKIP is final:** creating a group (or appending a group or datagram) at a skipped sequence is refused with `Error::Skipped` (JS `Skipped`). Re-creating a dropped or aborted sequence still works. -- **A SKIP replaces a stale (OLD) drop, but not a final one** (decided in - `/quest-iterate` from review, 2026-11-04). OLD is the only verdict an - update reopens: a SKIP that left an earlier OLD in place let the next - update supersede it, so a relay waited on or refilled a sequence its - upstream had declared never produced. A final DROP already settles every - reader and no update reopens it, so the first final verdict stands. The - rule lives in the model, so `skip_groups` and `append_group`'s implicit - skip follow it too. A received SKIP or final DROP also takes its groups +- ✅ **SKIP replaces every drop, final or stale** (maintainer, 2026-11-03, + `/quest-complete`, replacing the 2026-11-04 `/quest-iterate` rule that a + SKIP replaced only a stale OLD drop). "Never produced" is the most + authoritative fact about a sequence, so a received SKIP, `skip_groups`, or + `append_group`'s implicit skip overwrites an earlier DROP of any code (OLD, + aborted, evicted, NOT_FOUND), and a later DROP leaves a skip in place. + Readers that drop already ended stay ended, and nothing is re-delivered; the + run's reason becomes Skip, so every later reader, relay and downstream + report sees SKIP, and a relay forwards the SKIP even over a DROP it already + forwarded (`DropCursor::reported_skip`, `TrackRun::dropped`, and their JS + mirrors). A cached group is never overwritten: a received SKIP naming one + leaves the group, logs a protocol error, and keeps the session, since the + draft requires no more. A received SKIP or final DROP also takes its groups out of the subscriber's stale record, so the next update doesn't owe them - again. + again. Not chosen: SKIP replacing only a stale OLD drop. - ✅ **`append_group` is an implicit skip** (refinement, same session): it means "the next group", so the producer declares it won't go back, and it skips every sequence below it that never arrived and wasn't already given diff --git a/rs/moq-net/MIGRATE.md b/rs/moq-net/MIGRATE.md index 3ece960bddd..53463f1c315 100644 --- a/rs/moq-net/MIGRATE.md +++ b/rs/moq-net/MIGRATE.md @@ -4,6 +4,19 @@ How to update your code for each breaking change, newest first. ## Unreleased +### `track::Producer::skip_groups` declares groups never produced ([#7670](https://github.com/moq-dev/moq/pull/7670)) + +`track::Producer::drop_groups` is no longer public. To say the track will never hold some groups, call `skip_groups(range)`: readers stop waiting for them, no reader ends, and a relay forwards a lite-07 SUBSCRIBE_SKIP. + +```rust +// Before +producer.drop_groups(from..to, moq_net::StreamError::NotFound); +// After +producer.skip_groups(from..to); +``` + +A skip is final: `create_group` and `append_group` at a skipped sequence fail with `Error::Skipped`, and a later drop leaves it skipped. It replaces an earlier drop of the same groups, whatever its error, but never a cached group. `append_group` now skips the holes below it that earlier `create_group` calls left since the last append; call `create_group` for every sequence you still mean to fill before appending. + ### Datagrams are their own track kind ([#7752](https://github.com/moq-dev/moq/pull/7752)) A stream track no longer carries datagrams. `track::Producer::{append_datagram, insert_datagram}` and `track::Subscriber/Ordered::{poll_recv_datagram, recv_datagram}` are removed; a datagram track is created and read through the broadcast, in a namespace apart from stream tracks, with its own sequence counter. From 1ccc77932d0e975424f013b652045821547d6eb0 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 21:46:28 -0800 Subject: [PATCH 30/68] fix(net): a lite-07 group or final drop replaces a stale drop sent A front that switches routes can hand out the new route's group, or its final verdict, for a sequence the old route reported stale, before the update that reopens it. The publisher suppressed both, so the update left the subscriber owing a group nothing would re-offer. A lite-07 stale drop is no final verdict, so the group is served and the final drop sent. Also move the JS reaches doc above settles's. Co-Authored-By: Claude Opus 5.5 --- js/net/src/internal.ts | 5 +- js/net/src/lite/publisher.test.ts | 26 +++++++++- js/net/src/lite/publisher.ts | 39 ++++++++++---- js/net/src/track.ts | 16 +++--- rs/moq-net/src/lite/publisher.rs | 86 ++++++++++++++++++++++++++----- 5 files changed, 137 insertions(+), 35 deletions(-) diff --git a/js/net/src/internal.ts b/js/net/src/internal.ts index f6c3d016c1d..3dd9e0d4fcc 100644 --- a/js/net/src/internal.ts +++ b/js/net/src/internal.ts @@ -436,7 +436,10 @@ export interface ResumeSource { cachedGroup(sequence: bigint, from?: number): GroupConsumer | undefined; /** See {@link hooks.reofferGroups}, answered by the serving copy. */ reofferGroups(start: bigint, end: bigint): { groups: GroupConsumer[]; givenUp: Run[] }; - /** See {@link hooks.reofferStale}: the stale runs queued here and those the serving copy reported. */ + /** + * See {@link hooks.reofferStale}: the stale runs queued here, those this reader reported, which a + * route change keeps, and those the serving copy reported. + */ reofferStale(): Run[]; /** Let go of every copy. */ close(abort?: Error): void; diff --git a/js/net/src/lite/publisher.test.ts b/js/net/src/lite/publisher.test.ts index a063a5d889e..d20b8e8decf 100644 --- a/js/net/src/lite/publisher.test.ts +++ b/js/net/src/lite/publisher.test.ts @@ -6,7 +6,7 @@ import { Producer as BroadcastProducer } from "../broadcast.ts"; import { fromTransport, SessionCode, StreamCode, StreamError, toStreamCode } from "../error.ts"; import type { Producer as GroupProducer } from "../group.ts"; import { randomHop } from "../hop.ts"; -import { hooks } from "../internal.ts"; +import { hooks, staleHole } from "../internal.ts"; import { createMockTransportPair, textFrame } from "../mock.ts"; import { Producer as OriginProducer } from "../origin.ts"; import * as Path from "../path.ts"; @@ -1066,6 +1066,30 @@ for (const version of [Version.DRAFT_06, Version.DRAFT_07]) { } }); + // A lite-07 stale (OLD) drop is no final verdict, so a group handed out after it is served, as + // when a front switches to a route that delivers what the old route reported stale, before the + // update that reopens it. Earlier versions' drops are final. Mirrors Rust. + test(`lite ${version}: a group ${version === Version.DRAFT_07 ? "replaces" : "stays dropped after"} a stale drop`, async () => { + const sub = await servedSubscription({ version, startGroup: 0n }); + try { + sub.serve(0n); + expect(await sub.servedSequence()).toBe(0n); + expect("start" in (await decodeSubscribeResponse(sub.client.reader, version))).toBe(true); + + hooks.dropTrackGroups(sub.track, 1n, 2n, staleHole(60_000)); + const resp = await decodeSubscribeResponse(sub.client.reader, version); + if (!("drop" in resp)) throw new Error("expected SUBSCRIBE_DROP"); + expect([resp.drop.start, resp.drop.end]).toEqual([1n, 1n]); + + sub.serve(1n); + sub.serve(2n); + const next = await sub.servedSequence(); + expect(next).toBe(version === Version.DRAFT_07 ? 1n : 2n); + } finally { + await sub.close(); + } + }); + for (const declared of [false, true]) { const named = declared && version === Version.DRAFT_07 ? "in SUBSCRIBE_SKIP" : "in SUBSCRIBE_DROP"; test(`lite ${version}: the end sweep names a ${declared ? "declared skip" : "gap never declared"} ${named}`, async () => { diff --git a/js/net/src/lite/publisher.ts b/js/net/src/lite/publisher.ts index b819b203973..ee0017dd546 100644 --- a/js/net/src/lite/publisher.ts +++ b/js/net/src/lite/publisher.ts @@ -1287,6 +1287,9 @@ export class Publisher { // Lite-07: the part of `served` named in SUBSCRIBE_DROP, never served. A skip replaces a // drop, so it is sent over one of these, but never over a group served. let dropped: Run[] = []; + // Lite-07: the part of `dropped` named stale (OLD), which a final drop or a group replaces, + // as one a route the front switched to delivers after the update that reopens it. + let droppedStale: Run[] = []; // Give up, as `hole` says, whatever part of `[start, end)` this subscription has not been // served or given up yet. @@ -1294,17 +1297,24 @@ export class Publisher { if (!served) return true; const gaps = runGaps(served, start, end); if (hasSkip(this.version)) { - if (hole.skip) { - // A skip replaces a drop already sent, so the subscriber learns the groups were - // never produced; a reader the drop ended stays ended. - for (const run of dropped) { - const from = max(run.start, start); - const to = min(run.end, end); - if (from < to) insertRun(gaps, from, to); + // A skip replaces any drop already sent, so the subscriber learns the groups were + // never produced, and a final drop replaces a stale one, which the next update would + // otherwise reopen. A reader the earlier drop ended stays ended. + const replaces = hole.skip ? dropped : hole.stale === undefined ? droppedStale : []; + for (const run of replaces) { + const from = max(run.start, start); + const to = min(run.end, end); + if (from < to) insertRun(gaps, from, to); + } + for (const gap of gaps) { + if (hole.skip) { + dropped = removeRun(dropped, gap.start, gap.end); + droppedStale = removeRun(droppedStale, gap.start, gap.end); + continue; } - dropped = removeRun(dropped, start, end); - } else { - for (const gap of gaps) insertRun(dropped, gap.start, gap.end); + if (hole.stale === undefined) droppedStale = removeRun(droppedStale, gap.start, gap.end); + else insertRun(droppedStale, gap.start, gap.end); + insertRun(dropped, gap.start, gap.end); } } for (const gap of gaps) { @@ -1456,6 +1466,7 @@ export class Publisher { const owe = (start: bigint, end: bigint) => { if (served) served.splice(0, served.length, ...removeRun(served, start, end)); dropped = removeRun(dropped, start, end); + droppedStale = removeRun(droppedStale, start, end); }; for (const run of hooks.reofferStale(track)) owe(run.start, run.end); // A stale drop not sent yet, such as one an earlier update left to drop @@ -1663,7 +1674,13 @@ export class Publisher { // Never a second stream for a sequence this subscription already accounted for // (served, sent as a datagram, or dropped): before lite-07 a subscriber settles a // drop for good. A stale drop a lite-07 update reopened is owed again (see the update above). - if (served && runGaps(served, group.sequence, group.sequence + 1n).length === 0) { + // A lite-07 stale drop is no final verdict, and a Group Stream wins over it, so a group + // handed out after one is served, as one a route the front switched to delivers: the + // update that reopens the drop would never re-offer it. + if (runGaps(droppedStale, group.sequence, group.sequence + 1n).length === 0) { + droppedStale = removeRun(droppedStale, group.sequence, group.sequence + 1n); + dropped = removeRun(dropped, group.sequence, group.sequence + 1n); + } else if (served && runGaps(served, group.sequence, group.sequence + 1n).length === 0) { group.close(); continue; } diff --git a/js/net/src/track.ts b/js/net/src/track.ts index eaceb250a20..b0d7e5e9885 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -782,14 +782,6 @@ let goneAt: (producer: Producer, sequence: bigint) => bigint | undefined; /** A reader's last contiguous run of groups: its start, and its exclusive end unless it reads past the live edge. */ type Coverage = { start: bigint; end?: bigint } | undefined; -/** - * Whether `run` settles a reader judging under `budget` with `coverage`, on a track whose upstream - * judged stale runs against ranges ending at `judgedEnd` (undefined past the live edge, or - * unknown). Any run but a received stale one does. A stale verdict holds for a reader no wider - * than the budget it was judged under, whose range covers the run and every group above it up to - * where the upstream's range ended: its own successor and edge are then no older than those the - * verdict was judged against. Mirrors the Rust `settles`. - */ /** * Whether the live edge `received` reached `final`: no sequence below it is left that was neither * written nor given up, as a declared skip or a drop. Mirrors the Rust `TrackState::reaches`. @@ -798,6 +790,14 @@ function reaches(received: bigint, drops: readonly Run[], final: bigint): boolea return received >= final || runGaps(drops, received, final).length === 0; } +/** + * Whether `run` settles a reader judging under `budget` with `coverage`, on a track whose upstream + * judged stale runs against ranges ending at `judgedEnd` (undefined past the live edge, or + * unknown). Any run but a received stale one does. A stale verdict holds for a reader no wider + * than the budget it was judged under, whose range covers the run and every group above it up to + * where the upstream's range ended: its own successor and edge are then no older than those the + * verdict was judged against. Mirrors the Rust `settles`. + */ function settles(run: DropRun, budget: number, coverage: Coverage, judgedEnd: bigint | undefined): boolean { if (run.stale === undefined) return true; if (budget > run.stale || coverage === undefined || coverage.start > run.start) return false; diff --git a/rs/moq-net/src/lite/publisher.rs b/rs/moq-net/src/lite/publisher.rs index a948138defa..213a619188d 100644 --- a/rs/moq-net/src/lite/publisher.rs +++ b/rs/moq-net/src/lite/publisher.rs @@ -4444,6 +4444,9 @@ struct TrackRun { // Lite-07: the part of `served` named in SUBSCRIBE_DROP, never served. A skip replaces a // drop, so it is sent over one of these, but never over a group served. dropped: RangeSet, + // Lite-07: the part of `dropped` named stale (OLD), which a final drop or a group replaces, + // as one a route the front switched to delivers after the update that reopens it. + dropped_stale: RangeSet, /// The one group waiting for stream credit. The rest of the backlog waits in the cache, /// which bills, ages, and evicts it, and the cursor's budget skips whatever went stale /// meanwhile: a machine per parked group would hold state for as long as the peer @@ -4498,6 +4501,7 @@ impl TrackRun { end_sent: false, served, dropped: RangeSet::new(), + dropped_stale: RangeSet::new(), opening: None, children: kio::Tasks::new(), idle: None, @@ -4820,15 +4824,23 @@ impl TrackRun { // Never a second stream for a sequence this subscription already accounted // for (served or dropped): a group re-created after its // DROP stays dropped, since the subscriber may settle before it arrives. A - // stale drop a lite-07 update reopened is owed again (see `Self::owe`). - if self + // stale drop a lite-07 update reopened is owed again (see `Self::owe`). A + // lite-07 stale drop is no final verdict, and a Group Stream wins over it, so a + // group handed out after one is served, as one a route the front switched to + // delivers: the update that reopens the drop would never re-offer it. + let sequence = group.sequence; + if self.dropped_stale.contains(&sequence) + && let Some(end) = sequence.checked_add(1) + { + self.dropped_stale.remove(sequence..end); + self.dropped.remove(sequence..end); + } else if self .served .as_ref() - .is_some_and(|served| served.contains(&group.sequence)) + .is_some_and(|served| served.contains(&sequence)) { return Poll::Ready(Ok(ControlFlow::Continue(()))); } - let sequence = group.sequence; let Some(group) = position_group(group, &self.frames) else { // Its head is gone, and this subscriber didn't ask for a // partial group. Skip it rather than open a stream that can @@ -4944,6 +4956,7 @@ impl TrackRun { if let Some(served) = &mut self.served { for run in reopened { self.dropped.remove(run.clone()); + self.dropped_stale.remove(run.clone()); served.remove(run); } } @@ -4975,21 +4988,34 @@ impl TrackRun { } let mut gaps: RangeSet = served.gaps(&range).collect(); if self.ctx.version.has_skip() { + // A skip replaces any drop already sent, so the subscriber learns the groups were + // never produced, and a final drop replaces a stale one, which the next update + // would otherwise reopen. A reader the earlier drop ended stays ended. + let replaces = match hole { + Hole::Skip => &self.dropped, + hole if hole.stale_budget().is_none() => &self.dropped_stale, + Hole::Drop { .. } => &RangeSet::new(), + }; + let replaced: Vec<_> = replaces + .overlapping(&range) + .map(|run| run.start.max(range.start)..run.end.min(range.end)) + .collect(); + for run in replaced { + self.dropped_stale.remove(run.clone()); + gaps.insert(run); + } match hole { - // A skip replaces a drop already sent, so the subscriber learns the groups - // were never produced; a reader the drop ended stays ended. Hole::Skip => { - let replaced: Vec<_> = self - .dropped - .overlapping(&range) - .map(|run| run.start.max(range.start)..run.end.min(range.end)) - .collect(); - for run in replaced { + for run in gaps.iter() { self.dropped.remove(run.clone()); - gaps.insert(run); } } - Hole::Drop { .. } => self.dropped.extend(gaps.iter().cloned()), + Hole::Drop { .. } => { + self.dropped.extend(gaps.iter().cloned()); + if hole.stale_budget().is_some() { + self.dropped_stale.extend(gaps.iter().cloned()); + } + } } } served.extend(gaps.iter().cloned()); @@ -7080,6 +7106,38 @@ mod serve_group_test { drops } + /// A lite-07 stale (OLD) drop is no final verdict: a final drop of the same group replaces + /// it, and a group handed out after it is served, as when a front switches to a route that + /// delivers what the old route reported stale, before the update that reopens it. Earlier + /// versions' drops are final, so neither goes out there. + #[kio::test] + async fn a_final_drop_or_a_group_replaces_a_stale_drop_on_lite07() { + for version in [Version::Lite06, Version::Lite07] { + let mut track = track::Producer::new( + Arc::new(broadcast::Info::default()), + "test", + crate::track::Info::timed(), + ); + let subscriber = track.subscribe(None); + let (mut run, mut writer, log) = lite_run(version, SinkSession::new(Log::default()), subscriber); + write_group(&mut track, 0, 0); + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + assert!(run.start.is_some(), "{version}: started"); + let stale = Hole::stale(Duration::from_secs(1)); + run.drop_range(1..3, &stale, &mut writer).unwrap(); + run.drop_range(1..2, &Hole::drop(StreamError::Cancel), &mut writer).unwrap(); + write_group(&mut track, 2, 2); + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + + let (drops, opened) = match version.has_skip() { + true => (vec![(1, 2, 0), (1, 1, 0)], 2), + false => (vec![(1, 2, 0)], 1), + }; + assert_eq!(drop_updates(&mut writer, &log, version).await, drops, "{version}"); + assert_eq!(run.ctx.outcomes.opened.load(Ordering::Relaxed), opened, "{version}"); + } + } + /// A stale group already named in SUBSCRIBE_DROP stays dropped before lite-07 after any /// SUBSCRIBE_UPDATE, since those subscribers settle a drop for good. Lite-07 reopens it on /// every update: one that widens the max age past it serves it, and one that keeps the From 7837c35015dad3ef22497ca7bd33a4c057fedaa9 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 21:50:38 -0800 Subject: [PATCH 31/68] style: format Co-Authored-By: Claude Opus 5.5 --- js/net/MIGRATE.md | 2 +- js/net/src/lite/subscriber.test.ts | 13 ++++++------- js/net/src/track.ts | 4 +++- rs/moq-net/MIGRATE.md | 2 +- rs/moq-net/src/lite/publisher.rs | 9 +++------ 5 files changed, 14 insertions(+), 16 deletions(-) diff --git a/js/net/MIGRATE.md b/js/net/MIGRATE.md index 27109c5bcd1..bdc90b2d1b5 100644 --- a/js/net/MIGRATE.md +++ b/js/net/MIGRATE.md @@ -6,7 +6,7 @@ How to update your code for each breaking change, newest first. ### `Track.Producer.skipGroups` declares groups never produced ([#7670](https://github.com/moq-dev/moq/pull/7670)) -`skipGroups(start, end)` says the track will never hold `[start, end)`: readers stop waiting for them, no reader ends, and a relay forwards a lite-07 SUBSCRIBE_SKIP. +`skipGroups(start, end)` says the track will never hold `[start, end)`: readers stop waiting for them, no reader ends, and a relay forwards a lite-07 `SUBSCRIBE_SKIP`. A skip is final: `createGroup` and `appendGroup` at a skipped sequence throw `Skipped`, and a later drop leaves it skipped. It replaces an earlier drop of the same groups, whatever its error, but never a cached group. `appendGroup` now skips the holes below it that earlier `createGroup` calls left since the last append; create every sequence you still mean to fill before appending. diff --git a/js/net/src/lite/subscriber.test.ts b/js/net/src/lite/subscriber.test.ts index 5f3145209da..cfe096f46b5 100644 --- a/js/net/src/lite/subscriber.test.ts +++ b/js/net/src/lite/subscriber.test.ts @@ -2685,17 +2685,16 @@ test("a skip replaces a final drop but not a cached group on lite-07", async () const reader = entry.track.subscribe({ delay: { max: null } }); const got: unknown[] = []; - for (let next = hooks.tryRecvGroup(reader, true); next.kind !== "idle"; next = hooks.tryRecvGroup(reader, true)) { + for ( + let next = hooks.tryRecvGroup(reader, true); + next.kind !== "idle"; + next = hooks.tryRecvGroup(reader, true) + ) { if (next.kind === "group") got.push(next.group.sequence); else if (next.kind === "drop") got.push([next.start, next.end, next.skip === true]); else break; } - expect(got).toEqual([ - [1n, 3n, true], - [4n, 5n, true], - [6n, 7n, true], - 5n, - ]); + expect(got).toEqual([[1n, 3n, true], [4n, 5n, true], [6n, 7n, true], 5n]); reader.close(); } finally { warn.mockRestore(); diff --git a/js/net/src/track.ts b/js/net/src/track.ts index b0d7e5e9885..562f9f50b60 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -1131,7 +1131,9 @@ export class Producer { // A fill no longer owes what the track gave up. this.#fills.dropped(start, end); for (const state of [this.#state, ...this.#sinks]) { - state.drops.update((drops) => runs.reduce((drops, run) => insertDrop(drops, run.start, run.end, hole), drops)); + state.drops.update((drops) => + runs.reduce((drops, run) => insertDrop(drops, run.start, run.end, hole), drops), + ); } if (open && !hole.skip) { for (const [sequence, entry] of this.#cached) { diff --git a/rs/moq-net/MIGRATE.md b/rs/moq-net/MIGRATE.md index 53463f1c315..2afd76cd933 100644 --- a/rs/moq-net/MIGRATE.md +++ b/rs/moq-net/MIGRATE.md @@ -6,7 +6,7 @@ How to update your code for each breaking change, newest first. ### `track::Producer::skip_groups` declares groups never produced ([#7670](https://github.com/moq-dev/moq/pull/7670)) -`track::Producer::drop_groups` is no longer public. To say the track will never hold some groups, call `skip_groups(range)`: readers stop waiting for them, no reader ends, and a relay forwards a lite-07 SUBSCRIBE_SKIP. +`track::Producer::drop_groups` is no longer public. To say the track will never hold some groups, call `skip_groups(range)`: readers stop waiting for them, no reader ends, and a relay forwards a lite-07 `SUBSCRIBE_SKIP`. ```rust // Before diff --git a/rs/moq-net/src/lite/publisher.rs b/rs/moq-net/src/lite/publisher.rs index 213a619188d..99c68977dc5 100644 --- a/rs/moq-net/src/lite/publisher.rs +++ b/rs/moq-net/src/lite/publisher.rs @@ -4834,11 +4834,7 @@ impl TrackRun { { self.dropped_stale.remove(sequence..end); self.dropped.remove(sequence..end); - } else if self - .served - .as_ref() - .is_some_and(|served| served.contains(&sequence)) - { + } else if self.served.as_ref().is_some_and(|served| served.contains(&sequence)) { return Poll::Ready(Ok(ControlFlow::Continue(()))); } let Some(group) = position_group(group, &self.frames) else { @@ -7125,7 +7121,8 @@ mod serve_group_test { assert!(run.start.is_some(), "{version}: started"); let stale = Hole::stale(Duration::from_secs(1)); run.drop_range(1..3, &stale, &mut writer).unwrap(); - run.drop_range(1..2, &Hole::drop(StreamError::Cancel), &mut writer).unwrap(); + run.drop_range(1..2, &Hole::drop(StreamError::Cancel), &mut writer) + .unwrap(); write_group(&mut track, 2, 2); while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} From 143d1a31c59b6c98eebdf3357830f2cec23fee2c Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 22:03:17 -0800 Subject: [PATCH 32/68] fix(net): only a forwarded verdict replaces an earlier lite-07 drop The end sweep names what the subscription never got as NOT_FOUND, which replaced a stale drop already sent and made it final, so an update could no longer reach it. Only a skip or drop the reader reached replaces one. Co-Authored-By: Claude Opus 5.5 --- js/net/src/lite/publisher.ts | 28 ++++++++----- rs/moq-net/src/lite/publisher.rs | 71 ++++++++++++++++++-------------- 2 files changed, 58 insertions(+), 41 deletions(-) diff --git a/js/net/src/lite/publisher.ts b/js/net/src/lite/publisher.ts index ee0017dd546..82490ae078e 100644 --- a/js/net/src/lite/publisher.ts +++ b/js/net/src/lite/publisher.ts @@ -1297,15 +1297,6 @@ export class Publisher { if (!served) return true; const gaps = runGaps(served, start, end); if (hasSkip(this.version)) { - // A skip replaces any drop already sent, so the subscriber learns the groups were - // never produced, and a final drop replaces a stale one, which the next update would - // otherwise reopen. A reader the earlier drop ended stays ended. - const replaces = hole.skip ? dropped : hole.stale === undefined ? droppedStale : []; - for (const run of replaces) { - const from = max(run.start, start); - const to = min(run.end, end); - if (from < to) insertRun(gaps, from, to); - } for (const gap of gaps) { if (hole.skip) { dropped = removeRun(dropped, gap.start, gap.end); @@ -1325,6 +1316,21 @@ export class Publisher { return true; }; + // Forward a verdict the reader reached on `[start, end)`, as `drop` does. On lite-07 it also + // replaces what it overrides: a skip any drop already sent, and a final drop a stale one, + // which the next update would otherwise reopen. A reader the earlier drop ended stays ended. + const forward = async (start: bigint, end: bigint, hole: Hole): Promise => { + if (served && hasSkip(this.version)) { + const replaces = hole.skip ? dropped : hole.stale === undefined ? droppedStale : []; + for (const run of replaces) { + const from = max(run.start, start); + const to = min(run.end, end); + if (from < to) served.splice(0, served.length, ...removeRun(served, from, to)); + } + } + return drop(start, end, hole); + }; + // The track's exclusive final boundary. A Rust subscriber feeds SUBSCRIBE_END // straight into finish_at, so it must name the track's boundary (which counts // datagram sequences too), not the delivered range: a subscription cap can hold @@ -1395,7 +1401,7 @@ export class Publisher { // What was given up below the start is never named, so a wider budget may serve it. if (range.start < startSequence) hooks.unreport(track, range.start, min(range.end, startSequence)); - if (!(await drop(max(range.start, startSequence), range.end, range))) return false; + if (!(await forward(max(range.start, startSequence), range.end, range))) return false; } skipped.length = 0; } @@ -1526,7 +1532,7 @@ export class Publisher { throw recv.error; case "drop": if (!startSent) skipped.push(recv); - else if (!(await drop(recv.start, recv.end, recv))) return false; + else if (!(await forward(recv.start, recv.end, recv))) return false; continue; case "idle": // A start past everything the track has (a subscriber resuming just after diff --git a/rs/moq-net/src/lite/publisher.rs b/rs/moq-net/src/lite/publisher.rs index 99c68977dc5..d6bdb324332 100644 --- a/rs/moq-net/src/lite/publisher.rs +++ b/rs/moq-net/src/lite/publisher.rs @@ -4939,7 +4939,7 @@ impl TrackRun { match self.start { None => self.skipped.insert(range, hole), Some(_) => { - self.drop_range(range, &hole, writer)?; + self.forward_range(range, &hole, writer)?; } } Ok(true) @@ -4968,6 +4968,31 @@ impl TrackRun { } } + /// Forward a verdict the cursor reached on `range`, as [`Self::drop_range`] does. On lite-07 + /// it also replaces what it overrides: a skip any drop already sent, and a final drop a stale + /// one, which the next update would otherwise reopen. A reader the earlier drop ended stays + /// ended. + fn forward_range( + &mut self, + range: Range, + hole: &Hole, + writer: &mut Writer, + ) -> Result { + if self.ctx.version.has_skip() + && let Some(served) = &mut self.served + { + let replaces = match hole { + Hole::Skip => Some(&self.dropped), + Hole::Drop { stale: None, .. } => Some(&self.dropped_stale), + Hole::Drop { .. } => None, + }; + for run in replaces.into_iter().flat_map(|runs| runs.overlapping(&range)) { + served.remove(run.start.max(range.start)..run.end.min(range.end)); + } + } + self.drop_range(range, hole, writer) + } + /// Give up, as `hole` says, whatever part of `range` this subscription has not been /// served or given up yet. Returns whether any was buffered. fn drop_range( @@ -4982,34 +5007,20 @@ impl TrackRun { if range.is_empty() { return Ok(false); } - let mut gaps: RangeSet = served.gaps(&range).collect(); + let gaps: RangeSet = served.gaps(&range).collect(); if self.ctx.version.has_skip() { - // A skip replaces any drop already sent, so the subscriber learns the groups were - // never produced, and a final drop replaces a stale one, which the next update - // would otherwise reopen. A reader the earlier drop ended stays ended. - let replaces = match hole { - Hole::Skip => &self.dropped, - hole if hole.stale_budget().is_none() => &self.dropped_stale, - Hole::Drop { .. } => &RangeSet::new(), - }; - let replaced: Vec<_> = replaces - .overlapping(&range) - .map(|run| run.start.max(range.start)..run.end.min(range.end)) - .collect(); - for run in replaced { - self.dropped_stale.remove(run.clone()); - gaps.insert(run); - } - match hole { - Hole::Skip => { - for run in gaps.iter() { - self.dropped.remove(run.clone()); + for gap in gaps.iter() { + match hole { + Hole::Skip => { + self.dropped.remove(gap.clone()); + self.dropped_stale.remove(gap.clone()); } - } - Hole::Drop { .. } => { - self.dropped.extend(gaps.iter().cloned()); - if hole.stale_budget().is_some() { - self.dropped_stale.extend(gaps.iter().cloned()); + Hole::Drop { .. } => { + self.dropped.insert(gap.clone()); + match hole.stale_budget() { + Some(_) => self.dropped_stale.insert(gap.clone()), + None => self.dropped_stale.remove(gap.clone()), + } } } } @@ -5158,7 +5169,7 @@ impl TrackRun { // The subscriber now owes itself everything from the start, so a range given up // above it is named as it would have been after the start. for (range, hole) in skipped.iter() { - self.drop_range(range.start.max(start)..range.end, hole, writer)?; + self.forward_range(range.start.max(start)..range.end, hole, writer)?; } Ok(()) } @@ -7120,8 +7131,8 @@ mod serve_group_test { while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} assert!(run.start.is_some(), "{version}: started"); let stale = Hole::stale(Duration::from_secs(1)); - run.drop_range(1..3, &stale, &mut writer).unwrap(); - run.drop_range(1..2, &Hole::drop(StreamError::Cancel), &mut writer) + run.forward_range(1..3, &stale, &mut writer).unwrap(); + run.forward_range(1..2, &Hole::drop(StreamError::Cancel), &mut writer) .unwrap(); write_group(&mut track, 2, 2); while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} From 8458868a2cdfe381cd0c61697d30da82c97d9f06 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 23:34:03 -0800 Subject: [PATCH 33/68] fix(net): retire a lite-07 stale report its group or final verdict replaces A lite-07 publisher serves a group, or forwards a final drop, over an OLD it already sent (decision 14), but the reader's cursor kept the OLD in its stale record, so the next SUBSCRIBE_UPDATE reopened the served sequence, sent OLD again, and named it NOT_FOUND at the end. - A handed-out group, a SKIP, or a final drop retires the stale report in DropCursor, the resumed front's Reader, and JS. A plain cursor reports a final drop over a stale one it reported, as the resumed reader does. - The publisher records its own expired streams' OLD as stale drops sent, so a later group or final drop replaces them and no update drops them again. - A received OLD leaves a group whose stream arrived, still arriving or not, and an update owes again only the stale runs the track still holds, so a group that overtook or followed its OLD stays accounted for. - The lite draft says so. Co-Authored-By: Claude Opus 5.5 --- drafts/draft-lcurley-moq-lite.md | 5 +- js/net/src/internal.ts | 5 +- js/net/src/lite/publisher.test.ts | 86 ++++++++++++++++++++ js/net/src/lite/publisher.ts | 38 +++++---- js/net/src/lite/subscriber.test.ts | 82 +++++++++++++++++++ js/net/src/lite/subscriber.ts | 20 +++-- js/net/src/resume.test.ts | 34 ++++++++ js/net/src/resume.ts | 11 ++- js/net/src/track.ts | 38 ++++++--- quest/m1/drop-set.md | 19 +++++ rs/moq-net/src/lite/publisher.rs | 121 +++++++++++++++++++++++++++-- rs/moq-net/src/lite/subscriber.rs | 77 ++++++++++++++++-- rs/moq-net/src/model/resume.rs | 34 +++++++- rs/moq-net/src/model/track.rs | 91 ++++++++++++++++------ 14 files changed, 587 insertions(+), 74 deletions(-) diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index 1ac81905c93..d38e84717fb 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -1638,6 +1638,7 @@ SUBSCRIBE_DROP Message { **Error Code** is a [stream error code](#error-codes) saying why the groups were given up, as a reset retaining the Group Stream header would: for example OLD for one past the subscriber's budget, EVICTED for one the publisher no longer holds, or NOT_FOUND for one a relay never received. A subscriber MUST end the readers of a dropped group it holds with that error, including a group whose stream is still arriving, rather than end it cleanly. +A stale (OLD) drop is the exception: it ends no group whose Group Stream arrived, since that stream wins over it (see below), and a publisher that gives up a stream it opened resets it. **Update**: The [SUBSCRIBE](#subscribe) or [SUBSCRIBE_UPDATE](#subscribe-update) `Update` in effect when the publisher gave the groups up. @@ -1655,6 +1656,8 @@ A subscriber that already ended readers on the drop does not deliver anything ag A drop that follows a skip of the same sequences changes nothing. A Group Stream for a sequence the subscriber already saw dropped, as after such a re-offer, wins over the drop: the subscriber delivers it. +A Group Stream also wins over a stale drop of its sequence that arrives after it, since the stream can overtake a drop the publisher sent first. +Either way the Group Stream accounts for the sequence, and the next SUBSCRIBE_UPDATE does not reopen it: the publisher, having served it, neither serves it again nor names it in another drop. For every sequence in the subscription's range, the publisher MUST send a Group Stream or name it in SUBSCRIBE_SKIP or SUBSCRIBE_DROP. This includes streams reset before their headers were delivered. @@ -2016,7 +2019,7 @@ The `Message Length` describes the payload size on the wire. - SUBSCRIBE_DROP's Error Code is a stream error code, which ends the dropped groups' readers, including one still arriving. - Refuse a SUBSCRIBE_DROP whose Group Start is greater than Group End plus one on every version that carries it. - Added SUBSCRIBE_SKIP (type 0x3) for sequences the publisher declared it will never produce, which ends no reader and is final: a Group is never created at a skipped sequence. A skip overrides an earlier drop of the same sequences, whatever its Error Code, without re-delivering to readers the drop already ended, and never overrides a Group that arrived. A gap in the sequence numbers is not a skip, since Groups may be created out of order; a sequence never had and never declared skipped is named in SUBSCRIBE_DROP. SUBSCRIBE_DROP no longer names declared skips. -- Added `Update` to SUBSCRIBE, SUBSCRIBE_UPDATE, and SUBSCRIBE_DROP: the number of the terms in effect. A stale (OLD) drop holds only under the update it names, judged against the budget the subscriber sent with it; every later SUBSCRIBE_UPDATE reopens it, and the publisher re-offers the group or drops it again under the new `Update`. Any other drop is final. A Group Stream wins over its earlier drop. +- Added `Update` to SUBSCRIBE, SUBSCRIBE_UPDATE, and SUBSCRIBE_DROP: the number of the terms in effect. A stale (OLD) drop holds only under the update it names, judged against the budget the subscriber sent with it; every later SUBSCRIBE_UPDATE reopens it, and the publisher re-offers the group or drops it again under the new `Update`. Any other drop is final. A Group Stream wins over its earlier drop, and over a stale drop that arrives after it; no update reopens a sequence its Group Stream accounted for. - SUBSCRIBE_FINISH declares the final boundary while lower groups can still be in flight, and no longer closes the Subscribe Stream: the publisher keeps reading SUBSCRIBE_UPDATE, so an update can reach an ended Track's history. - A covered subscription, whose every Range has an End and whose every Group was sent or dropped, also stays open for an update; the publisher never FINs first. - A publisher that withheld SUBSCRIBE_OK sends it, then the drops it withheld, once no group can resolve the start: every Group of a bounded subscription was given up, or the Track finished. diff --git a/js/net/src/internal.ts b/js/net/src/internal.ts index 3dd9e0d4fcc..508a0494c2e 100644 --- a/js/net/src/internal.ts +++ b/js/net/src/internal.ts @@ -516,9 +516,10 @@ export const hooks: { /** * Forget every run a received copy holds as stale: the terms its upstream judged them under * were replaced, so they are unknown again. The new terms' ranges end at `judgedEnd`, - * undefined past the live edge or unknown. Mirrors the Rust `supersede_stale`. + * undefined past the live edge or unknown. Returns the runs forgotten: a group that arrived + * since its stale drop replaced it, so it is not among them. Mirrors the Rust `supersede_stale`. */ - supersedeStale: (producer: Producer, judgedEnd: bigint | undefined) => void; + supersedeStale: (producer: Producer, judgedEnd: bigint | undefined) => Run[]; /** Mark a track as fed by a wire subscription, which owes missing groups. Assigned by `track.ts`. */ receiveTrack: (producer: Producer) => void; /** Create a received group at its wire offset before publishing it to readers. Assigned by `track.ts`. */ diff --git a/js/net/src/lite/publisher.test.ts b/js/net/src/lite/publisher.test.ts index d20b8e8decf..4fe7034ddfa 100644 --- a/js/net/src/lite/publisher.test.ts +++ b/js/net/src/lite/publisher.test.ts @@ -1349,6 +1349,92 @@ test("lite draft-07: an update drops an expired stream again while it is still s } }); +// The drops and skips on a lite-07 subscribe stream up to its SUBSCRIBE_FINISH, with the update each +// drop names. +async function holesToEnd(reader: Reader): Promise { + const holes: unknown[] = []; + for (;;) { + const next = await decodeSubscribeResponseMaybe(reader, Version.DRAFT_07); + if (!next) return holes; + if ("drop" in next) holes.push(["drop", next.drop.start, next.drop.end, next.drop.update]); + if ("skip" in next) holes.push(["skip", next.skip.start, next.skip.end]); + if ("end" in next) return holes; + } +} + +// A group that wins over a lite-07 stale drop already sent retires it: the next update neither +// sends the drop again nor reopens the served group, and the end names nothing for it. Here a +// relay's upstream reported group 1 stale, then delivered it, as a front that switched routes +// does. Mirrors the Rust `a_group_served_over_a_stale_drop_retires_it`. +test("lite draft-07: a group served over a stale drop retires it", async () => { + const sub = await servedSubscription({ version: Version.DRAFT_07, startGroup: 0n }); + try { + sub.serve(0n); + expect(await sub.servedSequence()).toBe(0n); + expect("start" in (await decodeSubscribeResponse(sub.client.reader, Version.DRAFT_07))).toBe(true); + + hooks.dropTrackGroups(sub.track, 1n, 2n, staleHole(60_000)); + const resp = await decodeSubscribeResponse(sub.client.reader, Version.DRAFT_07); + if (!("drop" in resp)) throw new Error("expected SUBSCRIBE_DROP"); + expect([resp.drop.start, resp.drop.end]).toEqual([1n, 1n]); + sub.serve(1n); + expect(await sub.servedSequence()).toBe(1n); + + await replayUpdate({ priority: 1, startGroup: 0n, update: 1n }).encode(sub.client.writer, Version.DRAFT_07); + await flush(); + sub.serve(2n); + expect(await sub.servedSequence()).toBe(2n); + sub.track.finish(); + sub.track.close(); + // No second OLD, and no NOT_FOUND at the end. + expect(await holesToEnd(sub.client.reader)).toEqual([]); + } finally { + await sub.close(); + } +}); + +// A lite-07 stream that expired as stale is a stale drop like any other: a final drop the reader +// reaches for it later replaces it, and no update drops it again then. Mirrors the Rust +// `a_final_drop_replaces_an_expired_stream_on_lite07`. +test("lite draft-07: a final drop replaces an expired stream", async () => { + const clock = new Manual(); + const installed = clock.install(); + const budget = Milli(1_000); + const sub = await servedSubscription({ version: Version.DRAFT_07, startGroup: 0n, maxAge: budget }); + try { + sub.serve(0n); + expect(await sub.servedSequence()).toBe(0n); + expect("start" in (await decodeSubscribeResponse(sub.client.reader, Version.DRAFT_07))).toBe(true); + + const reset = sub.stall(); + sub.serve(1n); + expect(await sub.servedHeader()).toBe(1n); + await flush(); + await clock.advance(Nano.fromMilli(Milli(budget * 10))); + sub.serve(2n); + expect(await sub.servedSequence()).toBe(2n); + await clock.advance(Nano.fromMilli(budget)); + expect(toStreamCode(fromTransport(await reset))).toBe(StreamCode.Old); + const first = await decodeSubscribeResponse(sub.client.reader, Version.DRAFT_07); + if (!("drop" in first)) throw new Error("expected SUBSCRIBE_DROP"); + expect([first.drop.start, first.drop.end, first.drop.update]).toEqual([1n, 1n, 0n]); + + hooks.dropTrackGroups(sub.track, 1n, 2n, { error: new StreamError(StreamCode(70), { message: "cut" }) }); + const final = await decodeSubscribeResponse(sub.client.reader, Version.DRAFT_07); + if (!("drop" in final)) throw new Error("expected SUBSCRIBE_DROP"); + expect([final.drop.start, final.drop.end, final.drop.error]).toEqual([1n, 1n, 70]); + + await replayUpdate({ priority: 1, startGroup: 0n, update: 1n }).encode(sub.client.writer, Version.DRAFT_07); + await flush(); + sub.track.finish(); + sub.track.close(); + expect(await holesToEnd(sub.client.reader)).toEqual([]); + } finally { + await sub.close(); + installed.close(); + } +}); + // An update that lowers the start below SUBSCRIBE_START asks for those groups too, so the // end sweep drops them, as the Rust publisher does. test("lite draft-06: an update lowering the start is swept at the end", async () => { diff --git a/js/net/src/lite/publisher.ts b/js/net/src/lite/publisher.ts index 82490ae078e..9ff55425981 100644 --- a/js/net/src/lite/publisher.ts +++ b/js/net/src/lite/publisher.ts @@ -1290,6 +1290,9 @@ export class Publisher { // Lite-07: the part of `dropped` named stale (OLD), which a final drop or a group replaces, // as one a route the front switched to delivers after the update that reopens it. let droppedStale: Run[] = []; + // Lite-07 groups whose stream expired as stale, which every update drops again under its + // number: the subscriber reopens each stale drop then, and the stream already reset. + const expired: Run[] = []; // Give up, as `hole` says, whatever part of `[start, end)` this subscription has not been // served or given up yet. @@ -1322,15 +1325,26 @@ export class Publisher { const forward = async (start: bigint, end: bigint, hole: Hole): Promise => { if (served && hasSkip(this.version)) { const replaces = hole.skip ? dropped : hole.stale === undefined ? droppedStale : []; - for (const run of replaces) { + for (const run of [...replaces]) { const from = max(run.start, start); const to = min(run.end, end); - if (from < to) served.splice(0, served.length, ...removeRun(served, from, to)); + if (from >= to) continue; + replaceStale(from, to); + served.splice(0, served.length, ...removeRun(served, from, to)); } } return drop(start, end, hole); }; + // Forget that `[start, end)` went out as a drop: a verdict or group replaces it. A stream of + // this subscription that expired there is no longer owed again on an update, since what + // replaced it settles it. + const replaceStale = (start: bigint, end: bigint) => { + dropped = removeRun(dropped, start, end); + droppedStale = removeRun(droppedStale, start, end); + expired.splice(0, expired.length, ...removeRun(expired, start, end)); + }; + // The track's exclusive final boundary. A Rust subscriber feeds SUBSCRIBE_END // straight into finish_at, so it must name the track's boundary (which counts // datagram sequences too), not the delivered range: a subscription cap can hold @@ -1380,9 +1394,6 @@ export class Publisher { const drops = new Signal(0); // Groups whose stream failed, with why, for the control stream to name, as runs. const pendingDrops: { start: bigint; end: bigint; error: Error }[] = []; - // Lite-07 groups whose stream expired as stale, which every update drops again under its - // number: the subscriber reopens each stale drop then, and the stream already reset. - const expired: Run[] = []; // The expired groups an update re-offers, still cached and in range, served again in turn. const reoffered: group.Consumer[] = []; @@ -1407,13 +1418,15 @@ export class Publisher { } if (pendingDrops.length > 0) { for (const { start, end, error } of pendingDrops.splice(0)) { - // A stream expired under this subscription's budget is a stale verdict. - if (hasSkip(this.version) && toStreamCode(error) === StreamCode.Old) + // A stream expired under this subscription's budget is a stale verdict, which + // a later verdict or group replaces as any stale drop sent. + if (hasSkip(this.version) && toStreamCode(error) === StreamCode.Old) { insertRun(expired, start, end); - const dropped = SubscribeDrop.given(start, end - 1n, error, this.version, updateNumber); - if ( - !(await controls.response(encodeSubscribeResponse(stream, { drop: dropped }, this.version))) - ) + insertRun(dropped, start, end); + insertRun(droppedStale, start, end); + } + const msg = SubscribeDrop.given(start, end - 1n, error, this.version, updateNumber); + if (!(await controls.response(encodeSubscribeResponse(stream, { drop: msg }, this.version)))) return false; } } @@ -1684,8 +1697,7 @@ export class Publisher { // handed out after one is served, as one a route the front switched to delivers: the // update that reopens the drop would never re-offer it. if (runGaps(droppedStale, group.sequence, group.sequence + 1n).length === 0) { - droppedStale = removeRun(droppedStale, group.sequence, group.sequence + 1n); - dropped = removeRun(dropped, group.sequence, group.sequence + 1n); + replaceStale(group.sequence, group.sequence + 1n); } else if (served && runGaps(served, group.sequence, group.sequence + 1n).length === 0) { group.close(); continue; diff --git a/js/net/src/lite/subscriber.test.ts b/js/net/src/lite/subscriber.test.ts index cfe096f46b5..dd836ab06cc 100644 --- a/js/net/src/lite/subscriber.test.ts +++ b/js/net/src/lite/subscriber.test.ts @@ -2567,6 +2567,88 @@ test("a lite-07 publisher FIN keeps a covered bounded subscription's copy", asyn } }); +// A group that arrives over a lite-07 stale (OLD) drop wins over it, whichever lands first: the +// publisher serves a group after its stale drop, as a front that switched routes does, and the +// group's stream may overtake the drop. The next update owes it no more, and a group still +// arriving is never aborted by the drop. Mirrors the Rust `a_group_wins_over_its_stale_drop`. +test("a group wins over its stale drop on lite-07", async () => { + const version = Version.DRAFT_07; + const { quic, streams, opened } = fakeSession(); + const subscriber = new Subscriber(quic, version, HopSchema.parse(1n)); + type Entry = { + track: Producer; + tail: { covers(start: bigint, end: bigint): boolean; open(sequence: bigint): () => void }; + }; + let entry: Entry | undefined; + const internal = subscriber as unknown as { + runResponses: (stream: unknown, entry: Entry) => Promise; + }; + const runResponses = internal.runResponses.bind(internal); + const responses = spyOn(internal, "runResponses").mockImplementation((stream, captured) => { + entry = captured; + return runResponses(stream, captured); + }); + const broadcast = subscriber.consume(Path.from("room")); + const track = broadcast.track("video").subscribe({ delay: { max: Milli(500) } }); + try { + await opened(0); + await answerTrackInfo(streams[0], version); + const stream = await opened(1); + await stream.reading; + const send = async (...resps: SubscribeResponse[]) => { + const chunks: Uint8Array[] = []; + const writer = new Writer( + new WritableStream({ write: (chunk) => void chunks.push(new Uint8Array(chunk)) }), + version, + ); + for (const resp of resps) await encodeSubscribeResponse(writer, resp, version); + for (const chunk of chunks) stream.inbound.enqueue(chunk); + await drainReactions(); + }; + const old = (sequence: bigint): SubscribeResponse => ({ + drop: new SubscribeDrop({ start: sequence, end: sequence, error: StreamCode.Old }), + }); + // A group stream's header: the tail accounts for it, and the track holds the group. + const arrive = (sequence: bigint) => { + if (!entry) throw new Error("no subscription"); + const read = entry.tail.open(sequence); + return { group: hooks.receiveGroup(entry.track, { sequence, start: 0 }), read }; + }; + + await send({ start: new SubscribeStart([0n]) }); + // OLD, then the group, which finishes. + await send(old(1n)); + const one = arrive(1n); + one.group.finish(); + one.read(); + // The group first, still arriving when its OLD lands. + const two = arrive(2n); + await send(old(2n)); + // OLD, then the group, still arriving at the update. + await send(old(3n)); + const three = arrive(3n); + // OLD, and no group. + await send(old(4n)); + if (!entry) throw new Error("no subscription"); + + track.update({ priority: 1, delay: { max: Milli(500) } }); + await drainReactions(); + // Every group that arrived stays accounted for; the one that never arrived is owed again. + expect(entry.tail.covers(1n, 4n)).toBe(true); + expect(entry.tail.covers(4n, 5n)).toBe(false); + // Neither the drop nor the update aborts a group still arriving. + expect(two.group.closed.peek()).toBeUndefined(); + expect(three.group.closed.peek()).toBeUndefined(); + two.read(); + three.read(); + } finally { + responses.mockRestore(); + track.close(); + broadcast.close(); + subscriber.close(); + } +}); + // A SKIP, or any final drop, that follows an OLD drop of the same groups replaces it: an update // reopens only stale verdicts, so the groups stay accounted for, and the skip stays final rather // than being superseded with the OLD drop. An empty range names nothing. Mirrors Rust. diff --git a/js/net/src/lite/subscriber.ts b/js/net/src/lite/subscriber.ts index 1cb38a4d475..700663e48b2 100644 --- a/js/net/src/lite/subscriber.ts +++ b/js/net/src/lite/subscriber.ts @@ -48,7 +48,7 @@ import { closeUnwanted } from "../util/announce.ts"; import { Deadline, REQUEST_TIMEOUT_MS, requestOptions } from "../util/deadline.ts"; import { abandoned, isAbandoned, unwanted } from "../util/demand.ts"; import { LIMITS, type Limits, SlotMap, Slots } from "../util/limits.ts"; -import { min, UNBOUNDED } from "../util/sequence.ts"; +import { max, min, UNBOUNDED } from "../util/sequence.ts"; import { Budget } from "../util/yield.ts"; import { bounded, hasTime, rangesStart, sameRanges, Window, wireRanges, wireSpan } from "../window.ts"; import { overrideBroadcastWire, wireOf } from "../wire.ts"; @@ -1831,7 +1831,8 @@ export class Subscriber { if (keepsSubscribeOpen(this.version)) void this.#finishTail(stream, entry).catch(() => {}); } else if ("drop" in resp) { // The publisher will never send these groups, so they are accounted for without a - // stream, and a reader of one still arriving ends with the drop's error now. + // stream, and a reader of one still arriving ends with the drop's error now. A lite-07 + // stale drop leaves a group whose stream arrived, since the stream wins over it. const { start, end, update } = resp.drop; const error = resp.drop.cause(this.version); // On lite-07 an OLD drop is a stale verdict, which the next update reopens; every other @@ -1846,7 +1847,8 @@ export class Subscriber { const hole = stale ? staleHole(entry.maxDelay ?? 0) : { error }; const recorded = hooks.dropTrackGroups(entry.track, start, end + 1n, hole); // A final drop is never reopened, even where it replaces a stale one. A sequence - // already skipped stays skipped, so no update reopens it either. + // already skipped stays skipped, and one whose group arrived keeps it, so no update + // reopens either. entry.stale = stale ? recorded.reduce((runs, run) => insertDrop(runs, run.start, run.end, hole), entry.stale ?? []) : removeRun(entry.stale ?? [], start, end + 1n); @@ -2046,10 +2048,18 @@ export class Subscriber { // unknown again: a reader fills them rather than trust an old verdict. entry.update = update.update; entry.maxDelay = update.maxDelay; - hooks.supersedeStale(entry.track, judgedEnd(ranges)); + const held = hooks.supersedeStale(entry.track, judgedEnd(ranges)); // A lite-07 publisher re-offers every group it reported stale on any update, and drops // again under the new number what is still stale, so the tail owes every stale run again. - for (const run of entry.stale ?? []) entry.tail.unaccount(run.start, run.end); + // A group that arrived since its stale drop won over it: the publisher served it, so it + // is neither re-offered nor owed again. + for (const run of entry.stale ?? []) { + for (const reopened of held) { + const start = max(run.start, reopened.start); + const end = min(run.end, reopened.end); + if (start < end) entry.tail.unaccount(start, end); + } + } entry.stale = undefined; // Encoded apart from the write, so a bound the wire can't carry still fails the track. const encoded = await Writer.collect(this.version, (w) => update.encode(w, this.version)); diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts index 9ae70f6ac24..c6b16e8f724 100644 --- a/js/net/src/resume.test.ts +++ b/js/net/src/resume.test.ts @@ -1451,6 +1451,40 @@ test("a held-back stale run new terms supersede is not reported", async () => { } }); +// A group the new route delivers over a stale run the old route reported retires that report, and +// a final drop the new route gives replaces it: only what is still stale is taken back, so an +// update reopens no group already served. Mirrors Rust. +test("a resumed reader retires a stale report its group or final drop replaces", async () => { + const front = new Front(); + const a = new BroadcastProducer(); + const b = new BroadcastProducer(); + const ta = copy(a); + const tb = copy(b); + front.serve(a.consume()); + const sub = front.consumer.track("seq").subscribe({ delay: { max: Milli(60_000) } }); + ta.write(0n); + expect(drain(sub)).toEqual([0n]); + hooks.dropTrackGroups(ta.producer, 1n, 4n, staleHole(60_000)); + expect(hooks.tryDrop(sub)).toMatchObject({ kind: "drop", start: 1n, end: 4n, stale: 60_000 }); + + front.serve(b.consume()); + await settle(); + tb.write(1n); + expect(drain(sub)).toEqual([1n]); + // Reported once the replaced route no longer feeds. + ta.producer.abort(new StreamError(StreamCode.SessionClosed)); + await settle(); + const cut = new StreamError(StreamCode(70), { message: "cut" }); + hooks.dropTrackGroups(tb.producer, 2n, 3n, { error: cut }); + expect(hooks.tryDrop(sub)).toMatchObject({ kind: "drop", start: 2n, end: 3n, stale: undefined }); + // Only group 3 is still stale. + expect(hooks.reofferStale(sub)).toEqual([{ start: 3n, end: 4n }]); + sub.close(); + front.consumer.close(); + a.close(); + b.close(); +}); + test("a drop a replaced route held back is no verdict on the route serving next", async () => { const front = new Front(); const a = new BroadcastProducer(); diff --git a/js/net/src/resume.ts b/js/net/src/resume.ts index 380d0c41d21..a6013542000 100644 --- a/js/net/src/resume.ts +++ b/js/net/src/resume.ts @@ -798,9 +798,9 @@ class Reader implements ResumeSource { if (open) { // Report once without marking it delivered: a late fill may still land. this.#drops = removeRun(this.#drops, open.start, open.end); - // A skip is final, so no update reopens a stale report it replaced. - if (run.skip) this.#reportedStale = removeRun(this.#reportedStale, open.start, open.end); - else if (run.stale !== undefined) insertRun(this.#reportedStale, open.start, open.end); + if (run.stale !== undefined) insertRun(this.#reportedStale, open.start, open.end); + // A skip or final drop is final, so no update reopens a stale report it replaced. + else this.#reportedStale = removeRun(this.#reportedStale, open.start, open.end); return { kind: "drop", ...open, error: run.error, skip: run.skip, stale: run.stale }; } } @@ -1087,6 +1087,11 @@ class Reader implements ResumeSource { #handOut(copy: Copy, group: GroupConsumer): GroupConsumer { this.#newest = max(this.#newest ?? 0n, group.sequence); + // The group won over any stale report on it, whichever copy made it, so no update reopens + // it. + if (this.#reportedStale.length > 0) { + this.#reportedStale = removeRun(this.#reportedStale, group.sequence, group.sequence + 1n); + } // Another copy holding the group, now or once it lands there, is where a failure continues // it, so it stays while the group is read; see #backs. Recorded even with one copy, since a // later route's copy may come to hold it. diff --git a/js/net/src/track.ts b/js/net/src/track.ts index 562f9f50b60..1c1f440334b 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -1018,12 +1018,14 @@ export class Producer { hooks.dropTrackGroups = (producer, start, end, hole) => producer.#drop(start, end, hole, true); hooks.skipTrackGroups = (producer, start, end) => producer.#skipReceived(start, end); hooks.supersedeStale = (producer, judgedEnd) => { + const held = producer.#state.drops.peek().filter((run) => run.stale !== undefined); for (const state of [producer.#state, ...producer.#sinks]) { state.judgedEnd = judgedEnd; const drops = state.drops.peek(); if (drops.some((run) => run.stale !== undefined)) state.drops.set(drops.filter((run) => run.stale === undefined)); } + return held.map(({ start, end }) => ({ start, end })); }; hooks.dropAnchors = (producer) => { for (const state of [producer.#state, ...producer.#sinks]) state.dropsAnchors.set(true); @@ -1127,7 +1129,17 @@ export class Producer { #drop(start: bigint, end: bigint, hole: Hole, open = false): Run[] { if (start >= end) return []; const skipped = hole.skip ? [] : this.#state.drops.peek().filter((run) => run.skip); - const runs = runGaps(skipped, start, end); + let runs = runGaps(skipped, start, end); + // A received stale verdict leaves a group that arrived and was not aborted: a lite-07 + // publisher may serve a group after its stale drop, and the group's stream may land first. + // The group wins, still arriving or not. + if (open && hole.stale !== undefined) { + for (const [sequence, entry] of this.#cached) { + if (sequence >= start && sequence < end && !(entry.group.closed.peek() instanceof Error)) { + runs = removeRun(runs, sequence, sequence + 1n); + } + } + } // A fill no longer owes what the track gave up. this.#fills.dropped(start, end); for (const state of [this.#state, ...this.#sinks]) { @@ -1135,7 +1147,7 @@ export class Producer { runs.reduce((drops, run) => insertDrop(drops, run.start, run.end, hole), drops), ); } - if (open && !hole.skip) { + if (open && !hole.skip && hole.stale === undefined) { for (const [sequence, entry] of this.#cached) { if (sequence >= start && sequence < end && entry.group.closed.peek() === undefined) { entry.group.abort(hole.error); @@ -2499,6 +2511,11 @@ export class Subscriber { if (ordered) this.#delivered.forgetBelow(sequence); this.#delivered.add(sequence); this.#skipped.delete(sequence); + // The group won over any stale report on it, so no update reopens it. A resumed reader + // retires its own. + if (this.#reportedStale.length > 0) { + this.#reportedStale = removeRun(this.#reportedStale, sequence, sequence + 1n); + } if (this.#resume) this.#resume.forgetPastBudget(this.#delivered); else this.#forgetPastBudget(this.#delivered); // What it hands out, and forgets below it, it no longer wants from a fill: tell the fills @@ -3544,18 +3561,21 @@ export class Subscriber { } // The first part of `[from, to)` this cursor has not reported yet, now reported, with why. A - // skip goes out over a drop already reported, since it replaces it. + // skip goes out over a drop already reported, and a final drop over a stale one, since each + // replaces what it goes out over. #report(from: bigint, to: bigint, hole: Hole): (Recv & { kind: "drop" }) | undefined { const { start, end } = this.#cursor.peek(); + const final = !hole.skip && hole.stale === undefined; for (const run of this.#window.runs(max(from, start), min(to, end ?? to))) { - const [gap] = runGaps(hole.skip ? this.#reportedSkip : this.#reported, run.start, run.end); + let blocked = hole.skip ? this.#reportedSkip : this.#reported; + if (final) for (const stale of this.#reportedStale) blocked = removeRun(blocked, stale.start, stale.end); + const [gap] = runGaps(blocked, run.start, run.end); if (!gap) continue; insertRun(this.#reported, gap.start, gap.end); - if (hole.skip) { - // Final, so no update reopens a stale report it replaced. - insertRun(this.#reportedSkip, gap.start, gap.end); - this.#reportedStale = removeRun(this.#reportedStale, gap.start, gap.end); - } else if (hole.stale !== undefined) insertRun(this.#reportedStale, gap.start, gap.end); + if (hole.skip) insertRun(this.#reportedSkip, gap.start, gap.end); + if (hole.stale !== undefined) insertRun(this.#reportedStale, gap.start, gap.end); + // A skip or final drop is final, so no update reopens a stale report it replaced. + else this.#reportedStale = removeRun(this.#reportedStale, gap.start, gap.end); return { kind: "drop", ...gap, error: hole.error, skip: hole.skip, stale: hole.stale }; } return undefined; diff --git a/quest/m1/drop-set.md b/quest/m1/drop-set.md index 62255a664e4..c48a4fdacfb 100644 --- a/quest/m1/drop-set.md +++ b/quest/m1/drop-set.md @@ -122,6 +122,25 @@ decision above and the PR's `## Waiting` item on out-of-order creation: draft requires no more. A received SKIP or final DROP also takes its groups out of the subscriber's stale record, so the next update doesn't owe them again. Not chosen: SKIP replacing only a stale OLD drop. +- ✅ **A group or final drop replaces a lite-07 stale drop already sent** + (agent decision from review, 2026-11-04, `/quest-iterate`; the maintainer + chose to keep it and retire the stale report at the 2026-11-03 round-4 + checkpoint, `/quest-complete`). A front that switches routes can hand out + the new route's group, or its final verdict, for a sequence the old route + reported stale, before the update that reopens it, so the publisher serves + the group, or sends the final drop, over the OLD it sent. Its own expired + streams' OLD drops count the same. Whatever replaces a stale report + retires it: a group handed out, a SKIP, or a final drop takes the sequence + out of the reader's stale record (`DropCursor::reported_stale`, the + resumed `Reader::reported_stale`, JS `#reportedStale`) and the publisher's + (`TrackRun::dropped_stale`, `expired`), so the next SUBSCRIBE_UPDATE + neither sends OLD for it again nor names it at the end. On the receiving + side a Group Stream wins over a stale drop of its sequence in either order: + an OLD that lands after the stream leaves the group, still arriving or + not, and an update owes again only the stale runs the track still holds. + The end sweep's NOT_FOUND never replaces a stale drop, so an update can + still reach an ended track's history. Not chosen: reverting the + replacement; merging with the stale report left in place. - ✅ **`append_group` is an implicit skip** (refinement, same session): it means "the next group", so the producer declares it won't go back, and it skips every sequence below it that never arrived and wasn't already given diff --git a/rs/moq-net/src/lite/publisher.rs b/rs/moq-net/src/lite/publisher.rs index d6bdb324332..8c28d65ee0b 100644 --- a/rs/moq-net/src/lite/publisher.rs +++ b/rs/moq-net/src/lite/publisher.rs @@ -4720,6 +4720,7 @@ impl TrackRun { && let Some(end) = sequence.checked_add(1) { self.expired.insert(sequence..end); + self.sent_stale(sequence..end); } Hole::stale(self.track.budget()) } @@ -4734,6 +4735,7 @@ impl TrackRun { for run in std::mem::take(&mut self.redrop).iter() { writer.buffer(&self.hole(run.start..=run.end - 1, &hole))?; self.expired.insert(run.clone()); + self.sent_stale(run.clone()); } return Poll::Ready(Ok(ControlFlow::Continue(()))); } @@ -4832,8 +4834,7 @@ impl TrackRun { if self.dropped_stale.contains(&sequence) && let Some(end) = sequence.checked_add(1) { - self.dropped_stale.remove(sequence..end); - self.dropped.remove(sequence..end); + self.replace_stale(sequence..end); } else if self.served.as_ref().is_some_and(|served| served.contains(&sequence)) { return Poll::Ready(Ok(ControlFlow::Continue(()))); } @@ -4978,21 +4979,44 @@ impl TrackRun { hole: &Hole, writer: &mut Writer, ) -> Result { - if self.ctx.version.has_skip() - && let Some(served) = &mut self.served - { + if self.ctx.version.has_skip() && self.served.is_some() { let replaces = match hole { Hole::Skip => Some(&self.dropped), Hole::Drop { stale: None, .. } => Some(&self.dropped_stale), Hole::Drop { .. } => None, }; - for run in replaces.into_iter().flat_map(|runs| runs.overlapping(&range)) { - served.remove(run.start.max(range.start)..run.end.min(range.end)); + let replaced: Vec<_> = replaces + .into_iter() + .flat_map(|runs| runs.overlapping(&range)) + .map(|run| run.start.max(range.start)..run.end.min(range.end)) + .collect(); + for run in replaced { + self.replace_stale(run.clone()); + if let Some(served) = &mut self.served { + served.remove(run); + } } } self.drop_range(range, hole, writer) } + /// Record `run` as named stale (OLD) in a lite-07 drop, which a later verdict or group + /// replaces; see [`Self::dropped_stale`]. + fn sent_stale(&mut self, run: Range) { + self.dropped.insert(run.clone()); + self.dropped_stale.insert(run); + } + + /// Forget that `run` went out as a drop: a verdict or group replaces it. A stream of this + /// subscription that expired there is no longer owed again on an update, since what + /// replaced it settles it. + fn replace_stale(&mut self, run: Range) { + self.dropped.remove(run.clone()); + self.dropped_stale.remove(run.clone()); + self.expired.remove(run.clone()); + self.redrop.remove(run); + } + /// Give up, as `hole` says, whatever part of `range` this subscription has not been /// served or given up yet. Returns whether any was buffered. fn drop_range( @@ -7146,6 +7170,89 @@ mod serve_group_test { } } + /// A group that wins over a lite-07 stale drop already sent retires it: the next update + /// neither sends the drop again nor reopens the served group, and the end names nothing + /// for it. Here a relay's upstream reported group 1 stale, then delivered it, as a front + /// that switched routes does. + #[kio::test] + async fn a_group_served_over_a_stale_drop_retires_it() { + let budget = Duration::from_millis(1500); + let mut track = track::Producer::new( + Arc::new(broadcast::Info::default()), + "test", + crate::track::Info::timed(), + ); + let subscriber = + track.subscribe(track::Subscription::default().with_delay(track::Delay::default().with_max(budget))); + let (mut run, mut writer, log) = lite07_run(SinkSession::new(Log::default()), subscriber); + write_group(&mut track, 0, 0); + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + track.give_up(1..2, Hole::stale(budget)); + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + assert_eq!(drop_updates(&mut writer, &log, Version::Lite07).await, [(1, 1, 0)]); + write_group(&mut track, 1, 1000); + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + assert_eq!(run.ctx.outcomes.opened.load(Ordering::Relaxed), 2, "the group wins"); + + run.update(lite::SubscribeUpdate { + wanted_group: None, + priority: 1, + max_delay: budget, + min_delay: None, + ranges: vec![lite::Range::span(None, 0, None, None)], + order: Order::Descending, + update: 1, + }); + write_group(&mut track, 2, 2000); + track.finish().unwrap(); + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + assert_eq!( + run.ctx.outcomes.opened.load(Ordering::Relaxed), + 3, + "group 1 is not served again" + ); + assert_eq!( + drop_updates(&mut writer, &log, Version::Lite07).await, + [(1, 1, 0)], + "no second OLD, and no NOT_FOUND at the end" + ); + } + + /// A lite-07 stream that expired as stale is a stale drop like any other: a final drop + /// the reader reaches for it later replaces it, and no update drops it again then. + #[kio::test] + async fn a_final_drop_replaces_an_expired_stream_on_lite07() { + let mut track = track::Producer::new( + Arc::new(broadcast::Info::default()), + "test", + crate::track::Info::timed(), + ); + let subscriber = track.subscribe(None); + let (mut run, mut writer, log) = lite07_run(SinkSession::new(Log::default()), subscriber); + write_group(&mut track, 0, 0); + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + // Group 1's stream expired, as a blocked transport would leave it. + run.ctx.outcomes.dropped.lock().unwrap().push((1, StreamError::Old)); + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + run.forward_range(1..2, &Hole::drop(StreamError::Cancel), &mut writer) + .unwrap(); + run.update(lite::SubscribeUpdate { + wanted_group: None, + priority: 1, + max_delay: Duration::from_secs(3600), + min_delay: None, + ranges: vec![lite::Range::span(None, 0, None, None)], + order: Order::Descending, + update: 1, + }); + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + assert_eq!( + drop_updates(&mut writer, &log, Version::Lite07).await, + [(1, 1, 0), (1, 1, 0)], + "the final drop replaces the OLD, and the update drops nothing again" + ); + } + /// A stale group already named in SUBSCRIBE_DROP stays dropped before lite-07 after any /// SUBSCRIBE_UPDATE, since those subscribers settle a drop for good. Lite-07 reopens it on /// every update: one that widens the max age past it serves it, and one that keeps the diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index 7d6a8f18ef1..9ed92af0061 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -4698,6 +4698,63 @@ mod tests { ); } + /// A group that arrives over a lite-07 stale (OLD) drop wins over it, whichever lands first: + /// the publisher serves a group after its stale drop, as a front that switched routes does, + /// and the group's stream may overtake the drop. The next update owes it no more, and a + /// group still arriving is never aborted by the drop. + #[kio::test] + async fn a_group_wins_over_its_stale_drop() { + let version = Version::Lite07; + let (serve, mut serving, tail, _broadcast) = lite07_serving(Duration::from_millis(500)).await; + let now = || serve.subscriber.runtime.now(); + let code = u64::from(crate::coding::StreamCodes::encode_stream_code( + &version, + &crate::StreamError::Old, + )); + let old = |sequence| { + lite::SubscribeResponse::Drop(lite::SubscribeDrop { + start: sequence, + end: sequence, + error: code, + update: 0, + }) + }; + let frame = |group: &mut crate::group::Producer| { + group.write_frame(crate::Timestamp::from_millis(1).unwrap(), &b"x"[..])?; + group.finish() + }; + let mut arrive = |serving: &mut ServeLoop, sequence| { + let group = serving.serving.create_group(crate::group::Info { sequence }).unwrap(); + (group, Reading::open(&tail, Some(sequence), now())) + }; + + // OLD, then the group, which finishes. + serving.apply_response(&serve, old(1)).unwrap(); + let (mut one, reading) = arrive(&mut serving, 1); + frame(&mut one).unwrap(); + drop(reading); + // The group first, still arriving when its OLD lands. + let (mut two, _two) = arrive(&mut serving, 2); + serving.apply_response(&serve, old(2)).unwrap(); + // OLD, then the group, still arriving at the update. + serving.apply_response(&serve, old(3)).unwrap(); + let (mut three, _three) = arrive(&mut serving, 3); + // OLD, and no group. + serving.apply_response(&serve, old(4)).unwrap(); + + { + let Sub::Active(active) = &mut serving.sub else { + unreachable!("an active subscription"); + }; + buffer_update(active, &serving.serving, now()).unwrap(); + } + let covers = |groups: std::ops::Range| tail.read().covers(groups); + assert!(covers(1..4), "every group that arrived stays accounted for"); + assert!(!covers(4..5), "the group that never arrived is owed again"); + frame(&mut two).expect("the drop leaves a group still arriving"); + frame(&mut three).expect("the update leaves a group still arriving"); + } + /// A SKIP, or any final drop, that follows an OLD drop of the same groups replaces it: an /// update reopens only stale verdicts, so the groups stay accounted for, and the skip /// stays final rather than being superseded with the OLD drop. An empty range names @@ -9026,14 +9083,17 @@ fn judged_end(ranges: &[lite::Range]) -> Option { /// /// A lite-07 publisher re-offers every group it reported stale on any update, and drops /// again under the new number what is still stale, so the tail owes every stale run again, -/// from `now`. +/// from `now`. A group that arrived since its stale drop won over it: the publisher served +/// it, so it is neither re-offered nor owed again. fn buffer_update(active: &mut SubStream, serving: &track::Producer, now: crate::time::Instant) -> Result<(), Error> { active.update += 1; - serving.supersede_stale(judged_end(&active.ranges)); - let reopened = std::mem::take(&mut active.stale); + let held = serving.supersede_stale(judged_end(&active.ranges)); + let recorded = std::mem::take(&mut active.stale); if let Ok(mut tail) = active.tail.write() { - for run in reopened.iter() { - tail.unaccount(run.clone(), now); + for run in recorded.iter() { + for reopened in held.overlapping(run) { + tail.unaccount(reopened.start.max(run.start)..reopened.end.min(run.end), now); + } } } active.stream.writer.buffer(&lite::SubscribeUpdate { @@ -10242,7 +10302,9 @@ impl ServeLoop { } // The publisher will never send these groups, so they // are accounted for without a stream, and a reader of - // one still arriving ends with the drop's error now. + // one still arriving ends with the drop's error now. A + // lite-07 stale drop leaves a group whose stream arrived, + // since the stream wins over it. lite::SubscribeResponse::Drop(dropped) => { let groups = dropped.start..dropped.end.saturating_add(1); // An empty range (Group Start = Group End + 1) names nothing. @@ -10262,7 +10324,8 @@ impl ServeLoop { } match stale { // Judged under the budget this subscriber sent with that update. A - // sequence already skipped stays skipped, so no update reopens it. + // sequence already skipped stays skipped, and one whose group arrived + // keeps it, so no update reopens either. true => { let recorded = self.serving.give_up(groups, track::Hole::stale(active.max_delay)); active.stale.extend(recorded); diff --git a/rs/moq-net/src/model/resume.rs b/rs/moq-net/src/model/resume.rs index a95f19db22c..bbe43bcf8b0 100644 --- a/rs/moq-net/src/model/resume.rs +++ b/rs/moq-net/src/model/resume.rs @@ -1419,6 +1419,11 @@ impl Reader { fn hand_out(&mut self, index: usize, group: group::Consumer) -> group::Consumer { let sequence = group.sequence; self.newest = self.newest.max(Some(sequence)); + // The group won over any stale report on it, whichever copy made it, so no update + // reopens it. + if let Some(end) = sequence.checked_add(1) { + self.reported_stale.remove(sequence..end); + } // Another copy holding the group, now or once it lands there, is where a failure // continues it, so it stays while the group is read; see [`Copy::retains`]. Recorded // even with one copy, since a later route's copy may come to hold it. @@ -1840,10 +1845,9 @@ impl Reader { // up) may still land, and is handed out if it does. self.drops.remove(open.clone()); match &err { - // Final, so no update reopens a stale report it replaced. - track::Hole::Skip => self.reported_stale.remove(open.clone()), err if err.stale_budget().is_some() => self.reported_stale.insert(open.clone()), - track::Hole::Drop { .. } => {} + // A skip or final drop is final, so no update reopens a stale report it replaced. + track::Hole::Skip | track::Hole::Drop { .. } => self.reported_stale.remove(open.clone()), } Poll::Ready((open, err)) } @@ -3661,6 +3665,30 @@ mod test { assert_eq!(drain(&mut sub, false), [1, 2]); } + /// A group the new route delivers over a stale run the old route reported retires that + /// report, and a final drop that follows it replaces it: only what is still stale is taken + /// back, so an update reopens no group already served. + #[test] + fn a_resumed_reader_retires_a_stale_report_its_group_or_final_drop_replaces() { + let routes = Producer::new(); + let logical = logical(&routes); + let a = copy(); + routes.serve(a.consume()); + let mut sub = subscribe(&logical, HOUR); + write(&a, 0); + assert_eq!(drain(&mut sub, false), [0]); + a.give_up(1..4, track::Hole::stale(HOUR)); + assert_eq!(dropped(&mut sub), [(1..4, StreamError::Old)]); + a.drop_groups(2..3, StreamError::App(5)); + assert_eq!(dropped(&mut sub), [(2..3, StreamError::App(5))]); + + let b = copy(); + routes.serve(b.consume()); + write(&b, 1); + assert_eq!(drain(&mut sub, false), [1]); + assert_eq!(sub.reoffer_stale(), vec![3..4], "only group 3 is still stale"); + } + /// A run a replaced route held back is dropped with that route: the route serving next /// may still deliver it. #[test] diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index de9330f1768..0242616b717 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -1213,16 +1213,17 @@ impl TrackState { } /// Forget every run held as stale; see [`Producer::supersede_stale`]. - fn supersede_stale(&mut self) { - let stale: Vec<_> = self + fn supersede_stale(&mut self) -> RangeSet { + let stale: RangeSet = self .drops .iter() .filter(|(_, hole)| hole.stale_budget().is_some()) .map(|(run, _)| run.clone()) .collect(); - for run in stale { - self.change_drops(run, None); + for run in stale.iter() { + self.change_drops(run.clone(), None); } + stale } /// Keep the change index aligned with the range map, including the neighbours an @@ -3672,16 +3673,20 @@ impl Producer { /// Forget every run held as stale: the terms a received copy's upstream judged them /// under were replaced, so they are unknown again, and a reader that wants one fills it. /// The new terms' ranges end at `judged_end`, `None` past the live edge or unknown. + /// Returns the runs forgotten; a group that arrived since its stale drop replaced it, so + /// it is not among them. #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] - pub(crate) fn supersede_stale(&self, judged_end: Option) { - if let Ok(mut state) = self.modify() { - state.supersede_stale(); - state.judged_end = judged_end; - } + pub(crate) fn supersede_stale(&self, judged_end: Option) -> RangeSet { + let Ok(mut state) = self.modify() else { + return RangeSet::new(); + }; + state.judged_end = judged_end; + state.supersede_stale() } /// Record `groups` given up as `hole` describes, returning the runs recorded: a drop - /// leaves a skipped sequence skipped. + /// leaves a skipped sequence skipped, and a stale verdict leaves a group that arrived and + /// was not aborted, since a group stream wins over its drop. pub(crate) fn give_up(&self, groups: ops::Range, hole: Hole) -> RangeSet { let error = match &hole { Hole::Drop { error, .. } => error.clone(), @@ -3699,18 +3704,36 @@ impl Producer { } let (runs, open): (_, Vec) = match self.modify() { Ok(mut state) => { - let runs = state.drop_groups(groups.clone(), hole); // Only the live feed's copy: a drop judges the subscription, so a fetched head // or fetch-only backfill may still complete. - let open = state + let open: Vec<_> = state .lookup - .range(groups) - .filter(|(_, slot)| slot.visible) - .map(|(_, slot)| &slot.group) - .filter(|group| !group.is_aborted()) - .cloned() + .range(groups.clone()) + .filter(|(_, slot)| slot.visible && !slot.group.is_aborted()) + .map(|(&sequence, slot)| (sequence, slot.group.clone())) .collect(); - (runs, open) + match hole.stale_budget() { + // A lite-07 publisher may serve a group after its stale drop, and the group's + // stream may land first: the group wins, still arriving or not. + Some(_) => { + let mut left = RangeSet::new(); + left.insert(groups); + for (sequence, _) in &open { + if let Some(end) = sequence.checked_add(1) { + left.remove(*sequence..end); + } + } + let mut runs = RangeSet::new(); + for run in left.iter() { + runs.extend(state.drop_groups(run.clone(), hole.clone()).iter().cloned()); + } + (runs, Vec::new()) + } + None => ( + state.drop_groups(groups, hole), + open.into_iter().map(|(_, group)| group).collect(), + ), + } } Err(_) => return RangeSet::new(), }; @@ -6820,6 +6843,14 @@ impl DropCursor { stale.into_iter().collect() } + /// Retire a stale report on `sequence` once its group is handed out: the group won over + /// the drop, so no update reopens it. + fn retire(&mut self, sequence: u64) { + if let Some(end) = sequence.checked_add(1) { + self.reported_stale.remove(sequence..end); + } + } + /// Record a run this cursor's fills gave up on as a miss, as `hole` says; a reason /// already pending, such as the track's own drop, is kept. fn give_up(&mut self, run: ops::Range, hole: Hole) { @@ -6904,12 +6935,18 @@ impl DropCursor { if clipped.end < range.end { self.pending.insert(clipped.end..range.end, pending.clone()); } - // A skip goes out over a drop already reported, since it replaces it. - let reported = match pending.hole { - Hole::Skip => &self.reported_skip, - Hole::Drop { .. } => &self.reported, + // A skip goes out over a drop already reported, and a final drop over a stale one, + // since each replaces what it goes out over. + let mut blocked = match pending.hole { + Hole::Skip => self.reported_skip.overlapping(&clipped).cloned().collect(), + Hole::Drop { .. } => self.reported.overlapping(&clipped).cloned().collect::>(), }; - if let Some(gap) = reported.gaps(&clipped).next() { + if matches!(pending.hole, Hole::Drop { stale: None, .. }) { + for stale in self.reported_stale.overlapping(&clipped) { + blocked.remove(stale.clone()); + } + } + if let Some(gap) = blocked.gaps(&clipped).next() { if gap.end < range.end { self.pending.insert(gap.end..range.end, pending.clone()); } @@ -6922,7 +6959,7 @@ impl DropCursor { self.reported_stale.remove(gap.clone()); } hole if hole.stale_budget().is_some() => self.reported_stale.insert(gap.clone()), - Hole::Drop { .. } => {} + Hole::Drop { .. } => self.reported_stale.remove(gap.clone()), } return Some((gap, hole)); } @@ -8692,6 +8729,12 @@ impl Subscriber { cursor.prune_skipped(); } self.delivered.insert(sequence); + // The group won over any stale report on it; a resumed reader retires its own. + if let Inner::Plain(cursor) = &mut self.inner + && let Some(drops) = &mut cursor.drops + { + drops.retire(sequence); + } let mut delivered = std::mem::take(&mut self.delivered); self.forget_past_budget(&mut delivered); self.delivered = delivered; From e4c0ee2143a95e9fb1d671d06de2dd8727c0d35c Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 23:39:04 -0800 Subject: [PATCH 34/68] style(net): drop an unused mut in a test Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/src/lite/subscriber.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index 9ed92af0061..d8adc564be4 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -4723,7 +4723,7 @@ mod tests { group.write_frame(crate::Timestamp::from_millis(1).unwrap(), &b"x"[..])?; group.finish() }; - let mut arrive = |serving: &mut ServeLoop, sequence| { + let arrive = |serving: &mut ServeLoop, sequence| { let group = serving.serving.create_group(crate::group::Info { sequence }).unwrap(); (group, Reading::open(&tail, Some(sequence), now())) }; From 0fee3fd91ddb65ac12dcbfe938bc9091577bcb95 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 23:53:42 -0800 Subject: [PATCH 35/68] fix(js): close the groups an update re-offered when the subscription ends Each re-offer is a read of its own, which closing the track leaves open, so a subscriber that left before the loop served them kept their mirrors buffering frames. Co-Authored-By: Claude Opus 5.5 --- js/net/src/lite/publisher.ts | 2 ++ 1 file changed, 2 insertions(+) diff --git a/js/net/src/lite/publisher.ts b/js/net/src/lite/publisher.ts index 9ff55425981..0b6df39f012 100644 --- a/js/net/src/lite/publisher.ts +++ b/js/net/src/lite/publisher.ts @@ -1770,6 +1770,8 @@ export class Publisher { } } finally { if (!finished) unsubscribe(); + // An update's re-offers are reads of their own, which closing the track leaves open. + for (const group of reoffered.splice(0)) group.close(); priority.close(); } } From 8c2f1d9e03bca623370bf7268ee181fe8c0a3009 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 21:46:02 -0800 Subject: [PATCH 36/68] fix(net): address lite-07 drop-set review findings - A fill cannot cache a group at a skipped sequence (Error::Skipped). - A final OLD drop, as relayed from a lite-06 upstream, goes out as the stale drop the lite-07 wire names it, and each update drops it again. - An update judges the re-offers still waiting for stream credit again, and a verdict sent since keeps a queued re-offer from opening. - A SUBSCRIBE_DROP naming an update never sent is a protocol violation, in Rust, JS and the draft. - JS skipGroups checks its bounds, and insertDatagram refuses a skipped sequence. Co-Authored-By: Claude Opus 5.5 --- drafts/draft-lcurley-moq-lite.md | 1 + js/net/src/lite/publisher.test.ts | 36 ++++++++ js/net/src/lite/publisher.ts | 24 +++++- js/net/src/lite/subscriber.test.ts | 35 ++++++++ js/net/src/lite/subscriber.ts | 3 + js/net/src/track.test.ts | 3 + js/net/src/track.ts | 6 +- rs/moq-net/src/lite/publisher.rs | 131 ++++++++++++++++++++++++++--- rs/moq-net/src/lite/subscriber.rs | 25 ++++++ rs/moq-net/src/model/track.rs | 18 +++- rs/moq-net/src/model/track/fill.rs | 4 +- 11 files changed, 267 insertions(+), 19 deletions(-) diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index 3ce65d0756c..9ab498fc8f6 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -1686,6 +1686,7 @@ A stale (OLD) drop is the exception: it ends no group whose Group Stream arrived **Update**: The [SUBSCRIBE](#subscribe) or [SUBSCRIBE_UPDATE](#subscribe-update) `Update` in effect when the publisher gave the groups up. +A subscriber MUST treat an `Update` larger than the last one it sent as a PROTOCOL_VIOLATION. A drop whose Error Code is OLD is a stale verdict, which holds only under the update it names. The subscriber judges it against the `Subscriber Max Age` it sent with that update, from its own record: the drop does not carry the budget. diff --git a/js/net/src/lite/publisher.test.ts b/js/net/src/lite/publisher.test.ts index 188aee511db..44f8b5d1ac4 100644 --- a/js/net/src/lite/publisher.test.ts +++ b/js/net/src/lite/publisher.test.ts @@ -1602,6 +1602,42 @@ test("lite draft-07: a final drop replaces an expired stream", async () => { } }); +// A drop the reader gave up as final with OLD, as one relayed from a lite-06 upstream is, goes out as +// a stale drop on lite-07, which the subscriber reopens on its next update, so each update drops it +// again rather than leave it owed. Mirrors the Rust +// `a_final_old_drop_is_dropped_again_by_each_update_on_lite07`. +test("lite draft-07: a final OLD drop is dropped again by each update", async () => { + const sub = await servedSubscription({ version: Version.DRAFT_07, startGroup: 0n }); + try { + sub.serve(0n); + expect(await sub.servedSequence()).toBe(0n); + expect("start" in (await decodeSubscribeResponse(sub.client.reader, Version.DRAFT_07))).toBe(true); + + hooks.dropTrackGroups(sub.track, 1n, 2n, { error: new StreamError(StreamCode.Old) }); + const first = await decodeSubscribeResponse(sub.client.reader, Version.DRAFT_07); + if (!("drop" in first)) throw new Error("expected SUBSCRIBE_DROP"); + expect([first.drop.start, first.drop.end, first.drop.update]).toEqual([1n, 1n, 0n]); + + for (const update of [1n, 2n]) { + await replayUpdate({ + priority: 1, + startGroup: 0n, + update, + ...restate(Version.DRAFT_07, update - 1n), + }).encode(sub.client.writer, Version.DRAFT_07); + await flush(); + } + sub.track.finish(); + sub.track.close(); + expect(await holesToEnd(sub.client.reader)).toEqual([ + ["drop", 1n, 1n, 1n], + ["drop", 1n, 1n, 2n], + ]); + } finally { + await sub.close(); + } +}); + // An update that lowers the start below SUBSCRIBE_START asks for those groups too, so the // end sweep drops them, as the Rust publisher does. test("lite draft-06: an update lowering the start is swept at the end", async () => { diff --git a/js/net/src/lite/publisher.ts b/js/net/src/lite/publisher.ts index fe03576220a..80daeffe1c0 100644 --- a/js/net/src/lite/publisher.ts +++ b/js/net/src/lite/publisher.ts @@ -1391,6 +1391,9 @@ export class Publisher { ? { skip: new SubscribeSkip({ start, end }) } : { drop: SubscribeDrop.given(start, end, hole.error, this.version, updateNumber) }; + // Whether lite-07 names `hole` stale: an OLD drop, which the next update reopens. + const sentStale = (hole: Hole) => !hole.skip && toStreamCode(hole.error) === StreamCode.Old; + // Lite-07: the part of `served` named in SUBSCRIBE_DROP, never served. A skip replaces a // drop, so it is sent over one of these, but never over a group served. let dropped: Run[] = []; @@ -1413,9 +1416,16 @@ export class Publisher { droppedStale = removeRun(droppedStale, gap.start, gap.end); continue; } - if (hole.stale === undefined) droppedStale = removeRun(droppedStale, gap.start, gap.end); - else insertRun(droppedStale, gap.start, gap.end); insertRun(dropped, gap.start, gap.end); + // The wire names every OLD drop stale, which the next update reopens. One the + // reader gave up as final, such as one relayed from a version whose drops are all + // final, is dropped again by each update, as an expired stream is. + if (!sentStale(hole)) { + droppedStale = removeRun(droppedStale, gap.start, gap.end); + continue; + } + insertRun(droppedStale, gap.start, gap.end); + if (hole.stale === undefined) insertRun(expired, gap.start, gap.end); } } for (const gap of gaps) { @@ -1427,11 +1437,11 @@ export class Publisher { }; // Forward a verdict the reader reached on `[start, end)`, as `drop` does. On lite-07 it also - // replaces what it overrides: a skip any drop already sent, and a final drop a stale one, + // replaces what it overrides: a skip any drop already sent, and a final (non-OLD) drop a stale one, // which the next update would otherwise reopen. A reader the earlier drop ended stays ended. const forward = async (start: bigint, end: bigint, hole: Hole): Promise => { if (served && hasSkip(this.version)) { - const replaces = hole.skip ? dropped : hole.stale === undefined ? droppedStale : []; + const replaces = hole.skip ? dropped : sentStale(hole) ? [] : droppedStale; for (const run of [...replaces]) { const from = max(run.start, start); const to = min(run.end, end); @@ -1615,6 +1625,12 @@ export class Publisher { pendingDrops.length, ...pendingDrops.filter((d) => toStreamCode(d.error) !== StreamCode.Old), ); + // A group re-offered that has not opened yet is judged again too, since these + // terms may no longer admit it. + for (const group of reoffered.splice(0)) { + insertRun(expired, group.sequence, group.sequence + 1n); + group.close(); + } // A group whose stream already reset as stale: one still cached, in range, and // fresh under the new budget is served again; the rest is dropped again, by run. for (const run of expired.splice(0)) { diff --git a/js/net/src/lite/subscriber.test.ts b/js/net/src/lite/subscriber.test.ts index 196901266f9..cf3ce397f3c 100644 --- a/js/net/src/lite/subscriber.test.ts +++ b/js/net/src/lite/subscriber.test.ts @@ -3000,6 +3000,41 @@ test("an update whose bound the wire can't carry fails the track", async () => { } }); +// A SUBSCRIBE_DROP can only echo an update the subscriber sent: one naming a later update is a +// protocol violation. Mirrors the Rust `a_drop_naming_an_unsent_update_is_a_protocol_violation`. +test("a lite-07 drop naming an unsent update closes the session", async () => { + const version = Version.DRAFT_07; + const { quic, streams, opened, closes } = fakeSession(); + const subscriber = new Subscriber(quic, version, HopSchema.parse(1n)); + const broadcast = subscriber.consume(Path.from("room")); + const track = broadcast.track("video").subscribe(); + try { + await opened(0); + await answerTrackInfo(streams[0], version); + const stream = await opened(1); + await stream.reading; + const chunks: Uint8Array[] = []; + const writer = new Writer( + new WritableStream({ write: (chunk) => void chunks.push(new Uint8Array(chunk)) }), + version, + ); + await encodeSubscribeResponse(writer, { start: new SubscribeStart([0n]) }, version); + await encodeSubscribeResponse( + writer, + { drop: new SubscribeDrop({ start: 1n, end: 1n, error: StreamCode.Old, update: 1n }) }, + version, + ); + for (const chunk of chunks) stream.inbound.enqueue(chunk); + expect(await track.closed).toBeInstanceOf(ProtocolViolation); + await drainReactions(); + expect(closes.map((info) => info?.closeCode)).toEqual([SessionCode.ProtocolViolation]); + } finally { + track.close(); + broadcast.close(); + subscriber.close(); + } +}); + test("an unknown subscribe response type closes the session", async () => { const { quic, streams, opened, closes } = fakeSession(); const subscriber = new Subscriber(quic, Version.DRAFT_07, HopSchema.parse(1n)); diff --git a/js/net/src/lite/subscriber.ts b/js/net/src/lite/subscriber.ts index 524b24afcbc..84906c63201 100644 --- a/js/net/src/lite/subscriber.ts +++ b/js/net/src/lite/subscriber.ts @@ -1928,6 +1928,9 @@ export class Subscriber { // stream, and a reader of one still arriving ends with the drop's error now. A lite-07 // stale drop leaves a group whose stream arrived, since the stream wins over it. const { start, end, update } = resp.drop; + // It can only echo an update this subscriber sent. + if (update > (entry.update ?? 0n)) + throw new ProtocolViolation(`SUBSCRIBE_DROP names update ${update}, which was never sent`); const error = resp.drop.cause(this.version); // On lite-07 an OLD drop is a stale verdict, which the next update reopens; every other // drop is final, whatever update it names. diff --git a/js/net/src/track.test.ts b/js/net/src/track.test.ts index 1f6c03c3df9..af3b667d8a0 100644 --- a/js/net/src/track.test.ts +++ b/js/net/src/track.test.ts @@ -4993,6 +4993,9 @@ test("creating a skipped sequence throws Skipped", () => { producer.skipGroups(5n, 7n); expect(() => producer.createGroup(6n)).toThrow(Skipped); + expect(() => producer.insertDatagram(6n, undefined, new Uint8Array())).toThrow(Skipped); + // No skip below group 0. + expect(() => producer.skipGroups(-1n, 1n)).toThrow(RangeError); // The next sequence, 5, was skipped too. expect(() => producer.appendGroup()).toThrow(Skipped); // A later drop leaves a skipped sequence skipped. diff --git a/js/net/src/track.ts b/js/net/src/track.ts index 6f78bd262f6..4a45bcb2b2d 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -1842,6 +1842,8 @@ export class Producer { * cached keeps what it is, since it was produced after all. Mirrors the Rust `skip_groups`. */ skipGroups(start: bigint, end: bigint): void { + checkSequence(start); + checkSequence(end); this.#skipReceived(start, end); } @@ -1996,9 +1998,11 @@ export class Producer { * * Preserves the supplied sequence (later appends continue past it) so a relay can * forward a datagram without renumbering it. The size and timestamp rules of - * {@link appendDatagram} apply. Most origin publishers want {@link appendDatagram} instead. + * {@link appendDatagram} apply, and a skipped sequence throws {@link Skipped}. Most origin + * publishers want {@link appendDatagram} instead. */ insertDatagram(sequence: bigint, timestamp: Timestamp | undefined, payload: Uint8Array) { + this.#refuseSkipped(checkSequence(sequence)); this.#insertDatagram({ sequence, timestamp, payload }); } diff --git a/rs/moq-net/src/lite/publisher.rs b/rs/moq-net/src/lite/publisher.rs index bd343715e6b..943884880aa 100644 --- a/rs/moq-net/src/lite/publisher.rs +++ b/rs/moq-net/src/lite/publisher.rs @@ -4873,7 +4873,13 @@ impl TrackRun { // A group whose stream already reset as stale is judged again under the new terms: // one still cached, in range, and fresh under the new budget is served again. The // rest is dropped again, by run, on the next poll. A run an earlier update left to - // drop again is judged again too, since these terms may serve it. + // drop again is judged again too, since these terms may serve it, and so is a group + // one re-offered that has not opened yet, since they may no longer admit it. + for group in std::mem::take(&mut self.reoffered) { + if let Some(end) = group.sequence.checked_add(1) { + self.expired.insert(group.sequence..end); + } + } let pending = std::mem::take(&mut self.redrop); self.expired.extend(pending.iter().cloned()); for run in std::mem::take(&mut self.expired).iter() { @@ -5135,11 +5141,15 @@ impl TrackRun { return Poll::Ready(Ok(ControlFlow::Continue(()))); } - // A group an update re-offers takes the next stream credit, ahead of new ones. + // A group an update re-offers takes the next stream credit, ahead of new ones, unless a + // verdict sent since accounted for it. if self.opening.is_none() && let Some(group) = self.reoffered.pop_front() { let sequence = group.sequence; + if self.served.as_ref().is_some_and(|served| served.contains(&sequence)) { + return Poll::Ready(Ok(ControlFlow::Continue(()))); + } match position_group(group, &self.frames) { Some(group) => self.serve(group), None => { @@ -5307,8 +5317,8 @@ impl TrackRun { } /// Forward a verdict the cursor reached on `range`, as [`Self::drop_range`] does. On lite-07 - /// it also replaces what it overrides: a skip any drop already sent, and a final drop a stale - /// one, which the next update would otherwise reopen. A reader the earlier drop ended stays + /// it also replaces what it overrides: a skip any drop already sent, and a drop the wire + /// names final (any code but OLD) a stale one, which the next update would otherwise reopen. A reader the earlier drop ended stays /// ended. fn forward_range( &mut self, @@ -5319,8 +5329,8 @@ impl TrackRun { if self.ctx.version.has_skip() && self.served.is_some() { let replaces = match hole { Hole::Skip => Some(&self.dropped), - Hole::Drop { stale: None, .. } => Some(&self.dropped_stale), - Hole::Drop { .. } => None, + Hole::Drop { .. } if hole.error() == StreamError::Old => None, + Hole::Drop { .. } => Some(&self.dropped_stale), }; let replaced: Vec<_> = replaces .into_iter() @@ -5376,13 +5386,20 @@ impl TrackRun { self.dropped.remove(gap.clone()); self.dropped_stale.remove(gap.clone()); } - Hole::Drop { .. } => { + // The wire names every OLD drop stale, which the next update reopens. One the + // cursor gave up as final, such as one relayed from a version whose drops are + // all final, is dropped again by each update, as an expired stream is. + Hole::Drop { .. } if hole.error() == StreamError::Old => { self.dropped.insert(gap.clone()); - match hole.stale_budget() { - Some(_) => self.dropped_stale.insert(gap.clone()), - None => self.dropped_stale.remove(gap.clone()), + self.dropped_stale.insert(gap.clone()); + if hole.stale_budget().is_none() { + self.expired.insert(gap.clone()); } } + Hole::Drop { .. } => { + self.dropped.insert(gap.clone()); + self.dropped_stale.remove(gap.clone()); + } } } } @@ -7979,6 +7996,100 @@ mod serve_group_test { } } + /// A group an update re-offers waits for stream credit, and is judged again by whatever + /// arrives meanwhile: a later update whose range no longer admits it, or a final drop the + /// reader reaches for it, leaves it unserved. + #[kio::test] + async fn a_queued_reoffer_is_judged_again_before_it_opens_on_lite07() { + for narrow in [true, false] { + let mut track = track::Producer::new( + Arc::new(broadcast::Info::default()), + "test", + crate::track::Info::timed(), + ); + let budget = Duration::from_millis(500); + let subscriber = + track.subscribe(track::Subscription::default().with_delay(track::Delay::default().with_max(budget))); + let (mut run, mut writer, log) = lite07_run(SinkSession::new(Log::default()), subscriber); + write_group(&mut track, 0, 0); + write_group(&mut track, 1, 1000); + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + write_group(&mut track, 2, 3000); + run.ctx.outcomes.dropped.lock().unwrap().push((0, StreamError::Old)); + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + let opened = run.ctx.outcomes.opened.load(Ordering::Relaxed); + let update = |id, start, update| lite::SubscribeUpdate { + wanted_group: None, + priority: 0, + max_delay: Duration::from_secs(10), + min_delay: None, + ranges: vec![(id, lite::Range::span(start, 0, None, None))], + order: Order::Descending, + update, + }; + // Widened: group 0 is fresh again, and waits its turn. + run.update(update(0, Some(0), 1)); + assert_eq!(run.reoffered.len(), 1); + match narrow { + true => run.update(update(1, Some(1), 2)), + false => { + run.forward_range(0..1, &Hole::drop(StreamError::Cancel), &mut writer) + .unwrap(); + } + } + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + assert_eq!( + run.ctx.outcomes.opened.load(Ordering::Relaxed), + opened, + "narrow={narrow}: group 0 never opens again" + ); + let drops: &[(u64, u64, u64)] = match narrow { + // Outside the range now, so it is dropped again under the narrowing update. + true => &[(0, 0, 0), (0, 0, 2)], + false => &[(0, 0, 0), (0, 0, 1)], + }; + assert_eq!( + drop_updates(&mut writer, &log, Version::Lite07).await, + drops, + "narrow={narrow}" + ); + } + } + + /// A drop the cursor gave up as final with OLD, as one relayed from a lite-06 upstream is, + /// goes out as a stale drop on lite-07, which the subscriber reopens on its next update, so + /// each update drops it again rather than leave it owed. + #[kio::test] + async fn a_final_old_drop_is_dropped_again_by_each_update_on_lite07() { + let mut track = track::Producer::new( + Arc::new(broadcast::Info::default()), + "test", + crate::track::Info::timed(), + ); + let subscriber = track.subscribe(None); + let (mut run, mut writer, log) = lite07_run(SinkSession::new(Log::default()), subscriber); + write_group(&mut track, 0, 0); + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + run.forward_range(1..2, &Hole::drop(StreamError::Old), &mut writer) + .unwrap(); + for update in [1, 2] { + run.update(lite::SubscribeUpdate { + wanted_group: None, + priority: 1, + max_delay: Duration::from_secs(3600), + min_delay: None, + ranges: vec![(0, lite::Range::span(None, 0, None, None))], + order: Order::Descending, + update, + }); + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + } + assert_eq!( + drop_updates(&mut writer, &log, Version::Lite07).await, + [(1, 1, 0), (1, 1, 1), (1, 1, 2)] + ); + } + /// A run one update left to drop again is judged again under the next update that /// arrives before it goes out, rather than dropped under terms that may serve it. #[kio::test] diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index 55757c2a528..5105371aeea 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -4825,6 +4825,27 @@ mod tests { (serve, serving, tail, broadcast) } + /// A SUBSCRIBE_DROP can only echo an update the subscriber sent: one naming a later update + /// is a protocol violation, and accounts for nothing. + #[kio::test] + async fn a_drop_naming_an_unsent_update_is_a_protocol_violation() { + let (serve, mut serving, tail, _broadcast) = lite07_serving(Duration::from_millis(500)).await; + let drop = lite::SubscribeResponse::Drop(lite::SubscribeDrop { + start: 1, + end: 2, + error: u64::from(crate::coding::StreamCodes::encode_stream_code( + &Version::Lite07, + &crate::StreamError::Old, + )), + update: 1, + }); + assert!(matches!( + serving.apply_response(&serve, drop), + Err(Error::ProtocolViolation) + )); + assert!(!tail.read().covers(1..3)); + } + /// On lite-07 an OLD drop is a stale verdict judged under the budget this subscriber sent /// with the update it names. Any SUBSCRIBE_UPDATE reopens every stale run in the tail, /// whether or not it widens the budget, and an OLD drop naming a superseded update is no @@ -10743,6 +10764,10 @@ impl ServeLoop { if groups.is_empty() { return Ok(()); } + // It can only echo an update this subscriber sent. + if dropped.update > active.update { + return Err(Error::ProtocolViolation); + } let error = dropped.error(serve.subscriber.version); // On lite-07 an OLD drop is a stale verdict, which the next update reopens; every // other drop is final, whatever update it names. diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index 421cc6e6ac3..843e86f91d5 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -3131,6 +3131,10 @@ impl TrackState { { return Err(Error::Closed); } + // Never produced, so no fill holds it either. + if self.skipped(sequence) { + return Err(Error::Skipped); + } // Adopt the supplied info only if the track hasn't been accepted yet. Groups // created here charge the same account as any other, so backfill written @@ -22465,8 +22469,8 @@ mod test { assert_eq!(drops, [(1..4, Hole::Skip)]); } - /// A skip is final: creating a group at a sequence an append or `skip_groups` skipped is - /// refused, while a dropped one may be created again. + /// A skip is final: creating or filling a group at a sequence an append or `skip_groups` + /// skipped is refused, while a dropped one may be created again. #[test] #[cfg(feature = "async")] fn creating_a_skipped_sequence_is_refused() { @@ -22492,6 +22496,16 @@ mod test { producer.create_group(group::Info { sequence: 1 }), Err(Error::Skipped) )); + // No fill holds a skipped sequence either. + assert!(matches!( + producer + .state + .write() + .ok() + .unwrap() + .insert_group_request(6, group::Bounds::default(), None), + Err(Error::Skipped) + )); // A dropped sequence may be created again. producer.give_up(9..10, Hole::drop(StreamError::Cancel)); write_group(&producer, 9); diff --git a/rs/moq-net/src/model/track/fill.rs b/rs/moq-net/src/model/track/fill.rs index 882c748d382..939c8d9d188 100644 --- a/rs/moq-net/src/model/track/fill.rs +++ b/rs/moq-net/src/model/track/fill.rs @@ -811,8 +811,8 @@ impl Fill { /// frames written keep the indices they have in the group. `info` sets the track's /// [`Info`] if it hasn't been accepted yet (a fetch with no live subscription), and is /// ignored otherwise. Errors with [`Error::InvalidRange`] outside the fill, - /// [`Error::Duplicate`] if the group was already answered or cached, or the track's abort - /// error. + /// [`Error::Duplicate`] if the group was already answered or cached, [`Error::Skipped`] if + /// the track skipped it, or the track's abort error. pub fn accept(&self, sequence: u64, info: impl Into>) -> Result { self.inner.accept(sequence, info.into()) } From e790fe9c399d19e38741c5c5dcb4cb761f6ac177 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 21:50:26 -0800 Subject: [PATCH 37/68] fix(net): drop a needless mut in a subscriber test helper Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/src/lite/subscriber.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index 5105371aeea..49ebd778951 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -4798,7 +4798,7 @@ mod tests { let tail = kio::Producer::new(Tail::default()); serving.sub = Sub::Active(SubStream { order: crate::track::Order::Descending, - stream: Stream::open(&mut crate::driver::Transport::new(session.clone()), version) + stream: Stream::open(&crate::driver::Transport::new(session.clone()), version) .await .unwrap(), id: 0, From d5e72cc005e38ece5f6c7d8395efa29165ba93d3 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 22:09:03 -0800 Subject: [PATCH 38/68] fix(net): keep a fill owed across a stale verdict in JS A stale verdict settles only the readers it covers, so recording one no longer answers the fills other readers asked, as in Rust. Co-Authored-By: Claude Opus 5.5 --- js/net/src/fill.test.ts | 24 ++++++++++++++++++++++++ js/net/src/track.ts | 5 +++-- 2 files changed, 27 insertions(+), 2 deletions(-) diff --git a/js/net/src/fill.test.ts b/js/net/src/fill.test.ts index 83741d7e9d8..f2b82f254b3 100644 --- a/js/net/src/fill.test.ts +++ b/js/net/src/fill.test.ts @@ -336,6 +336,30 @@ test("fill: a fill names the most urgent priority and loosest budget among its r broadcast.close(); }); +// A stale verdict settles only the readers it covers, so a fetch it does not settle keeps its fill. +// Rust records a stale run without answering any fill. +test("fill: a stale verdict leaves an outstanding fill owed", async () => { + const broadcast = new BroadcastProducer(); + const track = broadcast.createTrack("video", { maxAge: null }); + const requests = track.requests(); + track.createGroup(3n).finish(); + const fetching = wireOf(broadcast).fetchGroup("video", 2n); + const fill = await requests.requestedFill(); + if (!fill) throw new Error("no fill"); + expect(fill.wanted()).toBe(2n); + + hooks.dropTrackGroups(track, 2n, 3n, staleHole(500)); + expect(fill.wanted()).toBe(2n); + const group = fill.accept(2n); + group.writeFrame({ payload: enc.encode("filled") }); + group.finish(); + expect(await frames(await fetching)).toEqual(["filled"]); + + fill.close(); + requests.close(); + broadcast.close(); +}); + test("fill: a group arriving live answers the fill", async () => { const broadcast = new BroadcastProducer(); const track = broadcast.createTrack("video", { maxAge: null }); diff --git a/js/net/src/track.ts b/js/net/src/track.ts index 4a45bcb2b2d..2271fd6422c 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -1220,8 +1220,9 @@ export class Producer { } } } - // A fill no longer owes what the track gave up. - this.#fills.dropped(start, end); + // A fill no longer owes what the track gave up for good. A stale verdict settles only + // the readers it covers, so a fill asked by any other is still owed. + if (hole.stale === undefined) this.#fills.dropped(start, end); for (const state of [this.#state, ...this.#sinks]) { state.drops.update((drops) => runs.reduce((drops, run) => insertDrop(drops, run.start, run.end, hole), drops), From 831959fadc53fc0fb627bef469a2cb6a458804b4 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 23:18:59 -0800 Subject: [PATCH 39/68] feat(net): hold a duplicate group stream until the first copy settles On lite-07 an expired group's OLD drop and the stream an update re-offers can both overtake the original stream's reset. The subscriber refused the re-offer as a duplicate and lost the group once the reset landed. It now holds the duplicate unread: discarded if the arriving copy finishes, taking the group if it fails. Rust and JS, with controlled-order tests; the draft states the hold. Co-Authored-By: Claude Opus 5.5 --- drafts/draft-lcurley-moq-lite.md | 5 +- js/net/src/internal.ts | 5 + js/net/src/lite/subscriber.test.ts | 73 ++++++++++ js/net/src/lite/subscriber.ts | 9 ++ js/net/src/track.ts | 4 + quest/m1/drop-set.md | 15 ++ rs/moq-net/src/lite/subscriber.rs | 215 ++++++++++++++++++++++++++--- rs/moq-net/src/model/track.rs | 37 +++-- 8 files changed, 336 insertions(+), 27 deletions(-) diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index 9ab498fc8f6..1370c595100 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -779,7 +779,8 @@ A frame MAY contain an empty payload, potentially indicating a gap in the group. The GROUP message's `Frame Start` gives the index of the first FRAME on the stream, so a stream carrying only part of a Group is self-describing. It is redundant with the subscription's own start Position in the steady state, and carried anyway because a SUBSCRIBE_UPDATE that moves the bound races the Group Streams already in flight: the two travel on different streams with no ordering between them, so the receiver would otherwise have to guess which bound a given Group Stream was opened under. -A publisher MUST NOT send two Group Streams for the same Group Sequence on one subscription. +A publisher MUST NOT send two Group Streams for the same Group Sequence on one subscription, except to re-offer a Group whose stream it reset (see [SUBSCRIBE_DROP](#subscribe-drop)). +The re-offer can arrive before that reset: a subscriber holds a Group Stream for a sequence whose earlier stream is still arriving unread until that stream ends, then discards it if the earlier stream completed the Group, or reads it if the earlier stream was reset. A subscriber assembling a Group from more than one publisher does so across separate subscriptions, and is responsible for concatenating the runs in index order and for ignoring any frame it has already received. Both the publisher and subscriber MAY reset the stream at any time. @@ -2067,7 +2068,7 @@ The `Message Length` describes the payload size on the wire. - SUBSCRIBE_DROP's Error Code is a stream error code, which ends the dropped groups' readers, including one still arriving. - Refuse a SUBSCRIBE_DROP whose Group Start is greater than Group End plus one on every version that carries it. - Added SUBSCRIBE_SKIP (type 0x3) for sequences the publisher declared it will never produce, which ends no reader and is final: a Group is never created at a skipped sequence. A skip overrides an earlier drop of the same sequences, whatever its Error Code, without re-delivering to readers the drop already ended, and never overrides a Group that arrived. A gap in the sequence numbers is not a skip, since Groups may be created out of order; a sequence never had and never declared skipped is named in SUBSCRIBE_DROP. SUBSCRIBE_DROP no longer names declared skips. -- Added `Update` to SUBSCRIBE, SUBSCRIBE_UPDATE, and SUBSCRIBE_DROP: the number of the terms in effect. A stale (OLD) drop holds only under the update it names, judged against the budget the subscriber sent with it; every later SUBSCRIBE_UPDATE reopens it, and the publisher re-offers the group or drops it again under the new `Update`. Any other drop is final. A Group Stream wins over its earlier drop, and over a stale drop that arrives after it; no update reopens a sequence its Group Stream accounted for. +- Added `Update` to SUBSCRIBE, SUBSCRIBE_UPDATE, and SUBSCRIBE_DROP: the number of the terms in effect. A stale (OLD) drop holds only under the update it names, judged against the budget the subscriber sent with it; every later SUBSCRIBE_UPDATE reopens it, and the publisher re-offers the group or drops it again under the new `Update`. Any other drop is final. A Group Stream wins over its earlier drop, and over a stale drop that arrives after it; no update reopens a sequence its Group Stream accounted for. A re-offered Group Stream that overtakes the reset of the stream it replaces waits, unread, until that stream ends, and is read only if it was reset. - SUBSCRIBE_FINISH declares the final boundary while lower groups can still be in flight, and no longer closes the Subscribe Stream: the publisher keeps reading SUBSCRIBE_UPDATE, so an update can reach an ended Track's history. - A covered subscription, whose every Range has an End and whose every Group was sent or dropped, also stays open for an update; the publisher never FINs first. - A publisher that withheld SUBSCRIBE_OK sends it, then the drops it withheld, once no group can resolve the start: every Group of a bounded subscription was given up, or the Track finished. diff --git a/js/net/src/internal.ts b/js/net/src/internal.ts index 95de5402bec..c47dd7c9d78 100644 --- a/js/net/src/internal.ts +++ b/js/net/src/internal.ts @@ -558,6 +558,8 @@ export const hooks: { receiveTrack: (producer: Producer, starts?: FillStarts) => void; /** Create a received group at its wire offset before publishing it to readers. Assigned by `track.ts`. */ receiveGroup: (producer: Producer, group: { sequence: bigint; start: number }) => GroupProducer; + /** How the copy of `sequence` still arriving ends, if one is, for a duplicate to wait on. Assigned by `track.ts`. */ + arrivingGroup: (producer: Producer, sequence: bigint) => GetPromise | undefined; /** The producer's largest reported location and its subsequent changes. */ producerLargest: (producer: Producer) => Getter; @@ -1035,6 +1037,9 @@ export const hooks: { receiveGroup: () => { throw new Error("track.ts not loaded"); }, + arrivingGroup: () => { + throw new Error("track.ts not loaded"); + }, groupOffset: () => { throw new Error("group.ts not loaded"); }, diff --git a/js/net/src/lite/subscriber.test.ts b/js/net/src/lite/subscriber.test.ts index cf3ce397f3c..c1c2687ee26 100644 --- a/js/net/src/lite/subscriber.test.ts +++ b/js/net/src/lite/subscriber.test.ts @@ -1456,6 +1456,79 @@ test("a group that beats its track info waits for it", async () => { } }); +// A lite-07 re-offer can overtake the reset of the stream it replaces: it waits, unread, until that +// copy ends. It is refused as a duplicate once the copy finishes, and takes the group once it fails. +test.each([false, true])("a duplicate of an arriving copy waits for it to settle (first fails: %p)", async (fails) => { + const version = Version.DRAFT_07; + const { quic, streams, opened } = fakeSession(); + const subscriber = new Subscriber(quic, version, HopSchema.parse(1n)); + const track = subscriber.consume(Path.from("room")).track("video").subscribe(); + try { + await opened(0); + await (await opened(1)).writing; + await answerTrackInfo(streams[0], version, 1000); + + // The original copy, still arriving: one frame in, no end yet. + const chunks: Uint8Array[] = []; + const writer = new Writer( + new WritableStream({ write: (chunk) => void chunks.push(new Uint8Array(chunk)) }), + version, + ); + await writer.u62(0n); + await writer.u53(1); + await writer.write(new Uint8Array([7])); + let first!: ReadableStreamDefaultController; + const arriving = new ReadableStream({ + start: (controller) => { + first = controller; + }, + }); + const original = subscriber.runGroup( + new Group({ subscribe: 0n, sequence: 0n }), + new Reader(arriving, undefined, version), + ); + for (const chunk of chunks) first.enqueue(chunk); + const group = await track.recvGroup(); + expect((await group?.readFrame())?.payload).toEqual(new Uint8Array([7])); + + const duplicate = await groupStream(0, version); + const stop = spyOn(duplicate, "stop"); + let held = true; + const receiving = subscriber.runGroup(new Group({ subscribe: 0n, sequence: 0n }), duplicate).finally(() => { + held = false; + }); + await drainReactions(); + expect(held).toBe(true); + expect(stop).not.toHaveBeenCalled(); + + if (!fails) { + first.close(); + await original; + await receiving; + expect(stop.mock.calls[0]?.[0]).toBeInstanceOf(Error); + expect(await group?.readFrame()).toBeUndefined(); + return; + } + + // The reset lands: the held stream takes the group. + first.error(new Error("peer reset")); + await original; + await receiving; + expect(await group?.closed).toBeInstanceOf(Error); + const later = subscriber.consume(Path.from("room")).track("video").subscribe(); + try { + const again = await later.recvGroup(); + expect(again?.sequence).toBe(0n); + expect((await again?.readFrame())?.payload).toEqual(new Uint8Array([0])); + } finally { + later.close(); + } + } finally { + track.close(); + subscriber.close(); + } +}); + // A TRACK reset while its SUBSCRIBE is live fails the subscription, so the origin can try another // route, and resets the SUBSCRIBE. test("a reset track rejects the subscription and resets its subscribe", async () => { diff --git a/js/net/src/lite/subscriber.ts b/js/net/src/lite/subscriber.ts index 84906c63201..33c2281fffa 100644 --- a/js/net/src/lite/subscriber.ts +++ b/js/net/src/lite/subscriber.ts @@ -2218,6 +2218,15 @@ export class Subscriber { track.abort(violation); throw violation; } + // A duplicate of a copy still arriving, such as a lite-07 re-offer that overtook the reset + // of the stream it replaces, waits unread, which flow control bounds. Created once that copy + // ends, it is refused as a duplicate if the copy finished and takes the group if it failed. + // Its own reset ends the wait; a clean end leaves it on the copy alone. + for (let arriving = hooks.arrivingGroup(track, group.sequence); arriving !== undefined; ) { + const settled = arriving; + await race([settled, stream.closed.then(() => settled)]); + arriving = hooks.arrivingGroup(track, group.sequence); + } producer = hooks.receiveGroup(track, { sequence: group.sequence, start: group.frameStart }); await readFrames(stream, producer, { diff --git a/js/net/src/track.ts b/js/net/src/track.ts index 2271fd6422c..de0e56a25a4 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -1067,6 +1067,10 @@ export class Producer { producer.#starts = starts; }; hooks.receiveGroup = (producer, { sequence, start }) => producer.#createGroup(sequence, start); + hooks.arrivingGroup = (producer, sequence) => { + const closed = producer.#cached.get(sequence)?.group.closed; + return closed?.peek() === undefined ? closed : undefined; + }; hooks.writeDatagram = (producer, datagram, inflated) => producer.#insertDatagram(datagram, inflated); sourceStart = (producer) => producer.#served && producer.#feedStart(); fillFloor = (producer, named) => producer.#fillFloor(named); diff --git a/quest/m1/drop-set.md b/quest/m1/drop-set.md index c48a4fdacfb..8642052197d 100644 --- a/quest/m1/drop-set.md +++ b/quest/m1/drop-set.md @@ -178,6 +178,21 @@ decision above and the PR's `## Waiting` item on out-of-order creation: finish-time inference for a local producer (a gap at finish is no more a declaration than one mid-stream). +Decided 2026-11-05 (maintainer, /quest-complete): a duplicate Group Stream +waits until the first copy settles. On lite-07 the OLD drop for an expired +group, and the stream the next update re-offers for it, can both reach the +subscriber before the original stream's reset, which used to refuse the +re-offer as a duplicate and lose the group once the reset landed. The +subscriber now holds a stream whose sequence another copy still arriving +holds, unread, and discards it if that copy finishes or takes the group if it +fails (Rust `GroupRecv`'s `Hold` wait over `track::Received::Held`, JS +`runGroup` over `receiveGroup`'s `arriving`). The held stream keeps the +lowest receive priority, so it holds at most its stream credit and is the +first a full receive window evicts. A reader already handed the failed copy +sees it fail, since no sequence is handed out twice; the cache, fills, and +readers that had not reached it get the re-offer. Not chosen: letting the +duplicate replace the copy still arriving; accepting the loss. + Decided 2026-10-27 (maintainer): this runs after the range count cost work, which finished in [moq-dev/moq#7542](https://github.com/moq-dev/moq/pull/7542), and before [Rust groups by start](/quest/m1/time-addressing/model.md), which diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index 49ebd778951..ec56432b3f2 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -199,10 +199,16 @@ struct Parked { /// The headers still waiting on the accept, by ticket, in arrival order. waiting: Vec<(u64, lite::Group)>, /// What the accept opened for each, until its stream takes it. - opened: HashMap>, + opened: HashMap>, next: u64, } +/// What a group stream's header opened in its subscription's copy. +enum Open { + Group(Opened), + Duplicate(Duplicate), +} + /// A group stream's group, opened in its subscription's copy for the stream to fill. struct Opened { group: group::Producer, @@ -211,6 +217,16 @@ struct Opened { reading: Reading, } +/// A group stream duplicating a copy still arriving, such as a lite-07 re-offer that +/// overtook the reset of the stream it replaces. It waits unread, which its stream credit +/// bounds, and is discarded if that copy finishes or takes the group if it fails. +struct Duplicate { + arriving: group::Handle, + track: track::Producer, + /// Counted already, and kept open while held, since the stream may still carry the group. + reading: Reading, +} + impl Drop for TrackEntry { fn drop(&mut self) { // Groups the accept opened for streams that never took them end as cancelled with @@ -223,7 +239,9 @@ impl Drop for TrackEntry { Err(_) => return, }; for opened in opened.into_values().flatten() { - let _ = opened.group.abort(Error::Cancel); + if let Open::Group(opened) = opened { + let _ = opened.group.abort(Error::Cancel); + } } } } @@ -296,7 +314,7 @@ impl TrackEntry { } /// Take what the accept opened for the parked stream `ticket`. - fn unpark(&mut self, ticket: u64) -> Result { + fn unpark(&mut self, ticket: u64) -> Result { let Some(parked) = &self.parked else { return Err(Error::Cancel); }; @@ -327,7 +345,7 @@ impl TrackEntry { } Err(_) => return, }; - if let Some(Ok(opened)) = opened { + if let Some(Ok(Open::Group(opened))) = opened { let _ = opened.group.abort(err.clone()); } if done { @@ -336,8 +354,8 @@ impl TrackEntry { } /// Open the group of a stream whose header is `hdr` in the copy. - fn open(&mut self, hdr: &lite::Group, version: Version, now: crate::time::Instant) -> Result { - let Some(Accepted { producer, framing }) = &mut self.copy else { + fn open(&mut self, hdr: &lite::Group, version: Version, now: crate::time::Instant) -> Result { + let Some(Accepted { producer, .. }) = &mut self.copy else { return Err(Error::Cancel); }; let group_info = group::Info { sequence: hdr.sequence }; @@ -370,8 +388,38 @@ impl TrackEntry { } } } + self.place(hdr, version, received, reading) + } + + /// Open the group of a held stream again, once the copy it waited on failed. + fn reopen(&self, hdr: &lite::Group, version: Version, reading: Reading) -> Result { + let Some(Accepted { producer, .. }) = &self.copy else { + return Err(Error::Cancel); + }; + let received = producer.create_group_at(group::Info { sequence: hdr.sequence }, hdr.frame_start); + self.place(hdr, version, received, reading) + } + + /// Give a stream what the cache answered for its header. + fn place( + &self, + hdr: &lite::Group, + version: Version, + received: Result, + reading: Reading, + ) -> Result { + let Some(Accepted { producer, framing }) = &self.copy else { + return Err(Error::Cancel); + }; let group = match received { - Ok(group) => group, + Ok(track::Received::Group(group)) => group, + Ok(track::Received::Held(arriving)) => { + return Ok(Open::Duplicate(Duplicate { + arriving, + track: producer.clone(), + reading, + })); + } // The group is at or past the end the publisher declared, which no // later stream can repair. Before lite-05 only a local finish sets // an end, and lite-05 specified an inclusive one, so its last @@ -388,12 +436,12 @@ impl TrackEntry { } Err(err) => return Err(err), }; - Ok(Opened { + Ok(Open::Group(Opened { group, track: producer.clone(), framing: *framing, reading, - }) + })) } } @@ -1358,7 +1406,8 @@ impl GroupRecv { None => Poll::Ready(Ok(())), }); self.park_wait = None; - let (ietf, hdr) = header; + let ietf: bool = header.0; + let hdr: lite::Group = header.1; if let (Err(err), Some((_, ticket))) = (&accepted, &parked) && let Some(entry) = self.subscriber.subscribes.lock().get_mut(&hdr.subscribe) { @@ -1366,7 +1415,7 @@ impl GroupRecv { } accepted?; - let opened = { + let open = { let mut subs = self.subscriber.subscribes.lock(); let entry = subs.get_mut(&hdr.subscribe).ok_or(Error::Cancel)?; match &parked { @@ -1374,15 +1423,45 @@ impl GroupRecv { None => entry.open(&hdr, self.subscriber.version, self.subscriber.runtime.now()), } }; + let mut open: Option = Some(open?); + // A duplicate of a copy still arriving waits, unread, for that copy to settle. + let opened: Opened = wait!(Hold => loop { + let duplicate = match open.take() { + Some(Open::Group(opened)) => break Poll::Ready(Ok(opened)), + Some(Open::Duplicate(duplicate)) => duplicate, + None => break Poll::Ready(Err(Error::Cancel)), + }; + if let Poll::Ready(err) = self.reader.poll_reset(waiter) { + break Poll::Ready(Err(err)); + } + if let Poll::Ready(err) = duplicate.track.poll_aborted(waiter) { + break Poll::Ready(Err(err)); + } + match duplicate.arriving.poll_closed(waiter) { + Poll::Pending => { + *open = Some(Open::Duplicate(duplicate)); + break Poll::Pending; + } + // The copy holds every frame, so this one only duplicates it. + Poll::Ready(Ok(())) => break Poll::Ready(Err(Error::Duplicate)), + Poll::Ready(Err(_)) => { + let subs = self.subscriber.subscribes.lock(); + let Some(entry) = subs.get(&hdr.subscribe) else { + break Poll::Ready(Err(Error::Cancel)); + }; + match entry.reopen(hdr, self.subscriber.version, duplicate.reading) { + Ok(reopened) => *open = Some(reopened), + Err(err) => break Poll::Ready(Err(err)), + } + } + } + })?; let Opened { group, track, framing, reading, - } = match opened { - Ok(opened) => opened, - Err(err) => return Poll::Ready(Err(err)), - }; + } = opened; let (priority, order) = track .subscription() .map_or((0, Default::default()), |sub| (sub.priority, sub.order)); @@ -4526,6 +4605,108 @@ mod tests { assert!(settle.poll(&kio::Waiter::noop(), |tail| tail.streams() >= 1).is_ready()); } + /// A lite-07 re-offer can overtake the reset of the stream it replaces: its stream waits, + /// unread, until that copy settles. It is discarded once the copy finishes, and takes the + /// group once the copy fails. + #[kio::test] + async fn a_duplicate_of_an_arriving_copy_waits_for_it_to_settle() { + use crate::transport::poll::Session as _; + + for first_fails in [false, true] { + let version = Version::Lite07; + let mut script = Vec::new(); + let mut encoder = crate::coding::Encoder::new(&mut script, version.into()); + lite::Group { + subscribe: 7, + sequence: 3, + frame_start: 0, + } + .encode(&mut encoder, version) + .unwrap(); + encoder.varint(crate::coding::varint::zigzag(300)).unwrap(); + encoder.bytes(b"again").unwrap(); + let session = crate::lite::test_transport::ScriptedSession::eof(script); + let clock = crate::time::Deadlines::new(crate::time::Instant::now()); + let subscriber = Subscriber::new(SubscriberConfig { + runtime: clock.clone(), + session: crate::driver::Transport::new(session.clone()), + origin: origin::Config::new(crate::Hop::new(1).unwrap()).produce(), + recv_bandwidth: None, + version, + peer_setup: crate::lite::PeerSetup::new(false, crate::transport::Address::Setup), + peer_hop: None, + cost: None, + going_away: Default::default(), + auth: crate::auth::Link::new(false), + }); + + let track = track::Producer::new( + std::sync::Arc::new(crate::broadcast::Info::default()), + "video", + crate::track::Info::timed(), + ); + let subscription = crate::track::Subscription::default() + .with_delay(crate::track::Delay::default().with_max(std::time::Duration::from_secs(10))) + .with_range(0..); + let mut reader = track.subscribe(subscription.clone()); + // The original copy, still arriving: one frame in, no end yet. + let Ok(track::Received::Group(mut first)) = track.create_group_at(group::Info { sequence: 3 }, 0) else { + panic!("the first copy opens"); + }; + first + .write_frame(Timestamp::from_millis(300).unwrap(), &b"first"[..]) + .unwrap(); + let tail = kio::Producer::new(Tail::default()); + subscriber.subscribes.lock().insert( + 7, + TrackEntry::accepted( + track.clone(), + lite::Framing { + timescale: Some(Timescale::MILLI), + ietf: None, + compressed: false, + }, + tail.clone(), + ), + ); + + let (_, recv) = session.open_bi(Default::default()).await.unwrap(); + let mut duplicate = GroupRecv::new( + subscriber, + Reader::new(crate::transport::boxed::RecvStream::new(recv), version), + ); + let waiter = kio::Waiter::noop(); + assert!(duplicate.poll_serve(&waiter).is_pending(), "first fails: {first_fails}"); + assert_eq!(duplicate.state.name(), "Hold", "first fails: {first_fails}"); + // Counted and holding the end open while it waits, since it may carry the group. + assert_eq!(tail.read().streams(), 1); + + let mut original = reader.assert_group(); + if !first_fails { + first.finish().unwrap(); + let res = kio::wait(|waiter| duplicate.poll_serve(waiter)).await; + assert!(matches!(res, Err(Error::Duplicate)), "{res:?}"); + let frame = original.read_frame().await.unwrap().unwrap(); + assert_eq!(&frame.payload[..], b"first"); + assert!(original.read_frame().await.unwrap().is_none()); + continue; + } + + // The reset lands: the held stream takes the group. + first.abort(Error::Cancel).unwrap(); + let res = kio::wait(|waiter| duplicate.poll_serve(waiter)).await; + assert!(res.is_ok(), "{res:?}"); + // A reader already handed the original sees it fail, since no sequence is handed + // out twice; one that had not reached it reads the re-offer. + assert!(original.read_frame().await.is_err()); + let mut later = track.subscribe(subscription); + let mut group = later.assert_group(); + assert_eq!(group.sequence, 3); + assert_eq!(&group.read_frame().await.unwrap().unwrap().payload[..], b"again"); + assert!(group.read_frame().await.unwrap().is_none()); + } + } + /// A SUBSCRIBE_FINISH below a group already received contradicts that group. lite-05 /// specified an inclusive end, and `@moq/net` 0.1.3 to 0.1.9 sent one, so there it only /// costs the early boundary; later drafts abort the track. @@ -6073,7 +6254,9 @@ mod tests { Err(_) => panic!("the entry holds the parked streams"), }; assert_eq!(opened, [kept], "the withdrawn group is gone"); - let Opened { mut group, .. } = entry.unpark(kept).expect("the kept group"); + let Ok(Open::Group(Opened { mut group, .. })) = entry.unpark(kept) else { + panic!("the kept group"); + }; group.finish().unwrap(); track.finish().unwrap(); // Only the kept group reaches the subscriber: the withdrawn one ended, so nothing diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index 843e86f91d5..ec09b2af258 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -3312,6 +3312,16 @@ fn largest_ended(state: &TrackState, waiter: &kio::Waiter) -> Poll<()> { } } +/// Where a received copy of a group goes. +pub(crate) enum Received { + /// A new group, for the copy to fill. + Group(group::Producer), + /// Another copy of the sequence that is still arriving: whether this one is needed + /// depends on how that one settles. + #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] + Held(group::Handle), +} + /// A producer for a track, used to create new groups. #[derive(Clone)] pub struct Producer { @@ -3408,16 +3418,22 @@ impl Producer { /// live edge, or the successor that ages the group before it) only once its first frame /// opens or it finishes. pub fn create_group(&self, group: group::Info) -> Result { - self.create_group_inner(group, None) + match self.create_group_inner(group, None)? { + Received::Group(group) => Ok(group), + Received::Held(_) => unreachable!("only a received copy waits on another"), + } } /// A received copy starts at its wire offset before any reader can see it. + /// + /// A copy that already answers from `start` holds the sequence: a finished one refuses + /// this copy with [`Error::Duplicate`], and one still arriving is returned to wait on. #[cfg(feature = "async")] - pub(crate) fn create_group_at(&self, group: group::Info, start: u64) -> Result { + pub(crate) fn create_group_at(&self, group: group::Info, start: u64) -> Result { self.create_group_inner(group, Some(start)) } - fn create_group_inner(&self, info: group::Info, received: Option) -> Result { + fn create_group_inner(&self, info: group::Info, received: Option) -> Result { let mut state = self.modify()?; if state.sealed || state.final_sequence().is_some_and(|fin| info.sequence >= fin) { return Err(Error::Closed); @@ -3429,8 +3445,11 @@ impl Producer { } let start = received.unwrap_or(0); let content = if received.is_some() { - if state.holding_group(info.sequence, start).is_some() { - return Err(Error::Duplicate); + if let Some(holding) = state.holding_group(info.sequence, start) { + return match holding.is_settled() { + true => Err(Error::Duplicate), + false => Ok(Received::Held(holding.clone())), + }; } state.content_for_copy(info.sequence)? } else { @@ -3450,7 +3469,7 @@ impl Producer { let deferred = state.deferred(); drop(state); deferred.wake(); - Ok(group) + Ok(Received::Group(group)) } /// Create a new group with the next sequence number: one past this producer's own writes. @@ -19587,9 +19606,9 @@ mod test { let mut fetched = pending.await.unwrap(); subscriber.assert_no_group(); - let mut live = producer - .create_group_at(0u64.into(), 0) - .expect("the feed takes the slot"); + let Ok(Received::Group(mut live)) = producer.create_group_at(0u64.into(), 0) else { + panic!("the feed takes the slot"); + }; live.write_frame(Timestamp::ZERO, bytes::Bytes::from_static(b"snapshot")) .unwrap(); let mut group = subscriber.assert_group(); From 964edfdad6a6e4e3c61d798b7d52393565c1a2c2 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 4 Nov 2026 23:35:18 -0800 Subject: [PATCH 40/68] fix(net): leave a superseded upstream stale report owed on an update A relay cursor that reported an upstream's received stale run, superseded by the relay's own update before the downstream update reopened it, gave it up again as OLD as if it had judged the run itself. The cursor now keeps which stale reports a received run decided, and an update leaves those to the upstream: folded in again if the track still holds the run, owed if new terms superseded it. Rust and JS. Co-Authored-By: Claude Opus 5.5 --- js/net/src/track.test.ts | 19 +++++++++ js/net/src/track.ts | 34 +++++++++++---- rs/moq-net/src/model/track.rs | 80 +++++++++++++++++++++++++++-------- 3 files changed, 107 insertions(+), 26 deletions(-) diff --git a/js/net/src/track.test.ts b/js/net/src/track.test.ts index af3b667d8a0..b7b63630426 100644 --- a/js/net/src/track.test.ts +++ b/js/net/src/track.test.ts @@ -3970,6 +3970,25 @@ test("superseding withdraws a partly reported stale run", () => { producer.close(); }); +// A received stale run a reader reported and the relay's own update then superseded is unknown +// again: the reader's update owes it until the upstream serves or drops it again, rather than +// dropping it again as if the reader had judged it stale itself. Mirrors Rust. +test("a superseded received stale report stays owed after an update", () => { + const producer = new TrackProducer("video").accept({}); + const reader = producer.subscribe({ delay: { max: Milli(500) }, ranges: [{ start: { absolute: 0n } }] }); + hooks.dropTrackGroups(producer, 1n, 3n, staleHole(500)); + expect(hooks.tryDrop(reader)).toMatchObject({ start: 1n, end: 3n, stale: 500 }); + hooks.supersedeStale(producer, undefined); + expect(hooks.reofferStale(reader)).toEqual([{ start: 1n, end: 3n }]); + expect(hooks.tryDrop(reader)).toBeUndefined(); + // The upstream drops it again under the new terms: reported again. + hooks.dropTrackGroups(producer, 1n, 3n, staleHole(500)); + expect(hooks.tryDrop(reader)).toMatchObject({ start: 1n, end: 3n, stale: 500 }); + reader.close(); + producer.finish(); + producer.close(); +}); + test("a received stale run settles a bounded reader only to the judged end", () => { const producer = new TrackProducer("video").accept({ maxAge: null }); hooks.receiveTrack(producer); diff --git a/js/net/src/track.ts b/js/net/src/track.ts index de0e56a25a4..d6ef2056b12 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -2376,6 +2376,9 @@ export class Subscriber { // The reported runs judged stale, for a wire whose update re-offers them; see // `hooks.reofferStale`. #reportedStale: Run[] = []; + // The part of #reportedStale an upstream's received stale run decided, which is the upstream's + // to judge again once an update reopens it. + #reportedReceived: Run[] = []; // The reported runs skipped: a skip replaces a drop, so a run reported dropped is reported // again once it is skipped, and only a skip already reported stops it. #reportedSkip: Run[] = []; @@ -2631,6 +2634,7 @@ export class Subscriber { // retires its own. if (this.#reportedStale.length > 0) { this.#reportedStale = removeRun(this.#reportedStale, sequence, sequence + 1n); + this.#reportedReceived = removeRun(this.#reportedReceived, sequence, sequence + 1n); } if (this.#resume) this.#resume.forgetPastBudget(this.#delivered); else this.#forgetPastBudget(this.#delivered); @@ -2750,12 +2754,18 @@ export class Subscriber { #reofferStale(): Run[] { if (this.#resume) return this.#resume.reofferStale(); const taken: Run[] = []; + const received = this.#reportedReceived; for (const run of this.#reportedStale) { this.#reported = removeRun(this.#reported, run.start, run.end); insertRun(taken, run.start, run.end); } this.#reportedStale = []; - for (const run of this.#pending) if (run.stale !== undefined) insertRun(taken, run.start, run.end); + this.#reportedReceived = []; + for (const run of this.#pending) { + if (run.stale === undefined) continue; + insertRun(taken, run.start, run.end); + if (run.received) insertRun(received, run.start, run.end); + } this.#pending = this.#pending.filter((run) => run.stale === undefined); // A received stale run is folded in again by the next scan, if the track still holds it. this.#dropsSeen = undefined; @@ -2763,11 +2773,13 @@ export class Subscriber { taken.some((run) => run.start <= sequence && sequence < run.end), ); // What no skip offers again, such as a group written off at the track's max age bound, - // which no budget reverses, is still stale under the new terms: reported again. A - // received stale run is left to the scan above. Mirrors Rust. + // which no budget reverses, is still stale under the new terms: reported again. What an + // upstream's stale run decided is the upstream's to judge again: the scan above folds it + // in again if the track still holds it, and it stays owed if new terms superseded it. + // Mirrors Rust. let kept: Run[] = taken.map((run) => ({ ...run })); for (const sequence of sequences) kept = removeRun(kept, sequence, sequence + 1n); - for (const dropped of this.#state.drops.peek()) kept = removeRun(kept, dropped.start, dropped.end); + for (const decided of received) kept = removeRun(kept, decided.start, decided.end); if (kept.length > 0) { const hole = staleHole(this.#drift().budget); for (const run of kept) this.#pending = insertDrop(this.#pending, run.start, run.end, hole); @@ -3699,7 +3711,7 @@ export class Subscriber { // The first part of `[from, to)` this cursor has not reported yet, now reported, with why. A // skip goes out over a drop already reported, and a final drop over a stale one, since each // replaces what it goes out over. - #report(from: bigint, to: bigint, hole: Hole): (Recv & { kind: "drop" }) | undefined { + #report(from: bigint, to: bigint, hole: Hole & { received?: boolean }): (Recv & { kind: "drop" }) | undefined { const { start, end } = this.#cursor.peek(); const final = !hole.skip && hole.stale === undefined; for (const run of this.#window.runs(max(from, start), min(to, end ?? to))) { @@ -3709,9 +3721,15 @@ export class Subscriber { if (!gap) continue; insertRun(this.#reported, gap.start, gap.end); if (hole.skip) insertRun(this.#reportedSkip, gap.start, gap.end); - if (hole.stale !== undefined) insertRun(this.#reportedStale, gap.start, gap.end); - // A skip or final drop is final, so no update reopens a stale report it replaced. - else this.#reportedStale = removeRun(this.#reportedStale, gap.start, gap.end); + if (hole.stale !== undefined) { + insertRun(this.#reportedStale, gap.start, gap.end); + if (hole.received) insertRun(this.#reportedReceived, gap.start, gap.end); + else this.#reportedReceived = removeRun(this.#reportedReceived, gap.start, gap.end); + } else { + // A skip or final drop is final, so no update reopens a stale report it replaced. + this.#reportedStale = removeRun(this.#reportedStale, gap.start, gap.end); + this.#reportedReceived = removeRun(this.#reportedReceived, gap.start, gap.end); + } return { kind: "drop", ...gap, error: hole.error, skip: hole.skip, stale: hole.stale }; } return undefined; diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index ec09b2af258..0cc58497b8b 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -6865,6 +6865,10 @@ struct DropCursor { /// [`Subscriber::reoffer_stale`]. #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] reported_stale: RangeSet, + /// The part of [`Self::reported_stale`] an upstream's received stale run decided, which + /// is the upstream's to judge again once an update reopens it. + #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] + reported_received: RangeSet, /// The reported runs skipped. A skip replaces a drop, so a run reported dropped is /// reported again once it is skipped, and only a skip already reported stops it. #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] @@ -6881,21 +6885,29 @@ impl DropCursor { scanned: None, reported: RangeSet::new(), reported_stale: RangeSet::new(), + reported_received: RangeSet::new(), reported_skip: RangeSet::new(), } } /// Take back every stale run, reported or not: an update reopens their verdict, so they - /// are judged again, and reported again under the new update if still stale. - fn take_stale(&mut self) -> Vec> { - let mut stale: RangeSet = self + /// are judged again, and reported again under the new update if still stale. Also returns + /// the part an upstream's received stale run decided. + fn take_stale(&mut self) -> (Vec>, RangeSet) { + let mut stale = RangeSet::new(); + let mut received = std::mem::take(&mut self.reported_received); + let queued: Vec<_> = self .pending .iter() .filter(|(_, pending)| pending.hole.stale_budget().is_some()) - .map(|(run, _)| run.clone()) + .map(|(run, pending)| (run.clone(), pending.received)) .collect(); - for run in stale.iter() { + for (run, from_upstream) in queued { self.pending.remove(run.clone()); + if from_upstream { + received.insert(run.clone()); + } + stale.insert(run); } for run in std::mem::take(&mut self.reported_stale).iter() { self.reported.remove(run.clone()); @@ -6903,7 +6915,7 @@ impl DropCursor { } // A received stale run is folded in again by the next scan, if the track still holds it. self.scanned = None; - stale.into_iter().collect() + (stale.into_iter().collect(), received) } /// Retire a stale report on `sequence` once its group is handed out: the group won over @@ -6911,6 +6923,7 @@ impl DropCursor { fn retire(&mut self, sequence: u64) { if let Some(end) = sequence.checked_add(1) { self.reported_stale.remove(sequence..end); + self.reported_received.remove(sequence..end); } } @@ -7016,13 +7029,21 @@ impl DropCursor { let hole = pending.hole; self.reported.insert(gap.clone()); match &hole { + hole if hole.stale_budget().is_some() => { + self.reported_stale.insert(gap.clone()); + match pending.received { + true => self.reported_received.insert(gap.clone()), + false => self.reported_received.remove(gap.clone()), + } + } // Final, so no update reopens a stale report it replaced. - Hole::Skip => { - self.reported_skip.insert(gap.clone()); + hole => { + if matches!(hole, Hole::Skip) { + self.reported_skip.insert(gap.clone()); + } self.reported_stale.remove(gap.clone()); + self.reported_received.remove(gap.clone()); } - hole if hole.stale_budget().is_some() => self.reported_stale.insert(gap.clone()), - Hole::Drop { .. } => self.reported_stale.remove(gap.clone()), } return Some((gap, hole)); } @@ -7505,12 +7526,17 @@ impl Cursor { let Some(drops) = &mut self.drops else { return Vec::new(); }; - let runs = drops.take_stale(); + let (runs, received) = drops.take_stale(); // What neither a skip nor a miss offers again: a group written off at the track's max - // age bound, which no budget reverses. It is still stale under the new terms. + // age bound, which no budget reverses. It is still stale under the new terms. What an + // upstream's stale run decided is the upstream's to judge again: the next scan folds + // it in again if the track still holds it, and it stays owed if new terms superseded it. let mut kept = RangeSet::new(); for run in &runs { kept.insert(run.clone()); + for decided in received.overlapping(run) { + kept.remove(decided.clone()); + } if let Some(misses) = &mut self.misses { for missed in misses.given_up().overlapping(run) { kept.remove(missed.clone()); @@ -7526,12 +7552,6 @@ impl Cursor { } let budget = { let state = self.state.read(); - // A received stale run is folded in again by the next scan, if the track holds it. - for run in kept.clone().iter() { - for (dropped, _) in state.drops.overlapping(run) { - kept.remove(dropped.clone()); - } - } clamp_max_delay(self.subscription.read().budget(), state.max_age_bound()) }; if let Some(drops) = &mut self.drops { @@ -22714,6 +22734,30 @@ mod test { assert!(subscriber.poll_drop(&waiter).is_pending(), "5 and 6 are unknown again"); } + /// A received stale run a reader reported and the relay's own update then superseded is + /// unknown again: the reader's update owes it until the upstream serves or drops it again, + /// rather than dropping it again as if the reader had judged it stale itself. + #[test] + #[cfg(feature = "async")] + fn a_superseded_received_stale_report_stays_owed_after_an_update() { + let producer = track_producer("test", Info::timed()); + let budget = Duration::from_millis(500); + let mut subscriber = producer.subscribe( + Subscription::default() + .with_delay(Delay::default().with_max(budget)) + .with_ranges([Range::groups(0..)]), + ); + let waiter = kio::Waiter::noop(); + producer.give_up(1..3, Hole::stale(budget)); + assert_eq!(subscriber.poll_drop(&waiter), Poll::Ready((1..3, Hole::stale(budget)))); + producer.supersede_stale(None); + assert_eq!(subscriber.reoffer_stale(), [1..3]); + assert!(subscriber.poll_drop(&waiter).is_pending(), "no verdict under the new terms"); + // The upstream drops it again under the new terms: reported again. + producer.give_up(1..3, Hole::stale(budget)); + assert_eq!(subscriber.poll_drop(&waiter), Poll::Ready((1..3, Hole::stale(budget)))); + } + /// A bounded reader is settled by a received stale run only when its range reaches where /// the upstream's did; otherwise it still waits for the groups the run names. #[test] From 2a43d68b5daefb0df66b165d066554940b83ad9a Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 00:13:54 -0800 Subject: [PATCH 41/68] test(net): format the superseded stale report test Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/src/model/track.rs | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index 0cc58497b8b..fdc11426f02 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -22739,6 +22739,7 @@ mod test { /// rather than dropping it again as if the reader had judged it stale itself. #[test] #[cfg(feature = "async")] + #[expect(clippy::single_range_in_vec_init, reason = "the reopened runs are a list, here one")] fn a_superseded_received_stale_report_stays_owed_after_an_update() { let producer = track_producer("test", Info::timed()); let budget = Duration::from_millis(500); @@ -22752,7 +22753,10 @@ mod test { assert_eq!(subscriber.poll_drop(&waiter), Poll::Ready((1..3, Hole::stale(budget)))); producer.supersede_stale(None); assert_eq!(subscriber.reoffer_stale(), [1..3]); - assert!(subscriber.poll_drop(&waiter).is_pending(), "no verdict under the new terms"); + assert!( + subscriber.poll_drop(&waiter).is_pending(), + "no verdict under the new terms" + ); // The upstream drops it again under the new terms: reported again. producer.give_up(1..3, Hole::stale(budget)); assert_eq!(subscriber.poll_drop(&waiter), Poll::Ready((1..3, Hole::stale(budget)))); From aad23d91906cda715088e20da3f99f8667c45e68 Mon Sep 17 00:00:00 2001 From: "pullfrog[bot]" <226033991+pullfrog[bot]@users.noreply.github.com> Date: Thu, 5 Nov 2026 08:46:54 +0000 Subject: [PATCH 42/68] test(net): wait for the accepted copy before checking the source start --- js/net/src/lite/tail.test.ts | 3 +++ 1 file changed, 3 insertions(+) diff --git a/js/net/src/lite/tail.test.ts b/js/net/src/lite/tail.test.ts index c7a6f530e3b..0ce5c9f87f1 100644 --- a/js/net/src/lite/tail.test.ts +++ b/js/net/src/lite/tail.test.ts @@ -115,6 +115,9 @@ async function settlesWithin(promise: Promise, ms: number): Promise { test("the source start waits for SUBSCRIBE_START where it resolves the start", async () => { const { reader, respond } = await subscribed(version, GRACE, { start: { absolute: 2n } }); + // The accepted copy registers once the SUBSCRIBE write resolves, which can trail the publisher + // reading it, so let that run before asking where the source starts. + await settle(); expect(hooks.sourceStart(reader)).toBe(resolvesStart(version) ? "pending" : undefined); await respond({ start: new SubscribeStart([3n]) }); await settle(); From a8f9dd7d35338409cfd466a91cfadf280a3d2e85 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 08:56:28 -0800 Subject: [PATCH 43/68] fix(net): a finished group wins over a late stale drop once evicted A lite-07 Group Stream can overtake the OLD drop the publisher sent before serving it, finish, and leave the cache before that OLD lands. The subscriber judged the late OLD against the cache alone, recorded it stale, and the next update owed the group again, which the publisher had already retired. The tail now records the group streams that finished since the last update, and a stale drop under that update leaves them (Rust and JS). Also check a SUBSCRIBE_DROP's Update before its empty range in Rust, as JS does. Co-Authored-By: Claude Opus 5.5 --- js/net/src/lite/subscriber.test.ts | 90 +++++++++++++++++++++++++++++- js/net/src/lite/subscriber.ts | 8 ++- js/net/src/tail.ts | 23 ++++++++ rs/moq-net/src/lite/subscriber.rs | 73 ++++++++++++++++++++++-- rs/moq-net/src/tail.rs | 30 ++++++++++ 5 files changed, 217 insertions(+), 7 deletions(-) diff --git a/js/net/src/lite/subscriber.test.ts b/js/net/src/lite/subscriber.test.ts index 609ad38310f..d8ee7bd5dfe 100644 --- a/js/net/src/lite/subscriber.test.ts +++ b/js/net/src/lite/subscriber.test.ts @@ -21,7 +21,7 @@ import type { Consumer as GroupConsumer } from "../group.ts"; import { HopSchema, isAnonymous, MAX_HOPS, Route, UNKNOWN_HOP } from "../hop.ts"; import * as Instance from "../instance.ts"; import { hooks } from "../internal.ts"; -import { GRACE } from "../ledger.ts"; +import { CACHE_EXPIRY, GRACE } from "../ledger.ts"; import * as Path from "../path.ts"; import { Reader, Writer } from "../stream.ts"; import { Timescale } from "../time.ts"; @@ -2735,6 +2735,94 @@ test("a group wins over its stale drop on lite-07", async () => { } }); +// A group whose stream finished wins over a lite-07 stale drop that lands after the cache evicted +// it: the publisher served it over that drop, so the next update owes it no more. A stream that +// finished under an earlier update says nothing of a drop under the new one. Mirrors the Rust +// `a_finished_group_wins_over_a_late_stale_drop_once_evicted`. +test("a finished group wins over a late stale drop once evicted on lite-07", async () => { + const clock = new Manual(); + const installed = clock.install(); + const version = Version.DRAFT_07; + const { quic, streams, opened } = fakeSession(); + const subscriber = new Subscriber(quic, version, HopSchema.parse(1n)); + type Entry = { + track: Producer; + tail: { + covers(start: bigint, end: bigint): boolean; + open(sequence: bigint): () => void; + finish(sequence: bigint): void; + }; + }; + let entry: Entry | undefined; + const internal = subscriber as unknown as { + runResponses: (stream: unknown, entry: Entry) => Promise; + }; + const runResponses = internal.runResponses.bind(internal); + const responses = spyOn(internal, "runResponses").mockImplementation((stream, captured) => { + entry = captured; + return runResponses(stream, captured); + }); + const broadcast = subscriber.consume(Path.from("room")); + const track = broadcast.track("video").subscribe({ delay: { max: Milli(500) } }); + try { + await opened(0); + await answerTrackInfo(streams[0], version); + const stream = await opened(1); + await stream.reading; + const send = async (...resps: SubscribeResponse[]) => { + const chunks: Uint8Array[] = []; + const writer = new Writer( + new WritableStream({ write: (chunk) => void chunks.push(new Uint8Array(chunk)) }), + version, + ); + for (const resp of resps) await encodeSubscribeResponse(writer, resp, version); + for (const chunk of chunks) stream.inbound.enqueue(chunk); + await drainReactions(); + }; + const old = (sequence: bigint, update: bigint): SubscribeResponse => ({ + drop: new SubscribeDrop({ start: sequence, end: sequence, error: StreamCode.Old, update }), + }); + // A group stream that arrives whole. + const arrive = (sequence: bigint) => { + if (!entry) throw new Error("no subscription"); + const read = entry.tail.open(sequence); + hooks.receiveGroup(entry.track, { sequence, start: 0 }).finish(); + entry.tail.finish(sequence); + read(); + }; + // Everything below the live edge ages out of the cache. + const evict = () => clock.advance(Nano.fromMilli(Milli(2 * CACHE_EXPIRY))); + let priority = 0; + const update = async () => { + track.update({ priority: ++priority, delay: { max: Milli(500) } }); + await drainReactions(); + }; + + await send({ start: new SubscribeStart([0n]) }); + arrive(1n); + arrive(10n); + await evict(); + await send(old(1n, 0n)); + await update(); + if (!entry) throw new Error("no subscription"); + expect(entry.tail.covers(1n, 2n)).toBe(true); + + // Finished under the first update, then judged stale under the second. + arrive(2n); + await update(); + await evict(); + await send(old(2n, 2n)); + await update(); + expect(entry.tail.covers(2n, 3n)).toBe(false); + } finally { + responses.mockRestore(); + track.close(); + broadcast.close(); + subscriber.close(); + installed.close(); + } +}); + // A SKIP, or any final drop, that follows an OLD drop of the same groups replaces it: an update // reopens only stale verdicts, so the groups stay accounted for, and the skip stays final rather // than being superseded with the OLD drop. An empty range names nothing. Mirrors Rust. diff --git a/js/net/src/lite/subscriber.ts b/js/net/src/lite/subscriber.ts index 4bbb287fc3a..aea48c30f14 100644 --- a/js/net/src/lite/subscriber.ts +++ b/js/net/src/lite/subscriber.ts @@ -2158,7 +2158,10 @@ export class Subscriber { entry.tail.account(start, end + 1n); // Judged under the budget this subscriber sent with that update. const hole = stale ? staleHole(entry.maxDelay ?? 0) : { error }; - const recorded = hooks.dropTrackGroups(entry.track, start, end + 1n, hole); + // A group whose stream finished under this update was served over a stale drop, so it + // won, even once the cache evicted it. + const named = stale ? entry.tail.unfinished(start, end + 1n) : [{ start, end: end + 1n }]; + const recorded = named.flatMap((run) => hooks.dropTrackGroups(entry.track, run.start, run.end, hole)); // A final drop is never reopened, even where it replaces a stale one. A sequence // already skipped stays skipped, and one whose group arrived keeps it, so no update // reopens either. @@ -2341,6 +2344,7 @@ export class Subscriber { // unknown again: a reader fills them rather than trust an old verdict. entry.update = update.update; entry.maxDelay = update.maxDelay; + entry.tail.updated(); const held = hooks.supersedeStale(entry.track, judgedEnd(ranges)); // A lite-07 publisher re-offers every group it reported stale on any update, and drops // again under the new number what is still stale, so the tail owes every stale run again. @@ -2451,6 +2455,8 @@ export class Subscriber { }); producer.finish(); + // Only a group that arrived whole, not one a drop or the track ended meanwhile. + if (producer.closed.peek() === null) tail.finish(group.sequence); stream.stop(StreamCode.Cancel); } catch (err: unknown) { // A violation ends the session, which the connection does once it escapes; anything diff --git a/js/net/src/tail.ts b/js/net/src/tail.ts index a948f6f0807..1b87fb6e44d 100644 --- a/js/net/src/tail.ts +++ b/js/net/src/tail.ts @@ -1,5 +1,6 @@ import { type Milli, Nano, now, sleep } from "@moq/clock"; import { type GetPromise, Signal } from "@moq/signals"; +import { insertRun, type Run, runGaps } from "./internal.ts"; import { GRACE, Ledger } from "./ledger.ts"; /** How a {@link Tail} measures its grace. */ @@ -29,6 +30,10 @@ export class Tail { #active = 0; #changed = new Signal(0); #grace: () => Milli; + // Groups whose stream finished since the last update was sent. A lite-07 stale drop under that + // update names one only if it was sent before the publisher served the group, so the group won + // over it, even once the cache evicted it. + #finished: Run[] = []; constructor(options: TailOptions = {}) { this.#grace = options.grace ?? (() => GRACE); @@ -89,6 +94,24 @@ export class Tail { }; } + /** Record that the group stream for `sequence` arrived whole. */ + finish(sequence: bigint): void { + insertRun(this.#finished, sequence, sequence + 1n); + } + + /** The parts of `[start, end)` whose stream did not finish since the last update was sent. */ + unfinished(start: bigint, end: bigint): Run[] { + return runGaps(this.#finished, start, end); + } + + /** + * Forget which streams finished, for an update being sent: a stale drop under an earlier one is + * no verdict, so none needs them. + */ + updated(): void { + this.#finished = []; + } + /** Re-check what {@link settle} waits on, after something outside the ledger changed. */ touch(): void { this.#bump(); diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index 90de2b0ba5b..275ba97ab04 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -1490,7 +1490,9 @@ impl GroupRecv { let res = match res { Ok(()) => { - let _ = group.finish(); + if group.finish().is_ok() { + reading.finish(); + } Ok(()) } Err(err @ (Error::Cancel | Error::Stream(crate::StreamError::Cancel))) => { @@ -5162,6 +5164,61 @@ mod tests { frame(&mut three).expect("the update leaves a group still arriving"); } + /// A group whose stream finished wins over a lite-07 stale drop that lands after the cache + /// evicted it: the publisher served it over that drop, so the next update owes it no more. + /// A stream that finished under an earlier update says nothing of a drop under the new one. + #[kio::test] + async fn a_finished_group_wins_over_a_late_stale_drop_once_evicted() { + let version = Version::Lite07; + let (serve, mut serving, tail, _broadcast) = lite07_serving(Duration::from_millis(500)).await; + let now = || serve.subscriber.runtime.now(); + let code = u64::from(crate::coding::StreamCodes::encode_stream_code( + &version, + &crate::StreamError::Old, + )); + let old = |sequence, update| { + lite::SubscribeResponse::Drop(lite::SubscribeDrop { + start: sequence, + end: sequence, + error: code, + update, + }) + }; + let arrive = |serving: &mut ServeLoop, sequence| { + let mut group = serving.serving.create_group(crate::group::Info { sequence }).unwrap(); + let reading = Reading::open(&tail, Some(sequence), now()); + group + .write_frame(crate::Timestamp::from_millis(1).unwrap(), &b"x"[..]) + .unwrap(); + group.finish().unwrap(); + reading.finish(); + serving.serving.evict(sequence); + }; + let update = |serving: &mut ServeLoop| { + let Sub::Active(active) = &mut serving.sub else { + unreachable!("an active subscription"); + }; + let ranges = active.live.unchanged(); + buffer_update(active, ranges, &serving.serving, now()).unwrap(); + }; + let covers = |groups: std::ops::Range| tail.read().covers(groups); + + arrive(&mut serving, 1); + serving.apply_response(&serve, old(1, 0)).unwrap(); + update(&mut serving); + assert!(covers(1..2), "the evicted group that arrived stays accounted for"); + + // Finished under the first update, then judged stale under the second. + arrive(&mut serving, 2); + update(&mut serving); + serving.apply_response(&serve, old(2, 2)).unwrap(); + update(&mut serving); + assert!( + !covers(2..3), + "a drop under a later update is the publisher's to re-offer" + ); + } + /// A SKIP, or any final drop, that follows an OLD drop of the same groups replaces it: an /// update reopens only stale verdicts, so the groups stay accounted for, and the skip /// stays final rather than being superseded with the OLD drop. An empty range names @@ -9625,6 +9682,7 @@ fn buffer_update( let held = serving.supersede_stale(judged_end(active.live.iter().map(|(_, range)| range))); let recorded = std::mem::take(&mut active.stale); if let Ok(mut tail) = active.tail.write() { + tail.updated(); for run in recorded.iter() { for reopened in held.overlapping(run) { tail.unaccount(reopened.start.max(run.start)..reopened.end.min(run.end), now); @@ -10942,15 +11000,15 @@ impl ServeLoop { // lite-07 stale drop leaves a group whose stream arrived, // since the stream wins over it. lite::SubscribeResponse::Drop(dropped) => { + // It can only echo an update this subscriber sent, even when it names nothing. + if dropped.update > active.update { + return Err(Error::ProtocolViolation); + } let groups = dropped.start..dropped.end.saturating_add(1); // An empty range (Group Start = Group End + 1) names nothing. if groups.is_empty() { return Ok(()); } - // It can only echo an update this subscriber sent. - if dropped.update > active.update { - return Err(Error::ProtocolViolation); - } let error = dropped.error(serve.subscriber.version); // On lite-07 an OLD drop is a stale verdict, which the next update reopens; every // other drop is final, whatever update it names. @@ -10973,6 +11031,11 @@ impl ServeLoop { for asked in active.fill_groups.overlapping(&groups) { given.remove(asked.clone()); } + // A group whose stream finished under this update was served over + // the drop, so it won, even once the cache evicted it. + for finished in active.tail.read().finished(&groups) { + given.remove(finished.clone()); + } for given in given.iter() { let recorded = self .serving diff --git a/rs/moq-net/src/tail.rs b/rs/moq-net/src/tail.rs index 36e6791f55d..2b65cd14525 100644 --- a/rs/moq-net/src/tail.rs +++ b/rs/moq-net/src/tail.rs @@ -12,6 +12,8 @@ use std::{ops::Range, task::Poll, time::Duration}; +use rangemap::RangeSet; + pub(crate) use crate::ledger::{GRACE, grace}; use crate::{ ledger::Ledger, @@ -28,6 +30,10 @@ pub(crate) struct Tail { // The publisher's FIN ended the track, its only end before lite-05, so a session that // dies before the tail settles declares that end before aborting. ended: bool, + // Groups whose stream finished since the last update was sent. A lite-07 stale drop under + // that update names one only if it was sent before the publisher served the group, so the + // group won over it, even once the cache evicted it. + finished: RangeSet, } impl Default for Tail { @@ -44,6 +50,7 @@ impl Tail { streams: 0, active: 0, ended: false, + finished: RangeSet::new(), } } @@ -103,6 +110,17 @@ impl Tail { self.ledger.folds_at(group, self.ledger.grace()) } + /// The runs of `groups` whose stream finished since the last update was sent. + pub fn finished(&self, groups: &Range) -> impl Iterator> + '_ { + self.finished.overlapping(groups) + } + + /// Forget which streams finished, for an update being sent: a stale drop under an earlier + /// one is no verdict, so none needs them. + pub fn updated(&mut self) { + self.finished = RangeSet::new(); + } + /// Whether a data stream whose header arrived is still being read. pub fn is_reading(&self) -> bool { self.active > 0 @@ -117,6 +135,7 @@ impl Tail { /// A data stream whose header arrived, holding its subscription's end open until dropped. pub(crate) struct Reading { tail: Option>, + group: Option, active: bool, } @@ -126,6 +145,7 @@ impl Reading { let Ok(mut state) = tail.write() else { return Self { tail: None, + group, active: false, }; }; @@ -136,10 +156,20 @@ impl Reading { } Self { tail: Some(tail.clone()), + group, active: true, } } + /// Record that the stream's group arrived whole, once it finished. + pub fn finish(&self) { + if let (Some(tail), Some(group)) = (&self.tail, self.group) + && let Ok(mut state) = tail.write() + { + state.finished.insert(group..group.saturating_add(1)); + } + } + /// Stop holding the end open while the stream waits on another one, which holds it /// open itself if its header arrived. pub fn park(&mut self) { From 947e05728c13c5be7161bb7ff6aa3f7ee7e51574 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 08:56:28 -0800 Subject: [PATCH 44/68] quest: confirm the drop set's agent decisions 14 and 16 to 25 Co-Authored-By: Claude Opus 5.5 --- quest/m1/drop-set.md | 40 +++++++++++++++++++++++++++++++++++++--- 1 file changed, 37 insertions(+), 3 deletions(-) diff --git a/quest/m1/drop-set.md b/quest/m1/drop-set.md index 8642052197d..96125559e57 100644 --- a/quest/m1/drop-set.md +++ b/quest/m1/drop-set.md @@ -21,8 +21,10 @@ only for readers it covers (`Drift::settles`, `settles` in (`supersede_stale`), and every lite-07 update re-offering what was reported stale (`reoffer_stale`). It kept today's storage: `TrackState::drops`, `DropCursor` with its `reported` and `reported_stale` sets, `Skipped`, -`Delivered`, `TrackRun::served`, and the JS `drops` signal, `#pending`, -`#reported`, and `#reportedStale`, none billed to anything. This quest is +`Delivered`, `TrackRun::served`, the lite subscriber's record of the group +streams that finished since its last update (`Tail::finished`, JS +`Tail#finished`), and the JS `drops` signal, `#pending`, `#reported`, and +`#reportedStale`, none billed to anything. This quest is what remains: bounding them. Decided 2026-11-03 (maintainer, /quest-unblock): `js/net` bounds run memory @@ -137,7 +139,9 @@ decision above and the PR's `## Waiting` item on out-of-order creation: neither sends OLD for it again nor names it at the end. On the receiving side a Group Stream wins over a stale drop of its sequence in either order: an OLD that lands after the stream leaves the group, still arriving or - not, and an update owes again only the stale runs the track still holds. + not, or finished under the same update and since evicted (agent decision + 26, 2026-11-05, for review), and an update owes again only the stale runs + the track still holds. The end sweep's NOT_FOUND never replaces a stale drop, so an update can still reach an ended track's history. Not chosen: reverting the replacement; merging with the stale report left in place. @@ -193,6 +197,36 @@ sees it fail, since no sequence is handed out twice; the cache, fills, and readers that had not reached it get the re-offer. Not chosen: letting the duplicate replace the copy still arriving; accepting the loss. +Decided 2026-11-05 (maintainer, /quest-plan): the agent decisions +[moq-dev/moq#7670](https://github.com/moq-dev/moq/pull/7670) listed for +review stand as recorded, numbered as in its body: + +- ✅ **14:** a Group Stream or final drop the publisher's reader reaches + replaces a stale drop it already sent (above). +- ✅ **16:** SUBSCRIBE_UPDATE carries `Update` after the Range edit and before + `Wanted Group`; SUBSCRIBE_SKIP keeps 0x3 beside SUBSCRIBE_STARTED at 0x4. +- ✅ **17:** an OLD drop records no stale run on groups a range fill asked + for, which recovery asks for again. Every drop the subscriber accounts, + and every SKIP, answers the fills owing those groups; an OLD drop under a + superseded `Update` answers none. +- ✅ **18:** the JS covered-FIN reopen loop and its stale-run supersede go, + as main deleted the loop. +- ✅ **19:** a final OLD drop, such as one relayed from lite-06, goes out on + lite-07 as the stale drop its code names, recorded and dropped again on + each update. +- ✅ **20:** an update judges again the re-offers still waiting for stream + credit, and a verdict sent for a queued re-offer keeps it from opening. +- ✅ **21:** a SUBSCRIBE_DROP naming an `Update` later than the last one sent + is a PROTOCOL_VIOLATION. +- ✅ **22:** a received stale run is judged again for a reader whose terms + changed, so a widened budget asks a fill for it. +- ✅ **23:** recording a stale verdict answers no outstanding fill (JS, as + Rust), `skipGroups` checks its bounds, and a fill cannot cache a skipped + sequence. +- ✅ **25:** a relay cursor's stale report that an upstream's received stale + run decided stays the upstream's to judge after an update + (`DropCursor::reported_received`, JS `#reportedReceived`). + Decided 2026-10-27 (maintainer): this runs after the range count cost work, which finished in [moq-dev/moq#7542](https://github.com/moq-dev/moq/pull/7542), and before [Rust groups by start](/quest/m1/time-addressing/model.md), which From 5408c041989ec59df5571756ecd8c8255bdc4a59 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 09:02:57 -0800 Subject: [PATCH 45/68] fix(net): return the tail's finished runs by value Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/src/lite/subscriber.rs | 4 ++-- rs/moq-net/src/tail.rs | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index 275ba97ab04..893bde783a4 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -11033,8 +11033,8 @@ impl ServeLoop { } // A group whose stream finished under this update was served over // the drop, so it won, even once the cache evicted it. - for finished in active.tail.read().finished(&groups) { - given.remove(finished.clone()); + for finished in active.tail.read().finished(groups.clone()) { + given.remove(finished); } for given in given.iter() { let recorded = self diff --git a/rs/moq-net/src/tail.rs b/rs/moq-net/src/tail.rs index 2b65cd14525..a5cb88c1778 100644 --- a/rs/moq-net/src/tail.rs +++ b/rs/moq-net/src/tail.rs @@ -111,8 +111,8 @@ impl Tail { } /// The runs of `groups` whose stream finished since the last update was sent. - pub fn finished(&self, groups: &Range) -> impl Iterator> + '_ { - self.finished.overlapping(groups) + pub fn finished(&self, groups: Range) -> impl Iterator> + '_ { + self.finished.overlapping(groups).cloned() } /// Forget which streams finished, for an update being sent: a stale drop under an earlier From b005f5cb9818e453f37a89f004ee0744945104d3 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 09:15:43 -0800 Subject: [PATCH 46/68] fix(net): record finished group streams only on lite-07 Co-Authored-By: Claude Opus 5.5 --- js/net/src/lite/subscriber.ts | 5 +++-- rs/moq-net/src/lite/subscriber.rs | 3 ++- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/js/net/src/lite/subscriber.ts b/js/net/src/lite/subscriber.ts index aea48c30f14..8b6d26173c3 100644 --- a/js/net/src/lite/subscriber.ts +++ b/js/net/src/lite/subscriber.ts @@ -2455,8 +2455,9 @@ export class Subscriber { }); producer.finish(); - // Only a group that arrived whole, not one a drop or the track ended meanwhile. - if (producer.closed.peek() === null) tail.finish(group.sequence); + // Only a group that arrived whole, not one a drop or the track ended meanwhile, and only + // where a lite-07 stale drop asks for it. + if (hasSkip(this.version) && producer.closed.peek() === null) tail.finish(group.sequence); stream.stop(StreamCode.Cancel); } catch (err: unknown) { // A violation ends the session, which the connection does once it escapes; anything diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index 893bde783a4..76d1dd38eae 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -1490,7 +1490,8 @@ impl GroupRecv { let res = match res { Ok(()) => { - if group.finish().is_ok() { + // Only a lite-07 stale drop asks which groups arrived whole. + if group.finish().is_ok() && self.subscriber.version.has_skip() { reading.finish(); } Ok(()) From 44d32a9f4a34fd365652a42f187e5ad82df14bc9 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 10:12:52 -0800 Subject: [PATCH 47/68] test: drive the finished-stream record through the group stream path The decision 26 regressions finished the tail by hand, so dropping the record from runGroup / GroupRecv left them green. Deliver each group over a real stream instead. Co-Authored-By: Claude Opus 5.5 --- js/net/src/lite/subscriber.test.ts | 22 ++++++-------- rs/moq-net/src/lite/subscriber.rs | 46 +++++++++++++++++++++++------- 2 files changed, 45 insertions(+), 23 deletions(-) diff --git a/js/net/src/lite/subscriber.test.ts b/js/net/src/lite/subscriber.test.ts index d8ee7bd5dfe..6b1c09a7765 100644 --- a/js/net/src/lite/subscriber.test.ts +++ b/js/net/src/lite/subscriber.test.ts @@ -2749,8 +2749,6 @@ test("a finished group wins over a late stale drop once evicted on lite-07", asy track: Producer; tail: { covers(start: bigint, end: bigint): boolean; - open(sequence: bigint): () => void; - finish(sequence: bigint): void; }; }; let entry: Entry | undefined; @@ -2782,14 +2780,12 @@ test("a finished group wins over a late stale drop once evicted on lite-07", asy const old = (sequence: bigint, update: bigint): SubscribeResponse => ({ drop: new SubscribeDrop({ start: sequence, end: sequence, error: StreamCode.Old, update }), }); - // A group stream that arrives whole. - const arrive = (sequence: bigint) => { - if (!entry) throw new Error("no subscription"); - const read = entry.tail.open(sequence); - hooks.receiveGroup(entry.track, { sequence, start: 0 }).finish(); - entry.tail.finish(sequence); - read(); - }; + // A group stream that arrives whole, through the path that records it. + const arrive = (sequence: bigint) => + subscriber.runGroup( + new Group({ subscribe: 0n, sequence }), + new Reader(new ReadableStream({ start: (controller) => controller.close() }), undefined, version), + ); // Everything below the live edge ages out of the cache. const evict = () => clock.advance(Nano.fromMilli(Milli(2 * CACHE_EXPIRY))); let priority = 0; @@ -2799,8 +2795,8 @@ test("a finished group wins over a late stale drop once evicted on lite-07", asy }; await send({ start: new SubscribeStart([0n]) }); - arrive(1n); - arrive(10n); + await arrive(1n); + await arrive(10n); await evict(); await send(old(1n, 0n)); await update(); @@ -2808,7 +2804,7 @@ test("a finished group wins over a late stale drop once evicted on lite-07", asy expect(entry.tail.covers(1n, 2n)).toBe(true); // Finished under the first update, then judged stale under the second. - arrive(2n); + await arrive(2n); await update(); await evict(); await send(old(2n, 2n)); diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index 76d1dd38eae..28bb073d495 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -5185,14 +5185,40 @@ mod tests { update, }) }; - let arrive = |serving: &mut ServeLoop, sequence| { - let mut group = serving.serving.create_group(crate::group::Info { sequence }).unwrap(); - let reading = Reading::open(&tail, Some(sequence), now()); - group - .write_frame(crate::Timestamp::from_millis(1).unwrap(), &b"x"[..]) - .unwrap(); - group.finish().unwrap(); - reading.finish(); + serve.subscriber.subscribes.lock().insert( + 0, + TrackEntry::accepted( + serving.serving.clone(), + lite::Framing { + timescale: Some(Timescale::MILLI), + ietf: None, + compressed: false, + }, + tail.clone(), + ), + ); + // A group stream that arrives whole, through the path that records it, then is evicted. + let arrive = async |serving: &mut ServeLoop, sequence| { + use crate::transport::poll::Session as _; + + let mut script = Vec::new(); + let mut encoder = crate::coding::Encoder::new(&mut script, version.into()); + lite::Group { + subscribe: 0, + sequence, + frame_start: 0, + } + .encode(&mut encoder, version) + .unwrap(); + encoder.varint(crate::coding::varint::zigzag(1)).unwrap(); + encoder.bytes(b"x").unwrap(); + let session = crate::lite::test_transport::ScriptedSession::eof(script); + let (_, recv) = session.open_bi(Default::default()).await.unwrap(); + let mut group = GroupRecv::new( + serve.subscriber.clone(), + Reader::new(crate::transport::boxed::RecvStream::new(recv), version), + ); + kio::wait(|waiter| group.poll_serve(waiter)).await.unwrap(); serving.serving.evict(sequence); }; let update = |serving: &mut ServeLoop| { @@ -5204,13 +5230,13 @@ mod tests { }; let covers = |groups: std::ops::Range| tail.read().covers(groups); - arrive(&mut serving, 1); + arrive(&mut serving, 1).await; serving.apply_response(&serve, old(1, 0)).unwrap(); update(&mut serving); assert!(covers(1..2), "the evicted group that arrived stays accounted for"); // Finished under the first update, then judged stale under the second. - arrive(&mut serving, 2); + arrive(&mut serving, 2).await; update(&mut serving); serving.apply_response(&serve, old(2, 2)).unwrap(); update(&mut serving); From c7640b9f23a27066d967f48c4a25a51a3e414b78 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 5 Nov 2026 23:44:21 -0800 Subject: [PATCH 48/68] fix(net): a queued drop poll keeps the last drift, and draft clarifications A queue-only poll judged no drift, so it changed the drop scan key and unsettled the received stale runs the probing poll had just settled, rescanning history and missing them until the next probe. The cursor now keeps the last drift it judged. Draft: an undeclared gap past the budget is OLD, an OLD drop reopens on the same subscription, and an update re-offers only what is no longer stale. Allow give_up to be unused without the async feature. Co-Authored-By: Claude Opus 5.5 --- drafts/draft-lcurley-moq-lite.md | 8 +++--- quest/m1/drop-set.md | 5 ++-- rs/moq-net/src/model/track.rs | 42 +++++++++++++++++++++++++++----- 3 files changed, 43 insertions(+), 12 deletions(-) diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index a47ccce4c3b..18bd4218d06 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -344,7 +344,7 @@ Sent when resetting a stream (RESET_STREAM), or when refusing to receive one (ST | ------- | ------------- | ----------- | | 0x32 | GROUP_TOO_LARGE | The group grew past the publisher's cache budget and was aborted. | | ------- | ------------- | ----------- | -| 0x33 | NOT_FOUND | The requested group, track, or broadcast is not here. A sequence its publisher declared it will never produce is named by SUBSCRIBE_SKIP instead; any other sequence a publisher never had, including a gap it never declared, is named with NOT_FOUND. | +| 0x33 | NOT_FOUND | The requested group, track, or broadcast is not here. A sequence its publisher declared it will never produce is named by SUBSCRIBE_SKIP instead; any other sequence a publisher never had, including a gap it never declared, is named with NOT_FOUND unless it is past the subscriber's budget (see [SUBSCRIBE_DROP](#subscribe-drop)). | | ------- | ------------- | ----------- | | 0x34 | OLD | The group was superseded by a newer group and dropped. In SUBSCRIBE_DROP, the next SUBSCRIBE_UPDATE reopens it: the publisher re-offers the group or drops it again. | | ------- | ------------- | ----------- | @@ -571,8 +571,8 @@ Drops and Groups name absolute sequences, so the subscriber judges them against The publisher never closes its side first: a subscription whose every Range has an `End` other than None and whose every Group has been delivered, skipped, or dropped stays open for an update to widen it. A subscriber fills Groups it misses by asking for them as Ranges, adding them to an open subscription with SUBSCRIBE_UPDATE or opening a bounded SUBSCRIBE when none is open, rather than with a FETCH of each Group; FETCH remains for one Group, or a slice of one. -A publisher never sends a Group twice on one subscription, so a Group the subscription already delivered or named in SUBSCRIBE_DROP is asked for with a SUBSCRIBE of its own. -The subscriber asks that way, once, for a Group a subscription dropped with OLD that it still wants, with a `Subscriber Max Age` its age does not reach, and never for a Group that may still arrive on the existing subscription. +A publisher never sends a Group twice on one subscription, except to re-offer one whose stream it reset, so a Group the subscription already delivered or named in a final SUBSCRIBE_DROP is asked for with a SUBSCRIBE of its own. +A Group dropped with OLD that the subscriber still wants needs no new SUBSCRIBE: a SUBSCRIBE_UPDATE with a `Subscriber Max Age` its age does not reach reopens it on the same subscription (see [SUBSCRIBE_UPDATE](#subscribe-update)). The subscriber closes its side (FIN) once it wants nothing more from the subscription, and the publisher then closes its side (FIN), cancelling any Group Stream still open. After SUBSCRIBE_FINISH, the subscriber SHOULD close its side once its tail accounting has settled: every received Group Stream has been read to its FIN, reset, or dropped, and every sequence in range has arrived, been named by SUBSCRIBE_SKIP or SUBSCRIBE_DROP, or exhausted the subscriber's grace period. @@ -1531,7 +1531,7 @@ Every field but the Ranges restates the subscription, and the latest one applies The Ranges are an edit, growing or shrinking the subscription: the update retires the live Ranges it names by Range ID, then adds its own, each taking the next Range ID. Lowering a `Start` covers the groups below the old one that the publisher still holds, not only those that arrive later; a group already sent is not sent again. Growing the subscription asks for what it newly covers: the publisher MUST serve the newly covered Groups it holds and never delivered on this subscription. -Every SUBSCRIBE_UPDATE, whatever it changes, reopens every stale (OLD) SUBSCRIBE_DROP sent before it: the publisher MUST serve again the Groups it named in one that it still holds and the new terms cover, and MUST name again, in a SUBSCRIBE_DROP carrying the new `Update`, those still stale. +Every SUBSCRIBE_UPDATE, whatever it changes, reopens every stale (OLD) SUBSCRIBE_DROP sent before it: the publisher MUST serve each Group named in one that it still holds, that the new terms cover, and that is no longer stale under them, and MUST name each that is still stale in a SUBSCRIBE_DROP carrying the new `Update`. A SUBSCRIBE_DROP with any other Error Code is final, and no update reopens it. A SUBSCRIBE_UPDATE never resets the subscription, since a reset would race Group Streams already in flight, and shrinking it sends no SUBSCRIBE_SKIP or SUBSCRIBE_DROP. diff --git a/quest/m1/drop-set.md b/quest/m1/drop-set.md index ee6b64e97f2..db190c099c7 100644 --- a/quest/m1/drop-set.md +++ b/quest/m1/drop-set.md @@ -147,8 +147,9 @@ decision above and the PR's `## Waiting` item on out-of-order creation: replacement; merging with the stale report left in place. - ✅ **`append_group` is an implicit skip** (refinement, same session): it means "the next group", so the producer declares it won't go back, and it - skips every sequence below it that never arrived and wasn't already given - up, as if `skip_groups` were called on them; those are then final. Today + skips every sequence below it, down to the lowest group created since the + last append, that it never produced, as if `skip_groups` were called on + them: a drop there becomes a skip, and they are then final. Today `append_group` leaves no gap of its own, so this only fires on holes earlier `create_group` calls left below the head; under time addressing it names the span between consecutive starts. `create_group` stays non-committal, diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index 3688a775a58..ba8f34f9247 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -3893,6 +3893,7 @@ impl Producer { /// Record `groups` given up as `hole` describes, returning the runs recorded: a drop /// leaves a skipped sequence skipped, and a stale verdict leaves a group that arrived and /// was not aborted, since a group stream wins over its drop. + #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] pub(crate) fn give_up(&self, groups: ops::Range, hole: Hole) -> RangeSet { let error = match &hole { Hole::Drop { error, .. } => error.clone(), @@ -7050,6 +7051,10 @@ struct DropCursor { /// What `pending` last folded in; see [`Scan`]. #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] scanned: Option, + /// The budget and coverage of the last drift judged, which settle received stale runs + /// until the next: a queue-only poll judges none, and must not unsettle them. + #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] + drift: Option<(Duration, Coverage)>, /// Every sequence this cursor already reported. #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] reported: RangeSet, @@ -7075,6 +7080,7 @@ impl DropCursor { cuts: state.cache.watch_cuts(), pending: RangeMap::new(), scanned: None, + drift: None, reported: RangeSet::new(), reported_stale: RangeSet::new(), reported_received: RangeSet::new(), @@ -7152,10 +7158,11 @@ impl DropCursor { } // Rescan only after the drops change, the window moves, or what settles a stale run // does; out-of-window and unsettling runs were skipped and may be owed now. - let settles = within - .drift - .map(|drift| (drift.budget, drift.coverage, state.judged_end)); - let scan = (state.drops_revision, span.clone(), within.revision, settles); + if let Some(drift) = within.drift { + self.drift = Some((drift.budget, drift.coverage)); + } + let judged = self.drift.map(|(budget, coverage)| (budget, coverage, state.judged_end)); + let scan = (state.drops_revision, span.clone(), within.revision, judged); if self.scanned.as_ref() != Some(&scan) { self.scanned = Some(scan); let stale: Vec<_> = self @@ -7170,8 +7177,8 @@ impl DropCursor { for (dropped, hole) in state.drops.overlapping(span.clone()) { let run = dropped.start.max(span.start)..dropped.end.min(span.end); // A received stale run that does not settle this reader is unknown to it. - let settled = match within.drift { - Some(drift) => drift.settles(state, run.start, hole), + let settled = match self.drift { + Some((budget, coverage)) => settles(budget, coverage, run.start, hole, state.judged_end), None => hole.stale_budget().is_none(), }; if settled { @@ -22568,6 +22575,29 @@ mod test { ); } + /// A queue-only poll, which judges no drift, still reports the received stale runs the + /// last probing poll found settled, rather than forgetting them until the next probe. + #[test] + #[cfg(feature = "async")] + fn a_queued_drop_poll_keeps_the_last_drift() { + let producer = track_producer("test", Info::timed()); + let budget = Duration::from_millis(500); + let mut subscriber = producer.subscribe( + Subscription::default() + .with_delay(Delay::default().with_max(budget)) + .with_ranges([Range::groups(0..)]), + ); + producer.give_up(1..2, Hole::stale(budget)); + producer.give_up(3..4, Hole::stale(budget)); + let waiter = kio::Waiter::noop(); + assert_eq!(subscriber.poll_drop(&waiter), Poll::Ready((1..2, Hole::stale(budget)))); + assert_eq!( + subscriber.poll_queued_drop(&waiter), + Poll::Ready((3..4, Hole::stale(budget))) + ); + assert!(subscriber.poll_queued_drop(&waiter).is_pending()); + } + /// A stale run that settled a reader is no verdict once the reader's budget widens past /// the one it was judged under: the wider reader asks a fill for it, as one that was /// always that wide does. From 6cc305be3e00ec9a5940879c5d227cf3687a1cf9 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 6 Nov 2026 00:02:54 -0800 Subject: [PATCH 49/68] docs(net): state where append_group's implicit skip starts It starts just past the previous append, or before the first append at the lowest group created, and replaces a drop there. Co-Authored-By: Claude Opus 5.5 --- js/net/src/track.ts | 8 ++++---- rs/moq-net/src/model/track.rs | 10 +++++----- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/js/net/src/track.ts b/js/net/src/track.ts index 0b58af09190..bef61190cad 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -1799,8 +1799,8 @@ export class Producer { * Append a new group with the next sequence number: one past this producer's own writes. * * Appending says the producer will not go back, so it skips every sequence below the new - * group that it never produced, from the lowest it produced since its last append, as - * {@link skipGroups} would, replacing any drop there. Those sequences are then final: creating one + * group that it never produced, from just past its previous append (before the first, from + * the lowest group it created), as {@link skipGroups} would, replacing any drop there. Those sequences are then final: creating one * throws {@link Skipped}. Appending alone never leaves such a gap; one left by out-of-order * {@link createGroup} calls is what it names. * @@ -1873,8 +1873,8 @@ export class Producer { } // Settle an append at `sequence`: skip every sequence below it this track's own publisher - // never produced, from the lowest it produced since its last append, since appending says it - // will not go back. The skip replaces a drop given there, as `skipGroups` does. + // never produced, from just past its previous append (before the first, from the lowest it + // produced), since appending says it will not go back. The skip replaces a drop given there, as `skipGroups` does. #settleAppend(sequence: bigint): void { const from = this.#appended ?? this.#produced[0]?.start; if (from !== undefined) { diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index ba8f34f9247..90f8472ba01 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -1206,8 +1206,8 @@ impl TrackState { } /// Settle an append at `sequence`: skip every sequence below it that this track's own - /// publisher never produced, from the lowest it produced since its last append, since - /// appending says it will not go back. The skip replaces a drop given there, as + /// publisher never produced, from just past its previous append (before the first, from + /// the lowest it produced), since appending says it will not go back. The skip replaces a drop given there, as /// [`Self::skip_groups`] does. fn appended(&mut self, sequence: u64) { if let Some(from) = self.appended.or(self.produced.iter().next().map(|run| run.start)) @@ -3645,9 +3645,9 @@ impl Producer { /// Create a new group with the next sequence number: one past this producer's own writes. /// /// Appending says the producer will not go back, so it skips every sequence below the new - /// group that it never produced and did not give up, from the lowest it produced since - /// its last append, as [`Self::skip_groups`] would. Those sequences are then final: creating - /// one is refused. Appending alone never leaves such a gap; one left by out-of-order + /// group that it never produced, from just past its previous append (before the first, + /// from the lowest group it created), as [`Self::skip_groups`] would, replacing any drop + /// there. Those sequences are then final: creating one is refused. Appending alone never leaves such a gap; one left by out-of-order /// [`Self::create_group`] calls is what it names. /// /// The broadcast remembers nothing of an earlier producer of the name, so a replacement From efb9123f9b7516f93fc8e22b95a797f447bac6e7 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 6 Nov 2026 01:58:06 -0800 Subject: [PATCH 50/68] style: format the merge and drift fix Co-Authored-By: Claude Opus 5.5 --- js/net/src/ietf/subscriber.ts | 9 +++------ rs/moq-net/src/model/track.rs | 4 +++- 2 files changed, 6 insertions(+), 7 deletions(-) diff --git a/js/net/src/ietf/subscriber.ts b/js/net/src/ietf/subscriber.ts index f1cb6fa3ac5..aa12891b552 100644 --- a/js/net/src/ietf/subscriber.ts +++ b/js/net/src/ietf/subscriber.ts @@ -1858,12 +1858,9 @@ export class Subscriber { }; // A group that will never arrive whole is dropped, so readers stop owing it. const dropHeadless = () => - hooks.dropTrackGroups( - track, - group.groupId, - group.groupId + 1n, - { error: new StreamError(StreamCode.NotFound, { message: "the group's head was lost" }) }, - ); + hooks.dropTrackGroups(track, group.groupId, group.groupId + 1n, { + error: new StreamError(StreamCode.NotFound, { message: "the group's head was lost" }), + }); // After an END_OF_GROUP or END_OF_TRACK status only the stream's end may follow: an // object past the group's final one makes the track malformed. const last = async () => { diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index 90f8472ba01..8b06df9aaf2 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -7161,7 +7161,9 @@ impl DropCursor { if let Some(drift) = within.drift { self.drift = Some((drift.budget, drift.coverage)); } - let judged = self.drift.map(|(budget, coverage)| (budget, coverage, state.judged_end)); + let judged = self + .drift + .map(|(budget, coverage)| (budget, coverage, state.judged_end)); let scan = (state.drops_revision, span.clone(), within.revision, judged); if self.scanned.as_ref() != Some(&scan) { self.scanned = Some(scan); From 7875affcba8e3ca423a72ad34ad45202418d5c89 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 6 Nov 2026 02:05:00 -0800 Subject: [PATCH 51/68] fix(net): main's headless IETF group drop records a Hole Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/src/ietf/subscriber.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rs/moq-net/src/ietf/subscriber.rs b/rs/moq-net/src/ietf/subscriber.rs index 5638b35bb9a..e9f7962e092 100644 --- a/rs/moq-net/src/ietf/subscriber.rs +++ b/rs/moq-net/src/ietf/subscriber.rs @@ -6106,7 +6106,7 @@ impl OpenGroup { { tracing::warn!(sequence, start, "no fill to stitch a mid-group stream onto"); // Never arriving whole, the group is dropped so readers stop owing it. - track.drop_groups(sequence..sequence + 1, crate::StreamError::NotFound); + track.give_up(sequence..sequence + 1, crate::track::Hole::drop(crate::StreamError::NotFound)); return Poll::Ready(Err(Error::Unsupported)); } return Poll::Ready(Ok(Opened::Group(create(track)?, start))); @@ -6149,7 +6149,7 @@ impl OpenGroup { (Some(start), Some((producer, _))) => Ok(Opened::Group(producer, start)), (Some(start), None) => { tracing::warn!(sequence, start, "no fill to stitch a mid-group stream onto"); - track.drop_groups(sequence..sequence + 1, crate::StreamError::NotFound); + track.give_up(sequence..sequence + 1, crate::track::Hole::drop(crate::StreamError::NotFound)); Err(Error::Unsupported) } (None, Some((producer, next))) => Ok(Opened::Group(producer, next)), From 08629581834caed4914650729c5b13f16130bc53 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 6 Nov 2026 02:12:58 -0800 Subject: [PATCH 52/68] style: format the IETF call-site fix Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/src/ietf/subscriber.rs | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/rs/moq-net/src/ietf/subscriber.rs b/rs/moq-net/src/ietf/subscriber.rs index e9f7962e092..0d908d8c8c7 100644 --- a/rs/moq-net/src/ietf/subscriber.rs +++ b/rs/moq-net/src/ietf/subscriber.rs @@ -6106,7 +6106,10 @@ impl OpenGroup { { tracing::warn!(sequence, start, "no fill to stitch a mid-group stream onto"); // Never arriving whole, the group is dropped so readers stop owing it. - track.give_up(sequence..sequence + 1, crate::track::Hole::drop(crate::StreamError::NotFound)); + track.give_up( + sequence..sequence + 1, + crate::track::Hole::drop(crate::StreamError::NotFound), + ); return Poll::Ready(Err(Error::Unsupported)); } return Poll::Ready(Ok(Opened::Group(create(track)?, start))); @@ -6149,7 +6152,10 @@ impl OpenGroup { (Some(start), Some((producer, _))) => Ok(Opened::Group(producer, start)), (Some(start), None) => { tracing::warn!(sequence, start, "no fill to stitch a mid-group stream onto"); - track.give_up(sequence..sequence + 1, crate::track::Hole::drop(crate::StreamError::NotFound)); + track.give_up( + sequence..sequence + 1, + crate::track::Hole::drop(crate::StreamError::NotFound), + ); Err(Error::Unsupported) } (None, Some((producer, next))) => Ok(Opened::Group(producer, next)), From 70c04a96f67ad6ba9b65b20c7dd9af4fb9d4934e Mon Sep 17 00:00:00 2001 From: "pullfrog[bot]" <226033991+pullfrog[bot]@users.noreply.github.com> Date: Fri, 6 Nov 2026 11:31:58 +0000 Subject: [PATCH 53/68] test(net): quarantine datagram-tracks' end-before-close case Every draft's case assumes the mock delivers the track's end in microtasks within one host task. The end can still be the publisher's host task when the 1000-microtask chain starts, which starves it, so the following `client.abort()` reaches the subscriber with `entry.end` unset and correctly cuts the track off. Reproduced on main at 6ca2ed5e3 (4 of 8 loaded whole-file runs) and on this branch (6 of 10), so it is not this PR's. The case is quarantined with its assertions intact, and quest/m0/test-flakes-2/datagram-end-close.md carries the evidence and the fix direction. --- js/net/src/lite/datagram-tracks.test.ts | 6 ++- quest/m0/test-flakes-2/README.md | 6 +++ quest/m0/test-flakes-2/datagram-end-close.md | 54 ++++++++++++++++++++ 3 files changed, 65 insertions(+), 1 deletion(-) create mode 100644 quest/m0/test-flakes-2/datagram-end-close.md diff --git a/js/net/src/lite/datagram-tracks.test.ts b/js/net/src/lite/datagram-tracks.test.ts index a79af11067f..00fa45a662c 100644 --- a/js/net/src/lite/datagram-tracks.test.ts +++ b/js/net/src/lite/datagram-tracks.test.ts @@ -450,8 +450,12 @@ for (const alpn of [ALPN_05, ALPN_06, ALPN_07_WIP]) { // A track's end waits a host task for the datagrams received with it. A session close in // between finishes the track rather than cutting it off: the publisher already ended it. +// Quarantined pending quest/m0/test-flakes-2/datagram-end-close.md: the end can still be the +// publisher's host task when the microtask chain starts, which starves that task, so the close +// lands first and legitimately cuts the track off. Lite-06 hits it most often, but lite-05 has +// too, so every draft waits on the fix. for (const alpn of [ALPN_05, ALPN_06, ALPN_07_WIP]) { - test(`${alpn}: a session close right after a datagram track's end keeps the end`, async () => { + test.skip(`${alpn}: a session close right after a datagram track's end keeps the end`, async () => { const source = new Origin(); const publication = source.route(path, { hidden: true }); const datagrams = publication.broadcast.createDatagrams("audio"); diff --git a/quest/m0/test-flakes-2/README.md b/quest/m0/test-flakes-2/README.md index 3359ece99e1..b2ed7f331af 100644 --- a/quest/m0/test-flakes-2/README.md +++ b/quest/m0/test-flakes-2/README.md @@ -95,6 +95,11 @@ test, which failed two merge-queue batches and two loaded local runs. Added 2026-11-05 (maintainer, `/quest-audit`): Interop's P2P data-channel timeout, one child. +Added 2026-11-06 (fix run on +[moq-dev/moq#7670](https://github.com/moq-dev/moq/pull/7670), which +quarantines it): `js/net`'s datagram end-before-close case, one child. It +reproduces on `main`, so it is not that PR's. + Public API: none. Wire: none. ## Required @@ -106,3 +111,4 @@ Public API: none. Wire: none. - [Diagnose and re-enable the FFI datagram connection test](/quest/m0/test-flakes-2/ffi-datagram-roundtrip.md) - diagnose the default-connection timeout and re-enable the quarantined test - [Test PulseAudio servers stay off the desktop's D-Bus](/quest/m0/test-flakes-2/pulse-dbus.md) - concurrent media runs on one desktop never collide on `org.PulseAudio1` - [The P2P data-channel test transfers its frame](/quest/m0/test-flakes-2/p2p-data-channel.md) - Interop's two-peer data-channel test stops stalling after a failed send +- [datagram-tracks' end-before-close case passes under load](/quest/m0/test-flakes-2/datagram-end-close.md) - every draft's end survives the session close that follows it in loaded multi-file runs, and the quarantine comes off diff --git a/quest/m0/test-flakes-2/datagram-end-close.md b/quest/m0/test-flakes-2/datagram-end-close.md new file mode 100644 index 00000000000..c5999b33a73 --- /dev/null +++ b/quest/m0/test-flakes-2/datagram-end-close.md @@ -0,0 +1,54 @@ +# [S] datagram-tracks' end-before-close case passes under load + +## Goal + +`a session close right after a datagram track's end keeps the end` in +`js/net/src/lite/datagram-tracks.test.ts` passes under load for every draft in +multi-file runs, fixed at its cause under the +[epic](/quest/m0/test-flakes-2/README.md)'s rules. Done when the quarantine +from [moq-dev/moq#7670](https://github.com/moq-dev/moq/pull/7670) is removed +with its assertions unchanged. + +## Plan + +Reproduced 2026-11-06 with the whole file beside six busy loops on a four-core +runner. It failed 6 of 10 runs on +[moq-dev/moq#7670](https://github.com/moq-dev/moq/pull/7670) at `13a544e41` +and 4 of 8 runs on `main` at `6ca2ed5e3`, always the lite-06 case, with +`subscribe error: id=1 broadcast=live track=audio error=remote error: 0`. The +file passes alone. So the draft is the trigger, not the PR, and neither the +test nor the code it covers changed between them: the PR touches no line in +`subscriber.ts` that mentions `entry.end` or `datagrams`. + +Lite-06 hits it far more often, but it is not the only draft: after +quarantining lite-06 alone, lite-05 failed 1 of 8 further loaded runs the same +way. So the whole case is quarantined, and the fix must cover all three. + +The test ends the publisher's track, then burns 1000 microtasks so the end +lands "all within this host task", then closes the session and expects +`recv()` to report the end rather than the close. That holds only when the +end's whole path runs in microtasks, and the failure says it did not: the +close reached the subscriber while `entry.end` was still unset, so it aborted +the datagram track. That is the correct answer for a close that precedes the +end, so the assertion is fine and the assumption about when the end has landed +is the suspect. A host task (`yieldNow`, which is `clock.task`) still +outstanding on the publisher's serve loop would be starved by the microtask +chain, and lite-06 fits it: its SUBSCRIBE carries no range list, so fewer +writes precede the publisher's read of the end than on lite-05 or lite-07. +Confirm which host task is outstanding before choosing the fix; a later host +task landing in the burst, rather than a leftover one, would need a different +answer. + +Fix it by waiting on an event that proves the subscriber read the end before +closing the session, rather than a microtask count: for example capture the +publisher's `SUBSCRIBE_FINISH` write through the mock as `holdNextAnswer` +does, or await a hook the subscriber calls on the response. Keep the close +between that evidence and `recv()`, so the case still covers the close that +lands after the end. Raising the microtask count, adding a wall-clock wait, or +retrying the test are not fixes. + +## Waiting + +- moq-dev/moq: reproduce the case under load with the fix in place, then + remove the `test.skip` quarantine and show the repeated multi-file run + passing. From 72bab5800f57eadbadebadaa64f0e4cd4bddd7af Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 6 Nov 2026 08:16:00 -0800 Subject: [PATCH 54/68] docs: append_group's implicit skip starts past the previous append; name the datagram flake's overlapping PRs Co-Authored-By: Claude Opus 5.5 --- js/net/src/track.ts | 9 +++++---- quest/m0/test-flakes-2/datagram-end-close.md | 18 +++++++++++++----- quest/m1/drop-set.md | 7 ++++--- rs/moq-net/src/model/track.rs | 8 ++++---- 4 files changed, 26 insertions(+), 16 deletions(-) diff --git a/js/net/src/track.ts b/js/net/src/track.ts index bef61190cad..ed6c22c3d07 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -1800,9 +1800,9 @@ export class Producer { * * Appending says the producer will not go back, so it skips every sequence below the new * group that it never produced, from just past its previous append (before the first, from - * the lowest group it created), as {@link skipGroups} would, replacing any drop there. Those sequences are then final: creating one - * throws {@link Skipped}. Appending alone never leaves such a gap; one left by out-of-order - * {@link createGroup} calls is what it names. + * the lowest group it created), as {@link skipGroups} would, replacing any drop there. Those + * sequences are then final: creating one throws {@link Skipped}. Appending alone never leaves + * such a gap; one left by out-of-order {@link createGroup} calls is what it names. * * The broadcast remembers nothing of an earlier producer of the name, so a replacement * starts at 0. Since a name always means the same content, a replacement continues past @@ -1874,7 +1874,8 @@ export class Producer { // Settle an append at `sequence`: skip every sequence below it this track's own publisher // never produced, from just past its previous append (before the first, from the lowest it - // produced), since appending says it will not go back. The skip replaces a drop given there, as `skipGroups` does. + // produced), since appending says it will not go back. The skip replaces a drop given there, + // as `skipGroups` does. #settleAppend(sequence: bigint): void { const from = this.#appended ?? this.#produced[0]?.start; if (from !== undefined) { diff --git a/quest/m0/test-flakes-2/datagram-end-close.md b/quest/m0/test-flakes-2/datagram-end-close.md index c5999b33a73..e84202e7c7f 100644 --- a/quest/m0/test-flakes-2/datagram-end-close.md +++ b/quest/m0/test-flakes-2/datagram-end-close.md @@ -47,8 +47,16 @@ between that evidence and `recv()`, so the case still covers the close that lands after the end. Raising the microtask count, adding a wall-clock wait, or retrying the test are not fixes. -## Waiting - -- moq-dev/moq: reproduce the case under load with the fix in place, then - remove the `test.skip` quarantine and show the repeated multi-file run - passing. +## Related + +- [moq-dev/moq#8000](https://github.com/moq-dev/moq/pull/8000), open, edits + this case where it still runs on `main`: it stubs `Budget.prototype.poll` so the + publisher's write budget never yields a host task, and raises the + microtask count to 10000. Stubbing the budget removes the suspected + outstanding `yieldNow`, so check whether it is the cause before choosing + this fix; the raised count alone is not one. Whichever lands second merges + the other's hunk. +- [moq-dev/moq#7981](https://github.com/moq-dev/moq/pull/7981), a draft, + quarantines only the lite-06 case and adds its own quest at this same + path. Whichever lands second keeps one quest and this whole-case + quarantine, since lite-05 failed too. diff --git a/quest/m1/drop-set.md b/quest/m1/drop-set.md index db190c099c7..e54f816ee43 100644 --- a/quest/m1/drop-set.md +++ b/quest/m1/drop-set.md @@ -147,9 +147,10 @@ decision above and the PR's `## Waiting` item on out-of-order creation: replacement; merging with the stale report left in place. - ✅ **`append_group` is an implicit skip** (refinement, same session): it means "the next group", so the producer declares it won't go back, and it - skips every sequence below it, down to the lowest group created since the - last append, that it never produced, as if `skip_groups` were called on - them: a drop there becomes a skip, and they are then final. Today + skips every sequence below it that it never produced, from just past the + previous append (before the first append, from the lowest group created), + as if `skip_groups` were called on them: a drop there becomes a skip, and + they are then final. Today `append_group` leaves no gap of its own, so this only fires on holes earlier `create_group` calls left below the head; under time addressing it names the span between consecutive starts. `create_group` stays non-committal, diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index f594ffa14b2..5bdf779762c 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -1207,8 +1207,8 @@ impl TrackState { /// Settle an append at `sequence`: skip every sequence below it that this track's own /// publisher never produced, from just past its previous append (before the first, from - /// the lowest it produced), since appending says it will not go back. The skip replaces a drop given there, as - /// [`Self::skip_groups`] does. + /// the lowest it produced), since appending says it will not go back. The skip replaces a + /// drop given there, as [`Self::skip_groups`] does. fn appended(&mut self, sequence: u64) { if let Some(from) = self.appended.or(self.produced.iter().next().map(|run| run.start)) && from < sequence @@ -3647,8 +3647,8 @@ impl Producer { /// Appending says the producer will not go back, so it skips every sequence below the new /// group that it never produced, from just past its previous append (before the first, /// from the lowest group it created), as [`Self::skip_groups`] would, replacing any drop - /// there. Those sequences are then final: creating one is refused. Appending alone never leaves such a gap; one left by out-of-order - /// [`Self::create_group`] calls is what it names. + /// there. Those sequences are then final: creating one is refused. Appending alone never + /// leaves such a gap; one left by out-of-order [`Self::create_group`] calls is what it names. /// /// The broadcast remembers nothing of an earlier producer of the name, so a replacement /// starts at 0. Since a name always means the same content, a replacement continues past From 236ce6d29ad2a03b5db57a6233477da5f8ab781e Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 6 Nov 2026 08:30:23 -0800 Subject: [PATCH 55/68] feat(ffi,c): expose skip_groups through the bindings Rust and JS publishers can skip a gap with skip_groups, but every SDK built on moq-ffi, and the C API, could only create groups out of order, leaving readers no way to learn a known gap is final. Co-Authored-By: Claude Opus 5.5 --- dart/moq_ffi/lib/src/moq.dart | 28 ++++++++++++++++++++++++++++ go/wrapper/publish.go | 6 ++++++ py/moq-rs/README.md | 1 + py/moq-rs/moq/publish.py | 7 +++++++ rs/moq-c/src/api.rs | 14 ++++++++++++++ rs/moq-c/src/publish.rs | 7 +++++++ rs/moq-c/src/test.rs | 17 +++++++++++++++++ rs/moq-ffi/src/producer.rs | 13 +++++++++++++ rs/moq-ffi/src/test.rs | 14 ++++++++++++++ swift/Sources/Moq/Track.swift | 6 ++++++ 10 files changed, 113 insertions(+) diff --git a/dart/moq_ffi/lib/src/moq.dart b/dart/moq_ffi/lib/src/moq.dart index e157887146f..d4fbed44f7e 100644 --- a/dart/moq_ffi/lib/src/moq.dart +++ b/dart/moq_ffi/lib/src/moq.dart @@ -14461,6 +14461,7 @@ abstract class MoqTrackProducerInterface { void finishAt({required int finalSequence}); MoqPosition? largest(); MoqTrackRequests requests(); + void skipGroups({required int start, required int end}); void writeFrame({required MoqFrame frame}); } @@ -14598,6 +14599,17 @@ class MoqTrackProducer implements MoqTrackProducerInterface { ); } + void skipGroups({required int start, required int end}) { + return rustCall((status) { + uniffi_moq_ffi_fn_method_moqtrackproducer_skip_groups( + uniffiClonePointer(), + FfiConverterUInt64.lower(start), + FfiConverterUInt64.lower(end), + status, + ); + }, moqExceptionErrorHandler); + } + void writeFrame({required MoqFrame frame}) { return rustCall((status) { uniffi_moq_ffi_fn_method_moqtrackproducer_write_frame( @@ -22067,6 +22079,16 @@ external Pointer uniffi_moq_ffi_fn_method_moqtrackproducer_requests( Pointer uniffiStatus, ); +@Native, Uint64, Uint64, Pointer)>( + assetId: _uniffiAssetId, +) +external void uniffi_moq_ffi_fn_method_moqtrackproducer_skip_groups( + Pointer ptr, + int start, + int end, + Pointer uniffiStatus, +); + @Native, RustBuffer, Pointer)>( assetId: _uniffiAssetId, ) @@ -23875,6 +23897,9 @@ external int uniffi_moq_ffi_checksum_method_moqtrackproducer_largest(); @Native(assetId: _uniffiAssetId) external int uniffi_moq_ffi_checksum_method_moqtrackproducer_requests(); +@Native(assetId: _uniffiAssetId) +external int uniffi_moq_ffi_checksum_method_moqtrackproducer_skip_groups(); + @Native(assetId: _uniffiAssetId) external int uniffi_moq_ffi_checksum_method_moqtrackproducer_write_frame(); @@ -24762,6 +24787,9 @@ void _checkApiChecksums() { if (uniffi_moq_ffi_checksum_method_moqtrackproducer_requests() != 60252) { throw UniffiInternalError.panicked("UniFFI API checksum mismatch"); } + if (uniffi_moq_ffi_checksum_method_moqtrackproducer_skip_groups() != 25592) { + throw UniffiInternalError.panicked("UniFFI API checksum mismatch"); + } if (uniffi_moq_ffi_checksum_method_moqtrackproducer_write_frame() != 46891) { throw UniffiInternalError.panicked("UniFFI API checksum mismatch"); } diff --git a/go/wrapper/publish.go b/go/wrapper/publish.go index 013ac28ad76..e97f618047f 100644 --- a/go/wrapper/publish.go +++ b/go/wrapper/publish.go @@ -205,6 +205,12 @@ func (t *TrackProducer) CreateGroup(sequence uint64) (*GroupProducer, error) { return &GroupProducer{inner: inner}, nil } +// SkipGroups declares that groups start up to the exclusive end will never be +// produced, so readers move past them. Creating a group there then fails. +func (t *TrackProducer) SkipGroups(start, end uint64) error { + return t.inner.SkipGroups(start, end) +} + // WriteFrame writes frame as a single-frame group. func (t *TrackProducer) WriteFrame(frame Frame) error { return t.inner.WriteFrame(frame) diff --git a/py/moq-rs/README.md b/py/moq-rs/README.md index e3bd133b9d1..d111b16b948 100644 --- a/py/moq-rs/README.md +++ b/py/moq-rs/README.md @@ -180,6 +180,7 @@ client = moq.Client( - `.write_frame(payload, timestamp=None)` writes a payload with optional capture time. - `await .ready(size)` (`GroupProducer`) waits until a frame of `size` bytes fits a window ahead of the fastest reader. Await it before each write for backpressure. A writer that doesn't wait sheds the front of the group instead. - `.create_group(sequence)` creates a sparse or replayed group at an explicit sequence. + - `.skip_groups(start, end)` (`TrackProducer`) declares groups `start` up to the exclusive `end` will never be produced, so readers move past them; creating one there then fails. - `.finish()` ends at the live edge; the handle remains so `.abort(error_code)` can still run. - `.finish_at(final_sequence)` declares the first group that will never be produced while leaving lower groups writable. - `.abort(error_code)` terminates the track or group with an application error. diff --git a/py/moq-rs/moq/publish.py b/py/moq-rs/moq/publish.py index ef3530a1799..96bdcccc8a8 100644 --- a/py/moq-rs/moq/publish.py +++ b/py/moq-rs/moq/publish.py @@ -136,6 +136,13 @@ def create_group(self, sequence: int) -> GroupProducer: """Create a group with an explicit sequence number.""" return GroupProducer(self._inner.create_group(sequence)) + def skip_groups(self, start: int, end: int) -> None: + """Declare that groups ``start`` up to the exclusive ``end`` will never be produced. + + Readers move past them, and creating a group there then fails. + """ + self._inner.skip_groups(start, end) + def write_frame(self, payload: bytes, timestamp: Timestamp | None = None) -> None: """Write a single-frame group with its presentation timestamp.""" self._inner.write_frame(Frame(payload, timestamp)._ffi()) diff --git a/rs/moq-c/src/api.rs b/rs/moq-c/src/api.rs index d8d438b7c50..618a7c6a43d 100644 --- a/rs/moq-c/src/api.rs +++ b/rs/moq-c/src/api.rs @@ -3221,6 +3221,20 @@ pub extern "C" fn moq_publish_track_group_at(track: u32, sequence: u64) -> i32 { }) } +/// Declare that a raw track will never produce groups `start` up to the exclusive `end`. +/// +/// Readers waiting on them move past without ending. The skip is final: creating a group +/// there with [moq_publish_track_group_at] then fails, though a group already written stays. +/// +/// Returns a zero on success, or a negative code on failure. +#[unsafe(no_mangle)] +pub extern "C" fn moq_publish_track_skip(track: u32, start: u64, end: u64) -> i32 { + ffi::enter(move || { + let track = ffi::parse_id(track)?; + State::lock().publish.track_skip(track, start, end) + }) +} + /// Write a single-frame group to a raw track with a timestamp. /// /// Convenience for the common one-frame-per-group pattern. Equivalent to diff --git a/rs/moq-c/src/publish.rs b/rs/moq-c/src/publish.rs index b399493045b..b63e228cdcb 100644 --- a/rs/moq-c/src/publish.rs +++ b/rs/moq-c/src/publish.rs @@ -1126,6 +1126,13 @@ impl Publish { self.groups.insert(group) } + /// Declare that a raw track will never produce groups `start..end`. + pub fn track_skip(&mut self, track: Id, start: u64, end: u64) -> Result<(), Error> { + let raw = self.tracks.get(track).ok_or(Error::TrackNotFound)?; + raw.inner.skip_groups(start..end); + Ok(()) + } + /// Write a single-frame group to a raw track, `None` when untimed. pub fn track_frame( &mut self, diff --git a/rs/moq-c/src/test.rs b/rs/moq-c/src/test.rs index 97f72e72e86..36d8b88c976 100644 --- a/rs/moq-c/src/test.rs +++ b/rs/moq-c/src/test.rs @@ -2394,6 +2394,23 @@ fn raw_track_sparse_groups_and_known_end() { assert_eq!(moq_publish_close(broadcast), 0); } +#[test] +fn raw_track_skips_are_final() { + let origin = id(moq_origin_create()); + let broadcast = publish_broadcast(origin, b"raw-track-skips-are-final"); + let name = b"skipped"; + let track = + id(unsafe { moq_publish_track(broadcast, name.as_ptr() as *const c_char, name.len(), std::ptr::null()) }); + + assert_eq!(moq_publish_track_skip(track, 1, 3), 0); + assert!(moq_publish_track_group_at(track, 2) < 0); + let group = id(moq_publish_track_group_at(track, 3)); + assert_eq!(moq_publish_group_finish(group), 0); + assert_eq!(moq_publish_track_finish(track), 0); + assert!(moq_publish_track_skip(track, 4, 5) < 0); + assert_eq!(moq_publish_close(broadcast), 0); +} + #[test] fn raw_track_and_group_abort_consume_their_handles() { let origin = id(moq_origin_create()); diff --git a/rs/moq-ffi/src/producer.rs b/rs/moq-ffi/src/producer.rs index a6bc644d0d5..18e4167b190 100644 --- a/rs/moq-ffi/src/producer.rs +++ b/rs/moq-ffi/src/producer.rs @@ -800,6 +800,19 @@ impl MoqTrackProducer { Ok(Arc::new(MoqGroupProducer::from_inner(group))) } + /// Declare that groups `start` up to the exclusive `end` will never be produced. + /// + /// Readers waiting on them move past without ending, and a relay forwards the skip. A + /// skip is final: creating a group there is then refused. A group already written keeps + /// what it is. [`append_group`](Self::append_group) skips the gap below it on its own. + pub fn skip_groups(&self, start: u64, end: u64) -> Result<(), MoqError> { + let _guard = crate::ffi::enter(); + let guard = self.inner.lock().unwrap(); + let raw = guard.as_ref().ok_or(MoqError::Closed)?; + raw.skip_groups(start..end); + Ok(()) + } + /// Write `frame` as a single-frame group. /// /// A timestamp at another scale than the track's is converted to it, rounding down. The frame must diff --git a/rs/moq-ffi/src/test.rs b/rs/moq-ffi/src/test.rs index f4f1fde0c14..42e5cdd9216 100644 --- a/rs/moq-ffi/src/test.rs +++ b/rs/moq-ffi/src/test.rs @@ -2225,6 +2225,20 @@ fn raw_track_supports_sparse_groups_and_known_end() { track.finish().unwrap(); } +#[test] +fn raw_track_skips_are_final() { + let broadcast = MoqBroadcastProducer::new().unwrap(); + let track = broadcast.publish_track("skipped".into(), None).unwrap(); + + track.create_group(0).unwrap().finish().unwrap(); + track.skip_groups(0, 3).unwrap(); + // The written group keeps what it is; the rest of the range is final. + assert!(track.create_group(2).is_err()); + track.create_group(3).unwrap().finish().unwrap(); + track.abort(1).unwrap(); + assert!(matches!(track.skip_groups(4, 5), Err(MoqError::Closed))); +} + #[tokio::test] async fn abort_after_finish_keeps_the_track_handle() { let broadcast = MoqBroadcastProducer::new().unwrap(); diff --git a/swift/Sources/Moq/Track.swift b/swift/Sources/Moq/Track.swift index d2d58cda5c9..8ed81f912ef 100644 --- a/swift/Sources/Moq/Track.swift +++ b/swift/Sources/Moq/Track.swift @@ -163,6 +163,12 @@ public final class TrackProducer: Sendable { GroupProducer(try ffi.createGroup(sequence: sequence)) } + /// Declare that groups `start` up to the exclusive `end` will never be produced. + /// Readers move past them, and creating a group there then throws. + public func skipGroups(start: UInt64, end: UInt64) throws { + try ffi.skipGroups(start: start, end: end) + } + /// Write a single-frame group with a presentation timestamp, or untimed when nil. public func writeFrame(_ payload: Data, timestamp: Timestamp? = nil) throws { try ffi.writeFrame(frame: Frame(payload: payload, timestamp: timestamp)) From fe6c1e123178b38b8a714186da9369e905482ce8 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 6 Nov 2026 08:45:32 -0800 Subject: [PATCH 56/68] fix(net): an unaccounted gap that joins an older one waits a fresh grace Unaccounting the top of a run enlarged the gap below the next run without restarting its age, so the reopened sequences folded on the older gap's schedule, possibly at once. Restart every gap the reopened range reaches. Co-Authored-By: Claude Opus 5.5 --- js/net/src/ledger.test.ts | 13 +++++++++++++ js/net/src/ledger.ts | 5 +++-- rs/moq-net/src/ledger.rs | 22 +++++++++++++++++++++- 3 files changed, 37 insertions(+), 3 deletions(-) diff --git a/js/net/src/ledger.test.ts b/js/net/src/ledger.test.ts index 27db4ab6e6c..fa14f9ffba0 100644 --- a/js/net/src/ledger.test.ts +++ b/js/net/src/ledger.test.ts @@ -125,3 +125,16 @@ test("unaccount reopens a gap for a fresh grace", () => { ledger.expire(at(2000), grace); expect(ledger.covers(0n, 10n)).toBe(true); }); + +test("unaccount restarts the older gap it joins", () => { + const ledger = new Ledger(); + const grace = Milli(1000); + ledger.account(0n, 2n, at(0)); + ledger.account(4n, 6n, at(0)); + ledger.unaccount(1n, 2n, at(900)); + // Its grace runs from the reversal, not from when the older gap opened. + ledger.expire(at(1000), grace); + expect(ledger.covers(1n, 2n)).toBe(false); + ledger.expire(at(1900), grace); + expect(ledger.covers(0n, 6n)).toBe(true); +}); diff --git a/js/net/src/ledger.ts b/js/net/src/ledger.ts index fda760e859a..ddf7617ecc9 100644 --- a/js/net/src/ledger.ts +++ b/js/net/src/ledger.ts @@ -157,7 +157,7 @@ export class Ledger { /** * Take `[start, end)` back out of what is accounted for, owed again from `at`: a verdict that * accounted for it was reversed. The gap it leaves opens at `at`, so it is waited for a fresh - * grace. Mirrors the Rust `Ledger::unaccount`. + * grace, even where it joins an older gap. Mirrors the Rust `Ledger::unaccount`. */ unaccount(start: bigint, end: bigint, at: Instant): void { if (start >= end) return; @@ -167,6 +167,7 @@ export class Ledger { if (run.start >= end) break; if (run.end > start) overlapping.push(run); } + if (overlapping.length === 0) return; for (const run of overlapping) { this.#remove(run); if (this.#last === run) this.#last = undefined; @@ -183,7 +184,7 @@ export class Ledger { } const last = this.#runs.maxKey(); this.#last = last === undefined ? undefined : this.#runs.get(last); - this.#nextExpiry(); + this.demand(start, end, at); } /** Restart the age of every gap reaching into `[start, end)`, which the demand newly asks for. */ diff --git a/rs/moq-net/src/ledger.rs b/rs/moq-net/src/ledger.rs index 50736298d7e..80caa9ffa1f 100644 --- a/rs/moq-net/src/ledger.rs +++ b/rs/moq-net/src/ledger.rs @@ -157,12 +157,15 @@ impl Ledger { /// Take `groups` back out of what is accounted for, owed again from `now`: a verdict that /// accounted for them was reversed. The gap they leave opens now, so it is waited for a - /// fresh grace. + /// fresh grace, even where it joins an older gap. pub fn unaccount(&mut self, groups: Range, now: Instant) { if groups.is_empty() { return; } let overlapping: Vec = self.accounted(groups.clone()).map(|run| run.start).collect(); + if overlapping.is_empty() { + return; + } for clipped in overlapping { let start = self .runs @@ -194,6 +197,7 @@ impl Ledger { self.changes.insert(groups.end..run.end); } } + self.demand(groups, now); } /// Account a live arrival, refreshing a cached incarnation even if its sequence was @@ -511,6 +515,22 @@ mod tests { assert!(ledger.covers(0..10)); } + /// A reopened gap that joins an older one is waited for a fresh grace as a whole, rather + /// than folding with the older gap's age. + #[test] + fn unaccount_restarts_the_older_gap_it_joins() { + let start = Instant::now(); + let mut ledger = Ledger::new(GRACE); + arrive(&mut ledger, 0..2, start); + arrive(&mut ledger, 4..6, start); + let reversed = start + GRACE * 9 / 10; + ledger.unaccount(1..2, reversed); + ledger.expire(start + GRACE); + assert!(!ledger.covers(1..2), "its grace runs from the reversal"); + ledger.expire(reversed + GRACE); + assert!(ledger.covers(0..6)); + } + #[test] fn demand_can_move_an_older_gap_after_a_newer_one() { let start = Instant::now(); From 915ed98b4e69d964afaeca834696a9095d5e6079 Mon Sep 17 00:00:00 2001 From: "pullfrog[bot]" <226033991+pullfrog[bot]@users.noreply.github.com> Date: Fri, 6 Nov 2026 17:38:09 +0000 Subject: [PATCH 57/68] test(net): keep the resume copy helper local, as Rust does The shared `copy()` helper marked every route copy as received, so a publisher's copy claimed a live feed it does not have: its own subscribers' demand made the feed start at 0, and `owedRuns` reported the gap below the resume point as still landing, so `a backfill waits on a fill the finished route still owes` no longer asked the route's fill handler. Rust keeps `copy()` local and has a separate received helper, so do the same here. Tests that need a received copy opt in with `hooks.receiveTrack`. --- js/net/src/resume.test.ts | 2 -- 1 file changed, 2 deletions(-) diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts index 723e7da6139..df4ac07bfc4 100644 --- a/js/net/src/resume.test.ts +++ b/js/net/src/resume.test.ts @@ -52,8 +52,6 @@ function serve(origin: Producer, source: BroadcastProducer, cost: bigint) { */ function copy(source: BroadcastProducer) { const producer = source.createTrack("seq", { maxAge: null }); - // A route's copy is received, so its groups arrive in any order. - hooks.receiveTrack(producer); return { producer, write(sequence: bigint, frames = 1, close = true) { From a8bf6e7603f6bf5f9332f2f258d1126f9bc3c96b Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 6 Nov 2026 09:45:43 -0800 Subject: [PATCH 58/68] test(net): main's finished-route backfill test uses authored copies This branch's route copies are received, so a live feed owes the groups below the first arrival for the reader's grace; main's new test expects the fill an authored copy asks for at once. Co-Authored-By: Claude Opus 5.5 --- js/net/src/resume.test.ts | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts index 723e7da6139..7c19b26feaa 100644 --- a/js/net/src/resume.test.ts +++ b/js/net/src/resume.test.ts @@ -51,9 +51,15 @@ function serve(origin: Producer, source: BroadcastProducer, cost: bigint) { * Resumed readers backfill superseded groups, so it sets no publisher limit. */ function copy(source: BroadcastProducer) { - const producer = source.createTrack("seq", { maxAge: null }); + const copied = authoredCopy(source); // A route's copy is received, so its groups arrive in any order. - hooks.receiveTrack(producer); + hooks.receiveTrack(copied.producer); + return copied; +} + +/** A {@link copy} its source writes itself, so no live feed owes the groups it lacks. */ +function authoredCopy(source: BroadcastProducer) { + const producer = source.createTrack("seq", { maxAge: null }); return { producer, write(sequence: bigint, frames = 1, close = true) { @@ -277,7 +283,9 @@ test("a backfill waits on a fill the finished route still owes", async () => { const front = new Front(); const sources = [new BroadcastProducer(), new BroadcastProducer()]; const handles = sources.map((source) => source.consume()); - const [ta, tb] = sources.map(copy); + // A received copy would owe the groups below its first arrival to its live feed for the + // reader's grace, an hour here, before a fill may serve them. + const [ta, tb] = sources.map(authoredCopy); const requests = tb.producer.requests(); const fills: track.Fill[] = []; void (async () => { From 29d2227107971644c0671aa723de537e7bd48ded Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 6 Nov 2026 09:49:52 -0800 Subject: [PATCH 59/68] fix(net): ask a fill for an unsettled stale run above the newest group A reader that a received stale run does not settle treats it as unknown and fills it, but fills were capped at the newest group received, so a stale run with nothing arrived above it was never asked for and a bounded reader waited forever. The run is now asked for wherever it lies: JS asks the unsettled stale runs past the received edge, and Rust extends a probe's span to them and counts a stale run as never fed by the live feed, as JS's landing already did. Co-Authored-By: Claude Opus 5.5 --- js/net/src/fill.test.ts | 30 ++++++++++++++++++++++ js/net/src/track.ts | 14 ++++++++--- rs/moq-net/src/model/track.rs | 33 ++++++++++++++++++++++++ rs/moq-net/src/model/track/miss.rs | 40 +++++++++++++++++++++++------- 4 files changed, 104 insertions(+), 13 deletions(-) diff --git a/js/net/src/fill.test.ts b/js/net/src/fill.test.ts index 2e752c3e323..217b785e78a 100644 --- a/js/net/src/fill.test.ts +++ b/js/net/src/fill.test.ts @@ -774,6 +774,36 @@ test("fill: a stale run that does not settle the reader is still asked for", asy broadcast.close(); }); +// A stale verdict this reader cannot trust is asked for even when no group has arrived to +// bound the misses, so the reader still settles. Mirrors Rust. +test("fill: an unsettled stale run is asked for with nothing received", async () => { + const broadcast = new BroadcastProducer(); + const track = broadcast.createTrack("video", { maxAge: null }); + hooks.receiveTrack(track); + const requests = track.requests(); + // The upstream judged ranges ending at 6 under 500ms, and gave up both of them as stale. + hooks.supersedeStale(track, 6n); + hooks.dropTrackGroups(track, 0n, 2n, staleHole(500)); + hooks.dropTrackGroups(track, 4n, 6n, staleHole(500)); + // The same ranges: only the last one is contiguous to the judged end, so the first is unknown. + const reader = track.subscribe({ + ranges: [ + { start: { absolute: 0n }, end: { absolute: 2n } }, + { start: { absolute: 4n }, end: { absolute: 6n } }, + ], + delay: { max: Milli(500) }, + }); + const done = drain(reader).then(() => "done"); + const fill = await Promise.race([requests.requestedFill(), yieldNow().then(() => undefined)]); + expect(fill && [fill.start, fill.end]).toEqual([0n, 2n]); + fill?.close(); + expect(await Promise.race([done, yieldNow().then(() => "pending")])).toBe("done"); + + reader.close(); + requests.close(); + broadcast.close(); +}); + test("fill: a cache-miss fetch of a dynamic track serves the one track the broadcast already has", async () => { const broadcast = new BroadcastProducer(); // Pulled first, so the broadcast serves tracks on demand. diff --git a/js/net/src/track.ts b/js/net/src/track.ts index 11cca3c8aa9..55eb0edc733 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -2919,10 +2919,16 @@ export class Subscriber { const budget = drift.budget; const coverage = this.#coverage(); const drops = this.#state.drops.peek(); - for (const run of this.#wants( - floor === undefined ? start : max(start, floor), - min(end, max(this.#state.received.peek(), (fillEdge(producer) ?? -1n) + 1n)), - )) { + const low = floor === undefined ? start : max(start, floor); + const edge = min(end, max(this.#state.received.peek(), (fillEdge(producer) ?? -1n) + 1n)); + const spans: Run[] = [{ start: low, end: edge }]; + // The upstream accounted for a stale run, so the live feed brings none of it unless an + // update supersedes it: asked even above the newest group received. + for (const dropped of drops) { + const span = { ...dropped, start: max(dropped.start, max(low, edge)), end: min(dropped.end, end) }; + if (span.start < span.end && !settles(span, budget, coverage, this.#state.judgedEnd)) spans.push(span); + } + for (const run of spans.flatMap((span) => this.#wants(span.start, span.end))) { const gone = [...accounted]; for (const dropped of drops) { if (dropped.end <= run.start || run.end <= dropped.start) continue; diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index 22b413aaf0f..b8e9e9786a7 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -22641,6 +22641,39 @@ mod test { assert_eq!((fill.start().group, fill.end()), (0, 3)); } + /// A stale verdict a reader cannot trust is asked for even when no group arrived to bound + /// the misses, so the reader still settles. Mirrors JS. + #[test] + #[cfg(feature = "async")] + fn an_unsettled_stale_run_is_asked_for_with_nothing_received() { + let producer = track_producer("test", Info::timed()); + producer.received(); + let dynamic = producer.requests(); + let budget = Duration::from_millis(500); + // The upstream judged ranges ending at 6 under 500ms, and gave up both of them as stale. + producer.supersede_stale(Some(6)); + producer.give_up(0..2, Hole::stale(budget)); + producer.give_up(4..6, Hole::stale(budget)); + // The same ranges: only the last one is contiguous to the judged end, so the first is + // unknown. + let mut subscriber = producer.subscribe( + Subscription::default() + .with_delay(Delay::default().with_max(budget)) + .with_ranges([Range::groups(0..2), Range::groups(4..6)]), + ); + let waiter = kio::Waiter::noop(); + assert!(subscriber.poll_recv_group(&waiter).is_pending()); + let Poll::Ready(Ok(fill)) = dynamic.poll_requested_fill(&waiter) else { + panic!("the unsettled stale run is asked for"); + }; + assert_eq!((fill.start().group, fill.end()), (0, 2)); + drop(fill); + assert!( + matches!(subscriber.poll_recv_group(&waiter), Poll::Ready(Ok(None))), + "nothing is owed once the fill gave up" + ); + } + /// Creating a group past the next sequence says nothing about the gap: groups may be /// created out of order, so a late one in it still reaches readers, and nothing is skipped. #[test] diff --git a/rs/moq-net/src/model/track/miss.rs b/rs/moq-net/src/model/track/miss.rs index 99dd35a605d..ccc5f0feeb6 100644 --- a/rs/moq-net/src/model/track/miss.rs +++ b/rs/moq-net/src/model/track/miss.rs @@ -608,7 +608,16 @@ impl Misses { if state.start_pending || !state.readable() && bound.is_none() { return asks; } - let Some(newest) = state.newest_produced.max(bound) else { + // A received stale run this reader cannot trust is still asked for, though nothing above + // it arrived; see `TrackState::unfed`. Only runs above the newest group need a look. + let unknown = state + .drops + .iter() + .rev() + .take_while(|(run, _)| state.newest_produced.is_none_or(|newest| run.end - 1 > newest)) + .find(|(run, hole)| !drift.settles(state, run.start, hole)) + .map(|(run, _)| run.end - 1); + let Some(newest) = state.newest_produced.max(bound).max(unknown) else { return asks; }; let at = ( @@ -1398,22 +1407,35 @@ impl TrackState { unfed.insert(hole); return (unfed, awaited); } - let mut from = hole.start; let grace = self.miss_grace(); let mut gap = |groups: ops::Range, unfed: &mut RangeSet| match self.arrivals.folds_at(groups.start, grace) { Some(at) if at <= now => unfed.insert(groups), folds => awaited.push((groups, folds)), }; - for arrived in self.arrivals.accounted(from..hole.end) { - if from < arrived.start { - gap(from..arrived.start, &mut unfed); + // The upstream accounted for a received stale run, so the feed brings none of it unless + // an update supersedes the run. + let mut fed = RangeSet::new(); + fed.insert(hole.clone()); + for (run, given) in self.drops.overlapping(&hole) { + if given.stale_budget().is_some() { + let run = run.start.max(hole.start)..run.end.min(hole.end); + fed.remove(run.clone()); + unfed.insert(run); } - from = arrived.end; - unfed.insert(arrived); } - if from < hole.end { - gap(from..hole.end, &mut unfed); + for hole in fed.iter() { + let mut from = hole.start; + for arrived in self.arrivals.accounted(hole.clone()) { + if from < arrived.start { + gap(from..arrived.start, &mut unfed); + } + from = arrived.end; + unfed.insert(arrived); + } + if from < hole.end { + gap(from..hole.end, &mut unfed); + } } (unfed, awaited) } From e958788d8b1827cd1c11560931fae38337ac5402 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 6 Nov 2026 09:58:46 -0800 Subject: [PATCH 60/68] fix(net): ask unsettled stale runs past the newest group only while a handler may fill them Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/src/model/track/miss.rs | 22 +++++++++++++++------- 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/rs/moq-net/src/model/track/miss.rs b/rs/moq-net/src/model/track/miss.rs index ccc5f0feeb6..1bf174e1b59 100644 --- a/rs/moq-net/src/model/track/miss.rs +++ b/rs/moq-net/src/model/track/miss.rs @@ -609,14 +609,22 @@ impl Misses { return asks; } // A received stale run this reader cannot trust is still asked for, though nothing above - // it arrived; see `TrackState::unfed`. Only runs above the newest group need a look. + // it arrived; see `TrackState::unfed`. Only runs above the newest group need a look, and + // only while a handler may fill them: otherwise the reader waits for new terms. let unknown = state - .drops - .iter() - .rev() - .take_while(|(run, _)| state.newest_produced.is_none_or(|newest| run.end - 1 > newest)) - .find(|(run, hole)| !drift.settles(state, run.start, hole)) - .map(|(run, _)| run.end - 1); + .fills + .read() + .has_handlers() + .then(|| { + state + .drops + .iter() + .rev() + .take_while(|(run, _)| state.newest_produced.is_none_or(|newest| run.end - 1 > newest)) + .find(|(run, hole)| !drift.settles(state, run.start, hole)) + .map(|(run, _)| run.end - 1) + }) + .flatten(); let Some(newest) = state.newest_produced.max(bound).max(unknown) else { return asks; }; From d7d84d77546f803d7d4bc4b5d5953911a753e2b7 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 6 Nov 2026 11:24:41 -0800 Subject: [PATCH 61/68] feat(net): a stale verdict accepts every range the upstream judged A relay now keeps every group its current upstream terms' ranges name, not only where they end, and a received stale run settles a reader that reads every one of those groups above the run's start, however many ranges they span. A reader with the upstream's own ranges, such as [0,2) and [4,6), is settled with nothing received and no fill asked. A reader narrower than the upstream's ranges still treats the run as unknown and asks a fill for it above the newest group received, since no update brings it back. That ask no longer waits for a fill handler: a received copy's serving session keeps one, and the gate kept a reader from joining a fill taken before the last handler closed, or reading its cached answer. Co-Authored-By: Claude Opus 5.5 --- drafts/draft-lcurley-moq-lite.md | 2 +- js/net/src/fill.test.ts | 83 +++++++-- js/net/src/internal.ts | 8 +- js/net/src/lite/subscriber.ts | 24 ++- js/net/src/track.test.ts | 14 +- js/net/src/track.ts | 84 +++++---- quest/m1/drop-set.md | 18 ++ rs/moq-net/src/lite/subscriber.rs | 36 ++-- rs/moq-net/src/model/track.rs | 283 +++++++++++++++++++---------- rs/moq-net/src/model/track/miss.rs | 24 +-- 10 files changed, 381 insertions(+), 195 deletions(-) diff --git a/drafts/draft-lcurley-moq-lite.md b/drafts/draft-lcurley-moq-lite.md index 18bd4218d06..f00ca15f760 100644 --- a/drafts/draft-lcurley-moq-lite.md +++ b/drafts/draft-lcurley-moq-lite.md @@ -1881,7 +1881,7 @@ A drop whose Error Code is OLD is a stale verdict, which holds only under the up The subscriber judges it against the `Subscriber Max Age` it sent with that update, from its own record: the drop does not carry the budget. A stale drop whose `Update` a later update has replaced is no verdict, since the update may have moved the Range or budget it was judged against: the subscriber still owes those groups, and a relay asks for such a group again rather than drop it for its own readers. The next SUBSCRIBE_UPDATE reopens every stale drop the subscriber accounted for, which it owes again until the publisher serves the group or drops it again under the new `Update`, as described in [SUBSCRIBE_UPDATE](#subscribe-update). -A relay that received a stale drop settles with it only a reader whose own budget is no wider than the one the relay sent with that update and whose Ranges cover the relay's own, which the verdict was judged against. +A relay that received a stale drop settles with it only a reader whose own budget is no wider than the one the relay sent with that update and whose Ranges include every group the relay's own Ranges name above the dropped sequence, since the verdict on a group was judged against the groups above it. A drop with any other Error Code is final, whatever its `Update`, and replaces a stale drop of the same sequences, so no update reopens them. diff --git a/js/net/src/fill.test.ts b/js/net/src/fill.test.ts index 217b785e78a..ac9e8a627f4 100644 --- a/js/net/src/fill.test.ts +++ b/js/net/src/fill.test.ts @@ -774,36 +774,85 @@ test("fill: a stale run that does not settle the reader is still asked for", asy broadcast.close(); }); -// A stale verdict this reader cannot trust is asked for even when no group has arrived to -// bound the misses, so the reader still settles. Mirrors Rust. -test("fill: an unsettled stale run is asked for with nothing received", async () => { +// The upstream judged 0..2 and 4..6 under 500ms, and gave up both of them as stale. +function judgedTwice() { const broadcast = new BroadcastProducer(); const track = broadcast.createTrack("video", { maxAge: null }); hooks.receiveTrack(track); const requests = track.requests(); - // The upstream judged ranges ending at 6 under 500ms, and gave up both of them as stale. - hooks.supersedeStale(track, 6n); + hooks.supersedeStale(track, [ + { start: 0n, end: 2n }, + { start: 4n, end: 6n }, + ]); hooks.dropTrackGroups(track, 0n, 2n, staleHole(500)); hooks.dropTrackGroups(track, 4n, 6n, staleHole(500)); - // The same ranges: only the last one is contiguous to the judged end, so the first is unknown. - const reader = track.subscribe({ - ranges: [ - { start: { absolute: 0n }, end: { absolute: 2n } }, - { start: { absolute: 4n }, end: { absolute: 6n } }, - ], - delay: { max: Milli(500) }, - }); - const done = drain(reader).then(() => "done"); + const reader = (ranges: [bigint, bigint][]) => + track.subscribe({ + ranges: ranges.map(([start, end]) => ({ start: { absolute: start }, end: { absolute: end } })), + delay: { max: Milli(500) }, + }); + return { broadcast, requests, reader }; +} + +// A stale verdict settles a reader that reads every range the upstream judged it against, however +// many there are, with nothing received and no fill asked. Mirrors Rust. +test("fill: a stale verdict settles a reader of every range it judged", async () => { + const { broadcast, requests, reader: subscribe } = judgedTwice(); + const reader = subscribe([ + [0n, 2n], + [4n, 6n], + ]); + expect(hooks.tryDrop(reader)).toMatchObject({ start: 0n, end: 2n, stale: 500 }); + expect(hooks.tryDrop(reader)).toMatchObject({ start: 4n, end: 6n, stale: 500 }); + expect(hooks.tryRecvGroup(reader, false)).toEqual({ kind: "done" }); const fill = await Promise.race([requests.requestedFill(), yieldNow().then(() => undefined)]); - expect(fill && [fill.start, fill.end]).toEqual([0n, 2n]); - fill?.close(); - expect(await Promise.race([done, yieldNow().then(() => "pending")])).toBe("done"); + expect(fill).toBeUndefined(); reader.close(); requests.close(); broadcast.close(); }); +// A reader whose ranges only partly overlap the ones a stale verdict was judged against is settled +// where it reads every judged group above the run, and asks a fill for the rest, even with nothing +// received: no update brings it, since its ranges add nothing to the upstream's terms. Mirrors Rust. +test("fill: a stale verdict settles a partly overlapping reader only where it reads above", async () => { + const { broadcast, requests, reader: subscribe } = judgedTwice(); + // Reads every judged group above each run, and more past them: settled. + const wider = subscribe([ + [1n, 2n], + [4n, 8n], + ]); + expect(hooks.tryDrop(wider)).toMatchObject({ start: 1n, end: 2n, stale: 500 }); + expect(hooks.tryDrop(wider)).toMatchObject({ start: 4n, end: 6n, stale: 500 }); + expect(hooks.tryDrop(wider)).toBeUndefined(); + expect(await Promise.race([requests.requestedFill(), yieldNow().then(() => undefined)])).toBeUndefined(); + // Stops at 4, which the upstream judged against 5: its own newest, which no verdict convicts. + // Both runs are unknown to it, and asked for. + const narrower = subscribe([ + [0n, 2n], + [4n, 5n], + ]); + expect(hooks.tryRecvGroup(narrower, false)).toEqual({ kind: "idle" }); + expect(hooks.tryDrop(narrower)).toBeUndefined(); + const asked: bigint[][] = []; + for (;;) { + const fill = await Promise.race([requests.requestedFill(), yieldNow().then(() => undefined)]); + if (!fill) break; + asked.push([fill.start, fill.end]); + } + asked.sort((a, b) => Number(a[0] - b[0])); + expect(asked).toEqual([ + [0n, 2n], + [4n, 5n], + ]); + + wider.close(); + narrower.close(); + requests.close(); + broadcast.close(); +}); + test("fill: a cache-miss fetch of a dynamic track serves the one track the broadcast already has", async () => { const broadcast = new BroadcastProducer(); // Pulled first, so the broadcast serves tracks on demand. diff --git a/js/net/src/internal.ts b/js/net/src/internal.ts index fc6e9c85910..5e63f1ee2e3 100644 --- a/js/net/src/internal.ts +++ b/js/net/src/internal.ts @@ -549,11 +549,11 @@ export const hooks: { skipTrackGroups: (producer: Producer, start: bigint, end: bigint) => Run[]; /** * Forget every run a received copy holds as stale: the terms its upstream judged them under - * were replaced, so they are unknown again. The new terms' ranges end at `judgedEnd`, - * undefined past the live edge or unknown. Returns the runs forgotten: a group that arrived - * since its stale drop replaced it, so it is not among them. Mirrors the Rust `supersede_stale`. + * were replaced, so they are unknown again. `judged` is every group the new terms' ranges name, + * undefined when unknown. Returns the runs forgotten: a group that arrived since its stale drop + * replaced it, so it is not among them. Mirrors the Rust `supersede_stale`. */ - supersedeStale: (producer: Producer, judgedEnd: bigint | undefined) => Run[]; + supersedeStale: (producer: Producer, judged: readonly Run[] | undefined) => Run[]; /** * Mark a track as fed by a wire subscription, which owes missing groups. `starts` says how * that wire's start bounds the fills its readers ask below it. Assigned by `track.ts`. diff --git a/js/net/src/lite/subscriber.ts b/js/net/src/lite/subscriber.ts index c51e3f16597..5c710d95fff 100644 --- a/js/net/src/lite/subscriber.ts +++ b/js/net/src/lite/subscriber.ts @@ -29,9 +29,11 @@ import { hiddenBelow, hooks, insertDrop, + insertRun, maxDelayOf, minDelayOf, removeRun, + type Run, scopeCaptures, scopeHead, scopeOverlaps, @@ -253,16 +255,18 @@ export class Minted { } /** - * Where a stale drop judged under `ranges` stops looking: the exclusive end past every group they - * name, undefined when one stays open or ends at an anchor only the publisher resolves. + * Every group `ranges` name, which a stale drop under them was judged against. A bound only the + * publisher resolves is taken at its widest: a start from group 0, an end past the live edge + * (`UNBOUNDED`). Mirrors the Rust `judged`. */ -function judgedEnd(ranges: readonly track.Range[]): bigint | undefined { - let end: bigint | undefined; +function judged(ranges: readonly track.Range[]): Run[] { + const runs: Run[] = []; for (const range of ranges) { - if (range.end?.absolute === undefined) return undefined; - if (end === undefined || range.end.absolute > end) end = range.end.absolute; + const start = range.start?.absolute ?? 0n; + const end = range.end?.absolute ?? UNBOUNDED; + if (start < end) insertRun(runs, start, end); } - return end; + return runs; } /** Subscribe to the peer and own its receive-side protocol tasks. @@ -1568,8 +1572,8 @@ export class Subscriber { resolvesStart: resolvesStart(this.version), fillsFromStart: fillsFromStart(this.version), }); - // The SUBSCRIBE's terms: where a stale drop judged under them stops looking. - hooks.supersedeStale(producer, judgedEnd(ranges)); + // The SUBSCRIBE's terms: the groups a stale drop under them was judged against. + hooks.supersedeStale(producer, judged(ranges)); if (!hasRanges(this.version)) hooks.dropAnchors(producer); // Lite-06+ resolves where the feed starts only with SUBSCRIBE_START, so a relay serving // this copy holds its first group until the answer arrives. @@ -2371,7 +2375,7 @@ export class Subscriber { entry.update = update.update; entry.maxDelay = update.maxDelay; entry.tail.updated(); - const held = hooks.supersedeStale(entry.track, judgedEnd(ranges)); + const held = hooks.supersedeStale(entry.track, judged(ranges)); // A lite-07 publisher re-offers every group it reported stale on any update, and drops // again under the new number what is still stale, so the tail owes every stale run again. // A group that arrived since its stale drop won over it: the publisher served it, so it diff --git a/js/net/src/track.test.ts b/js/net/src/track.test.ts index 5dcb51c50dd..04f0aa0a63f 100644 --- a/js/net/src/track.test.ts +++ b/js/net/src/track.test.ts @@ -3504,7 +3504,7 @@ test("a superseded received stale report stays owed after an update", () => { producer.close(); }); -test("a received stale run settles a bounded reader only to the judged end", () => { +test("a received stale run settles a bounded reader only through the judged ranges", () => { const producer = new TrackProducer("video").accept({ maxAge: null }); hooks.receiveTrack(producer); const bounded = (max: number) => @@ -3515,7 +3515,7 @@ test("a received stale run settles a bounded reader only to the judged end", () const wider = bounded(5000); const matching = bounded(500); // The upstream judged its own range of 0..2 under 500ms. - hooks.supersedeStale(producer, 2n); + hooks.supersedeStale(producer, [{ start: 0n, end: 2n }]); hooks.dropTrackGroups(producer, 0n, 2n, staleHole(500)); // A wider budget still wants the groups, however the feed accounted for them. expect(hooks.tryRecvGroup(wider, false)).toEqual({ kind: "idle" }); @@ -3544,15 +3544,15 @@ test("a superseded stale run leaves a bounded reader waiting, and a later reader ranges: [{ start: { absolute: 0n }, end: { absolute: 2n } }], }); const wider = bounded(5000); - hooks.supersedeStale(producer, 2n); + hooks.supersedeStale(producer, [{ start: 0n, end: 2n }]); producer.createGroup(2n).finish(); hooks.dropTrackGroups(producer, 0n, 2n, staleHole(500)); - // A reader added after the verdict judges it against the same judged end. + // A reader added after the verdict judges it against the same judged ranges. const later = bounded(500); expect(hooks.tryDrop(later)).toMatchObject({ start: 0n, end: 2n, stale: 500 }); expect(hooks.tryRecvGroup(wider, false)).toEqual({ kind: "idle" }); // New terms make the groups unknown again: still wanted, never proven delivered. - hooks.supersedeStale(producer, 2n); + hooks.supersedeStale(producer, [{ start: 0n, end: 2n }]); expect(hooks.tryRecvGroup(wider, false)).toEqual({ kind: "idle" }); for (const reader of [wider, later]) reader.close(); producer.finish(); @@ -4584,11 +4584,11 @@ test("a skip is reported over a reported drop", () => { }); // A received stale run is judged from where it meets the reader's span, as Rust clips it: a reader -// asking only its suffix, up to where the upstream's range ended, is settled by that suffix. +// asking only its suffix, through every group the upstream's range named, is settled by that suffix. test("a received stale run settles the suffix a narrower range asks for", () => { const producer = new TrackProducer("video").accept({}); hooks.receiveTrack(producer); - hooks.supersedeStale(producer, 10n); + hooks.supersedeStale(producer, [{ start: 0n, end: 10n }]); const reader = producer.subscribe({ delay: { max: Milli(500) }, ranges: [{ start: { absolute: 5n }, end: { absolute: 10n } }], diff --git a/js/net/src/track.ts b/js/net/src/track.ts index 55eb0edc733..06412ecdace 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -705,9 +705,9 @@ class TrackState { // DROP costs one entry. Replaced rather than mutated, so a cursor can tell it changed by // identity. drops = new Signal([]); - // Where the ranges a received copy's upstream judges stale runs against end: the exclusive end - // of its current terms, undefined past the live edge or unknown. - judgedEnd?: bigint; + // Every group the ranges of a received copy's current upstream terms name, which its stale runs + // were judged against; undefined when unknown, which counts as every group. + judged?: readonly Run[]; // Every group still in the producer's replay cache, including groups this // subscriber already consumed, sorted by sequence. Drift anchors have the same // lifetime as content. Sorted so the latency guard, evaluated per group and per @@ -845,8 +845,18 @@ let startsOf: (producer: Producer) => readonly Start[]; // deliver, undefined while it may; assigned in its static block. let goneAt: (producer: Producer, sequence: bigint) => bigint | undefined; -/** A reader's last contiguous run of groups: its start, and its exclusive end unless it reads past the live edge. */ -type Coverage = { start: bigint; end?: bigint } | undefined; +/** The groups a reader reads, and the exclusive end it stops at; undefined before its ranges resolve. */ +type Coverage = { runs: readonly Run[]; cap?: bigint } | undefined; + +/** Whether two coverages hold the same groups and cap. */ +function sameCoverage(a: Coverage, b: Coverage): boolean { + if (a === undefined || b === undefined) return a === b; + return ( + a.cap === b.cap && + a.runs.length === b.runs.length && + a.runs.every((run, i) => run.start === b.runs[i].start && run.end === b.runs[i].end) + ); +} /** * Whether the live edge `received` reached `final`: no sequence below it is left that was neither @@ -858,17 +868,28 @@ function reaches(received: bigint, drops: readonly Run[], final: bigint): boolea /** * Whether `run` settles a reader judging under `budget` with `coverage`, on a track whose upstream - * judged stale runs against ranges ending at `judgedEnd` (undefined past the live edge, or - * unknown). Any run but a received stale one does. A stale verdict holds for a reader no wider - * than the budget it was judged under, whose range covers the run and every group above it up to - * where the upstream's range ended: its own successor and edge are then no older than those the - * verdict was judged against. Mirrors the Rust `settles`. + * judged stale runs against ranges naming the groups in `judged` (undefined for every group). Any + * run but a received stale one does. A stale verdict holds for a reader no wider than the budget it + * was judged under that reads every group the upstream's ranges name above the run's start: its + * own successor and edge are then no older than those the verdict was judged against, since a + * group's verdict looks only above it. Mirrors the Rust `settles`. */ -function settles(run: DropRun, budget: number, coverage: Coverage, judgedEnd: bigint | undefined): boolean { +function settles(run: DropRun, budget: number, coverage: Coverage, judged: readonly Run[] | undefined): boolean { if (run.stale === undefined) return true; - if (budget > run.stale || coverage === undefined || coverage.start > run.start) return false; - if (coverage.end === undefined) return true; - return judgedEnd !== undefined && coverage.end >= judgedEnd; + if (budget > run.stale || coverage === undefined) return false; + const above = run.start + 1n; + // Whether the reader reads every group from `start` up to the exclusive `end`, `UNBOUNDED` + // past the live edge. + const reads = (start: bigint, end: bigint): boolean => { + if (coverage.cap !== undefined && (end === UNBOUNDED || end > coverage.cap)) return false; + const held = coverage.runs.find((run) => run.start <= start && start < run.end); + return held !== undefined && held.end >= end; + }; + for (const range of judged ?? [{ start: 0n, end: UNBOUNDED }]) { + if (range.end <= above) continue; + if (!reads(max(range.start, above), range.end)) return false; + } + return true; } // Actual object metadata outlives cached payloads, unlike empty group heads in model largest. let objectLargest: (producer: Producer) => Position | undefined; @@ -1111,10 +1132,10 @@ export class Producer { hooks.trackFinished = (producer) => producer.#state.end.peek() !== undefined; hooks.dropTrackGroups = (producer, start, end, hole) => producer.#drop(start, end, hole, true); hooks.skipTrackGroups = (producer, start, end) => producer.#skipReceived(start, end); - hooks.supersedeStale = (producer, judgedEnd) => { + hooks.supersedeStale = (producer, judged) => { const held = producer.#state.drops.peek().filter((run) => run.stale !== undefined); for (const state of [producer.#state, ...producer.#sinks]) { - state.judgedEnd = judgedEnd; + state.judged = judged; const drops = state.drops.peek(); if (drops.some((run) => run.stale !== undefined)) state.drops.set(drops.filter((run) => run.stale === undefined)); @@ -1450,7 +1471,7 @@ export class Producer { this.#prune(); for (const entry of this.#cache) this.#mirror(entry, sink); sink.drops.set(this.#state.drops.peek()); - sink.judgedEnd = this.#state.judgedEnd; + sink.judged = this.#state.judged; sink.received.set(this.#received); sink.endError = this.#state.endError; @@ -2284,7 +2305,7 @@ export class Subscriber { revision: number; budget: number; coverage?: Coverage; - judgedEnd?: bigint; + judged?: readonly Run[]; }; #reported: Run[] = []; // The reported runs judged stale, for a wire whose update re-offers them; see @@ -2926,7 +2947,7 @@ export class Subscriber { // update supersedes it: asked even above the newest group received. for (const dropped of drops) { const span = { ...dropped, start: max(dropped.start, max(low, edge)), end: min(dropped.end, end) }; - if (span.start < span.end && !settles(span, budget, coverage, this.#state.judgedEnd)) spans.push(span); + if (span.start < span.end && !settles(span, budget, coverage, this.#state.judged)) spans.push(span); } for (const run of spans.flatMap((span) => this.#wants(span.start, span.end))) { const gone = [...accounted]; @@ -2934,7 +2955,7 @@ export class Subscriber { if (dropped.end <= run.start || run.end <= dropped.start) continue; // Judged from where it meets the asked run, as the reader would read it. const clipped = { ...dropped, start: max(dropped.start, run.start) }; - if (settles(clipped, budget, coverage, this.#state.judgedEnd)) + if (settles(clipped, budget, coverage, this.#state.judged)) insertRun(gone, clipped.start, dropped.end); } for (const open of runGaps(gone, run.start, run.end)) { @@ -3081,8 +3102,8 @@ export class Subscriber { const overlapping = all .filter((run) => run.start < gap.end && gap.start < run.end) .map((run) => ({ ...run, start: max(run.start, gap.start), end: min(run.end, gap.end) })); - const drops = overlapping.filter((run) => settles(run, budget, coverage, this.#state.judgedEnd)); - const unknown = overlapping.filter((run) => !settles(run, budget, coverage, this.#state.judgedEnd)); + const drops = overlapping.filter((run) => settles(run, budget, coverage, this.#state.judged)); + const unknown = overlapping.filter((run) => !settles(run, budget, coverage, this.#state.judged)); for (const open of runGaps(drops, gap.start, gap.end)) { if (unknown.some((run) => run.start < open.end && open.start < run.end)) return false; // Not received yet, so it may still arrive. @@ -3685,7 +3706,7 @@ export class Subscriber { const revision = this.#revision; const { budget } = this.#drift(); const coverage = this.#coverage(); - const judgedEnd = this.#state.judgedEnd; + const judged = this.#state.judged; const seen = this.#dropsSeen; if ( !seen || @@ -3694,11 +3715,10 @@ export class Subscriber { seen.end !== end || seen.revision !== revision || seen.budget !== budget || - seen.coverage?.start !== coverage?.start || - seen.coverage?.end !== coverage?.end || - seen.judgedEnd !== judgedEnd + !sameCoverage(seen.coverage, coverage) || + seen.judged !== judged ) { - this.#dropsSeen = { drops, start, end, revision, budget, coverage, judgedEnd }; + this.#dropsSeen = { drops, start, end, revision, budget, coverage, judged }; // The track's runs are folded in afresh, so one it forgot, such as a stale run new terms // superseded, or one that stopped settling this reader, leaves nothing behind. this.#pending = this.#pending.filter((run) => !run.received); @@ -3707,7 +3727,7 @@ export class Subscriber { const run = { ...dropped, start: max(dropped.start, start), end: min(dropped.end, end ?? dropped.end) }; if (run.start >= run.end) continue; // A received stale run that does not settle this reader is unknown to it. - if (!settles(run, budget, coverage, judgedEnd)) continue; + if (!settles(run, budget, coverage, judged)) continue; this.#pending = insertDrop(this.#pending, run.start, run.end, { ...run, received: true }); } } @@ -3720,14 +3740,10 @@ export class Subscriber { return undefined; } - // This reader's last contiguous run of groups, capped; see `Coverage`. + // The groups this reader reads, and its cap; see `Coverage`. #coverage(): Coverage { if (this.#window.pending) return undefined; - const last = this.#window.runs().at(-1); - if (!last) return undefined; - const cap = this.#cursor.peek().end; - const end = last.end === UNBOUNDED ? cap : cap === undefined ? last.end : min(last.end, cap); - return { start: last.start, end }; + return { runs: this.#window.runs(), cap: this.#cursor.peek().end }; } // A group from the routes' copies; see #resume. diff --git a/quest/m1/drop-set.md b/quest/m1/drop-set.md index e54f816ee43..c2a7e791366 100644 --- a/quest/m1/drop-set.md +++ b/quest/m1/drop-set.md @@ -235,6 +235,24 @@ review stand as recorded, numbered as in its body: run decided stays the upstream's to judge after an update (`DropCursor::reported_received`, JS `#reportedReceived`). +Decided 2026-11-06 (maintainer, /quest-complete), replacing agent decision 27 +of [moq-dev/moq#7670](https://github.com/moq-dev/moq/pull/7670), which asked a +fill for every unknown stale run instead: + +- ✅ **27: a stale verdict accepts every range.** A relay keeps every group + its current upstream terms' ranges name (`TrackState::judged`, JS + `judged`), taking a bound only the publisher resolves at its widest. A + received stale run settles a reader no wider than the budget it was judged + under that reads every one of those groups above the run's start, however + many ranges they span, since a group's verdict looks only above it. A + reader with the upstream's own ranges is settled with nothing received. A + reader narrower than the upstream's ranges still treats the run as unknown, + and asks a fill for it even above the newest group received, since its + ranges add nothing to the upstream's terms and no update brings the run + back. Not chosen: asking a fill for every unknown stale run without the + multi-range verdict; reversing the fix and tracking the hang as a + follow-up. + Decided 2026-10-27 (maintainer): this runs after the range count cost work, which finished in [moq-dev/moq#7542](https://github.com/moq-dev/moq/pull/7542), and before [Rust groups by start](/quest/m2/time-addressing/model.md), which diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index eae0c1251c0..ea2dc621ccc 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -9723,7 +9723,7 @@ impl TrackServe { // Open an upstream SUBSCRIBE for the first subscriber. Its terms are new, so a // stale run an earlier subscription reported is no verdict on them. Sub::None => { - producer.supersede_stale(judged_end(&wire.ranges)); + producer.supersede_stale(judged(&wire.ranges)); Ok(Begin::Establish(self.prepare_establish(producer, wire, &fills.groups))) } Sub::Active(active) => { @@ -10062,19 +10062,25 @@ fn wire_max_delay(max_delay: Option) -> std::time::Duration max_delay.unwrap_or(std::time::Duration::from_millis(super::MAX_SAFE_DELAY_MS)) } -/// Where a stale drop judged under `ranges` stops looking: the exclusive end past every -/// group they name, `None` when one stays open or ends at an anchor only the publisher -/// resolves. -fn judged_end<'a>(ranges: impl IntoIterator) -> Option { - ranges - .into_iter() - .map(|range| match range.end { - Some(lite::Anchor::Absolute(end)) => Some(end.group.saturating_add(u64::from(end.frame > 0))), - _ => None, - }) - .collect::>>()? - .into_iter() - .max() +/// Every group `ranges` name, which a stale drop under them was judged against. A bound only +/// the publisher resolves is taken at its widest: a start from group 0, an end past the live +/// edge (an exclusive end of `u64::MAX`). +fn judged<'a>(ranges: impl IntoIterator) -> Option> { + let mut judged = RangeSet::new(); + for range in ranges { + let start = match range.start { + Some(lite::Anchor::Absolute(start)) => start.group, + _ => 0, + }; + let end = match range.end { + Some(lite::Anchor::Absolute(end)) => end.group.saturating_add(u64::from(end.frame > 0)), + _ => u64::MAX, + }; + if start < end { + judged.insert(start..end); + } + } + Some(judged) } /// Buffer the current SUBSCRIBE parameters as the next update, with `ranges` the change to @@ -10095,7 +10101,7 @@ fn buffer_update( now: crate::time::Instant, ) -> Result<(), Error> { active.update += 1; - let held = serving.supersede_stale(judged_end(active.live.iter().map(|(_, range)| range))); + let held = serving.supersede_stale(judged(active.live.iter().map(|(_, range)| range))); let recorded = std::mem::take(&mut active.stale); if let Ok(mut tail) = active.tail.write() { tail.updated(); diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index dfa963b6714..af8a6c57313 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -521,10 +521,11 @@ pub(crate) struct TrackState { // One past the last `append_group`: every sequence below it arrived, was given up, or // was skipped. appended: Option, - // Where the ranges a received copy's upstream judges stale runs against end: the - // exclusive end of its current terms, `None` past the live edge or unknown. - judged_end: Option, - // Bumped on every insert into `drops`, so a cursor rescans it only after a change. + // Every group the ranges of a received copy's current upstream terms name, which its + // stale runs were judged against; `None` when unknown, which counts as every group. + judged: Option>, + // Bumped on every insert into `drops`, and when `judged` changes, so a cursor rescans + // it only after a change. drops_revision: u64, drops_changes: ledger::Changes, // How many copies the cache took in that the live feed didn't deliver: fetched backfill, @@ -3878,15 +3879,18 @@ impl Producer { /// Forget every run held as stale: the terms a received copy's upstream judged them /// under were replaced, so they are unknown again, and a reader that wants one fills it. - /// The new terms' ranges end at `judged_end`, `None` past the live edge or unknown. - /// Returns the runs forgotten; a group that arrived since its stale drop replaced it, so - /// it is not among them. + /// `judged` is every group the new terms' ranges name, `None` when unknown. Returns the + /// runs forgotten; a group that arrived since its stale drop replaced it, so it is not + /// among them. #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] - pub(crate) fn supersede_stale(&self, judged_end: Option) -> RangeSet { + pub(crate) fn supersede_stale(&self, judged: Option>) -> RangeSet { let Ok(mut state) = self.modify() else { return RangeSet::new(); }; - state.judged_end = judged_end; + if state.judged != judged { + state.judged = judged; + state.drops_revision += 1; + } state.supersede_stale() } @@ -6261,41 +6265,56 @@ struct Drift { /// The budget is the track's max age bound, which no widening passes: its verdicts are final. bounded: bool, edge: Edge, - /// This reader's last contiguous run of groups: where it starts and its exclusive end, - /// `None` when it reads up past the live edge; see [`Coverage`]. + /// The groups this reader reads; see [`Coverage`]. coverage: Coverage, } -/// A reader's last contiguous run of groups, its start and exclusive end (`None` past the -/// live edge), or `None` before its ranges resolve. -type Coverage = Option<(u64, Option)>; +/// The groups a reader reads, and the exclusive end it stops at, or `None` before its ranges +/// resolve. +type Coverage = Option<(Layered, Option)>; /// Whether a reader judging under `budget` with `coverage` is settled by `hole` over a run -/// from `start`, on a track whose upstream judged stale runs against ranges ending at -/// `judged_end` (`None` past the live edge, or unknown). Any hole but a received stale one -/// settles every reader. A stale verdict holds for a reader no wider than the budget it was -/// judged under, whose range covers the run and every group above it up to where the -/// upstream's range ended: its own successor and edge are then no older than those the -/// verdict was judged against. Any other reader treats the run as unknown, and fills it. -fn settles(budget: Duration, coverage: Coverage, start: u64, hole: &Hole, judged_end: Option) -> bool { - let Some(judged) = hole.stale_budget() else { +/// from `start`, on a track whose upstream judged stale runs against ranges naming the groups +/// in `judged` (`None` for every group). Any hole but a received stale one settles every +/// reader. A stale verdict holds for a reader no wider than the budget it was judged under +/// that reads every group the upstream's ranges name above the run's start: its own successor +/// and edge are then no older than those the verdict was judged against, since a group's +/// verdict looks only above it. Any other reader treats the run as unknown, and fills it. +fn settles(budget: Duration, coverage: &Coverage, start: u64, hole: &Hole, judged: Option<&RangeSet>) -> bool { + let Some(stale) = hole.stale_budget() else { return true; }; - let covers = coverage.is_some_and(|(from, until)| { - from <= start - && match (until, judged_end) { - (None, _) => true, - (Some(until), Some(end)) => until >= end, - (Some(_), None) => false, - } - }); - budget <= judged && covers + let Some((groups, cap)) = coverage else { + return false; + }; + let Some(above) = start.checked_add(1) else { + return budget <= stale; + }; + // Whether the reader reads every group of `run`, an exclusive end of `u64::MAX` past the + // live edge. + let reads = |run: ops::Range| { + let open = run.end == u64::MAX; + if cap.is_some_and(|cap| open || run.end > cap) { + return false; + } + groups.first(run.start, run.start).is_some_and(|held| match open { + true => *held.end() == u64::MAX, + false => *held.end() >= run.end - 1, + }) + }; + let covers = match judged { + Some(judged) => judged + .overlapping(&(above..u64::MAX)) + .all(|run| reads(run.start.max(above)..run.end)), + None => reads(above..u64::MAX), + }; + budget <= stale && covers } impl Drift { /// Whether `hole`, over a run from `start`, settles this reader on `state`; see [`settles`]. fn settles(&self, state: &TrackState, start: u64, hole: &Hole) -> bool { - settles(self.budget, self.coverage, start, hole, state.judged_end) + settles(self.budget, &self.coverage, start, hole, state.judged.as_ref()) } } @@ -7059,7 +7078,7 @@ impl Pending { /// The [`TrackState::drops_revision`], span, window revision, and the budget and coverage /// that settle received stale runs (see [`Drift::settles`]) a [`DropCursor`] folded in. -type Scan = (u64, ops::Range, u64, Option<(Duration, Coverage, Option)>); +type Scan = (u64, ops::Range, u64, Option<(Duration, Coverage)>); /// What a wire publisher's cursor gave up, found as it changes rather than by rescanning /// the cache on every poll. @@ -7181,12 +7200,9 @@ impl DropCursor { // Rescan only after the drops change, the window moves, or what settles a stale run // does; out-of-window and unsettling runs were skipped and may be owed now. if let Some(drift) = within.drift { - self.drift = Some((drift.budget, drift.coverage)); + self.drift = Some((drift.budget, drift.coverage.clone())); } - let judged = self - .drift - .map(|(budget, coverage)| (budget, coverage, state.judged_end)); - let scan = (state.drops_revision, span.clone(), within.revision, judged); + let scan = (state.drops_revision, span.clone(), within.revision, self.drift.clone()); if self.scanned.as_ref() != Some(&scan) { self.scanned = Some(scan); let stale: Vec<_> = self @@ -7201,8 +7217,8 @@ impl DropCursor { for (dropped, hole) in state.drops.overlapping(span.clone()) { let run = dropped.start.max(span.start)..dropped.end.min(span.end); // A received stale run that does not settle this reader is unknown to it. - let settled = match self.drift { - Some((budget, coverage)) => settles(budget, coverage, run.start, hole, state.judged_end), + let settled = match &self.drift { + Some((budget, coverage)) => settles(*budget, coverage, run.start, hole, state.judged.as_ref()), None => hole.stale_budget().is_none(), }; if settled { @@ -7531,31 +7547,30 @@ impl Cursor { let coverage = self.coverage(); let state = self.state.read(); let budget = clamp_max_delay(self.subscription.read().budget(), state.max_age_bound()); - let accounted = |gap: ops::Range| { - let mut left = RangeSet::new(); - left.insert(gap.clone()); - for run in self.filter.written_off.overlapping(&gap) { - left.remove(run.clone()); - } - if let Some(given_up) = given_up { - for run in given_up.overlapping(&gap) { + let accounted = + |gap: ops::Range| { + let mut left = RangeSet::new(); + left.insert(gap.clone()); + for run in self.filter.written_off.overlapping(&gap) { left.remove(run.clone()); } - } - // A received stale run that does not settle this reader is unknown to it, so it - // still wants those groups. - for (run, _) in state - .drops - .overlapping(&gap) - .filter(|(run, hole)| settles(budget, coverage, run.start.max(gap.start), hole, state.judged_end)) - { - left.remove(run.clone()); - } - for sequence in self.seek_pending.range(gap).map(|(sequence, _)| *sequence) { - left.remove(sequence..sequence + 1); - } - left.is_empty() - }; + if let Some(given_up) = given_up { + for run in given_up.overlapping(&gap) { + left.remove(run.clone()); + } + } + // A received stale run that does not settle this reader is unknown to it, so it + // still wants those groups. + for (run, _) in state.drops.overlapping(&gap).filter(|(run, hole)| { + settles(budget, &coverage, run.start.max(gap.start), hole, state.judged.as_ref()) + }) { + left.remove(run.clone()); + } + for sequence in self.seek_pending.range(gap).map(|(sequence, _)| *sequence) { + left.remove(sequence..sequence + 1); + } + left.is_empty() + }; self.filter .window .runs(self.filter.window.floor().unwrap_or(0)) @@ -7853,19 +7868,9 @@ impl Cursor { std::mem::take(&mut self.stale) } - /// This reader's last contiguous run of groups, capped; see [`Coverage`]. + /// The groups this reader reads, and its cap; see [`Coverage`]. fn coverage(&self) -> Coverage { - let run = self.filter.window.groups()?.last(0, u64::MAX)?; - let end = match *run.end() { - u64::MAX => None, - last => Some(last + 1), - }; - let end = match (end, self.end_sequence) { - (end, None) => end, - (None, Some(cap)) => Some(cap), - (Some(end), Some(cap)) => Some(end.min(cap)), - }; - Some((*run.start(), end)) + Some((self.filter.window.groups()?.clone(), self.end_sequence)) } /// This subscriber's clamped drift budget and the live edge to measure against, @@ -7891,7 +7896,7 @@ impl Cursor { budget: clamp_max_delay(max_delay, bound), bounded: bound.is_some_and(|bound| max_delay >= bound), edge: state.drift_edge(servable), - coverage, + coverage: coverage.clone(), })) }) } @@ -22730,6 +22735,9 @@ mod test { #[cfg(feature = "async")] fn a_received_stale_drop_settles_only_the_readers_it_covers() { let producer = track_producer("test", Info::timed()); + // A received copy's serving session keeps a handler, so an unsettled run is asked + // rather than given up as unfillable. + let _dynamic = producer.requests(); let budget = Duration::from_millis(500); let reader = |max: Duration, groups: Range| { producer.subscribe( @@ -22825,37 +22833,126 @@ mod test { assert_eq!((fill.start().group, fill.end()), (0, 3)); } - /// A stale verdict a reader cannot trust is asked for even when no group arrived to bound - /// the misses, so the reader still settles. Mirrors JS. + /// Every group a set of ranges names, as a received copy's upstream terms judge them. + fn judged(runs: impl IntoIterator>) -> Option> { + Some(runs.into_iter().collect()) + } + + /// A stale verdict settles a reader that reads every range the upstream judged it + /// against, however many there are, with nothing received and no fill asked. Mirrors JS. #[test] #[cfg(feature = "async")] - fn an_unsettled_stale_run_is_asked_for_with_nothing_received() { + fn a_stale_verdict_settles_a_reader_of_every_range_it_judged() { let producer = track_producer("test", Info::timed()); producer.received(); let dynamic = producer.requests(); let budget = Duration::from_millis(500); - // The upstream judged ranges ending at 6 under 500ms, and gave up both of them as stale. - producer.supersede_stale(Some(6)); + // The upstream judged 0..2 and 4..6 under 500ms, and gave up both of them as stale. + producer.supersede_stale(judged([0..2, 4..6])); producer.give_up(0..2, Hole::stale(budget)); producer.give_up(4..6, Hole::stale(budget)); - // The same ranges: only the last one is contiguous to the judged end, so the first is - // unknown. let mut subscriber = producer.subscribe( Subscription::default() .with_delay(Delay::default().with_max(budget)) .with_ranges([Range::groups(0..2), Range::groups(4..6)]), ); let waiter = kio::Waiter::noop(); - assert!(subscriber.poll_recv_group(&waiter).is_pending()); + assert_eq!(subscriber.poll_drop(&waiter), Poll::Ready((0..2, Hole::stale(budget)))); + assert_eq!(subscriber.poll_drop(&waiter), Poll::Ready((4..6, Hole::stale(budget)))); + assert!( + matches!(subscriber.poll_recv_group(&waiter), Poll::Ready(Ok(None))), + "nothing is owed" + ); + assert!(dynamic.poll_requested_fill(&waiter).is_pending(), "nothing is asked"); + } + + /// A reader whose ranges only partly overlap the ones a stale verdict was judged against + /// is settled where it reads every judged group above the run, and asks a fill for the + /// rest, even with nothing received: no update brings it, since its ranges add nothing to + /// the upstream's terms. Mirrors JS. + #[test] + #[cfg(feature = "async")] + fn a_stale_verdict_settles_a_partly_overlapping_reader_only_where_it_reads_above() { + let producer = track_producer("test", Info::timed()); + producer.received(); + let dynamic = producer.requests(); + let budget = Duration::from_millis(500); + producer.supersede_stale(judged([0..2, 4..6])); + producer.give_up(0..2, Hole::stale(budget)); + producer.give_up(4..6, Hole::stale(budget)); + let reader = |ranges: [Range; 2]| { + producer.subscribe( + Subscription::default() + .with_delay(Delay::default().with_max(budget)) + .with_ranges(ranges), + ) + }; + let waiter = kio::Waiter::noop(); + // Reads every judged group above each run, and more past them: settled. + let mut wider = reader([Range::groups(1..2), Range::groups(4..8)]); + assert_eq!(wider.poll_drop(&waiter), Poll::Ready((1..2, Hole::stale(budget)))); + assert_eq!(wider.poll_drop(&waiter), Poll::Ready((4..6, Hole::stale(budget)))); + assert!(wider.poll_drop(&waiter).is_pending()); + assert!(dynamic.poll_requested_fill(&waiter).is_pending(), "nothing is asked"); + // Stops at 4, which the upstream judged against 5: its own newest, which no verdict + // convicts. Both runs are unknown to it, and asked for. + let mut narrower = reader([Range::groups(0..2), Range::groups(4..5)]); + assert!(narrower.poll_recv_group(&waiter).is_pending()); + assert!(narrower.poll_drop(&waiter).is_pending()); + let mut asked = Vec::new(); + while let Poll::Ready(Ok(fill)) = dynamic.poll_requested_fill(&waiter) { + asked.push((fill.start().group, fill.end())); + } + asked.sort(); + assert_eq!(asked, [(0, 2), (4, 5)]); + } + + /// A fill taken before the last handler closed still answers an unsettled stale run: a + /// reader that asks afterwards joins it while it runs, and reads its groups from the cache + /// once it is done. + #[test] + #[cfg(feature = "async")] + fn a_taken_fill_answers_an_unsettled_stale_run_after_its_handler_closes() { + let producer = track_producer("test", Info::timed()); + producer.received(); + let dynamic = producer.requests(); + let budget = Duration::from_millis(500); + producer.supersede_stale(judged([0..2, 4..6])); + producer.give_up(0..2, Hole::stale(budget)); + producer.give_up(4..6, Hole::stale(budget)); + // Stops at 2, though the upstream judged 0..2 against 4..6 too: the run is unknown to it. + let reader = || { + producer.subscribe( + Subscription::default() + .with_delay(Delay::default().with_max(budget)) + .with_ranges([Range::groups(0..2)]), + ) + }; + let waiter = kio::Waiter::noop(); + let mut first = reader(); + assert!(first.poll_recv_group(&waiter).is_pending()); let Poll::Ready(Ok(fill)) = dynamic.poll_requested_fill(&waiter) else { panic!("the unsettled stale run is asked for"); }; assert_eq!((fill.start().group, fill.end()), (0, 2)); + drop(dynamic); + let mut joined = reader(); + assert!(joined.poll_recv_group(&waiter).is_pending(), "it joins the taken fill"); + for sequence in 0..2 { + let mut group = fill.accept(sequence, None).unwrap(); + group + .write_frame(Timestamp::from_millis(sequence).unwrap(), &b"x"[..]) + .unwrap(); + group.finish().unwrap(); + } drop(fill); - assert!( - matches!(subscriber.poll_recv_group(&waiter), Poll::Ready(Ok(None))), - "nothing is owed once the fill gave up" - ); + let mut late = reader(); + for subscriber in [&mut first, &mut joined, &mut late] { + let mut got = drain_groups(subscriber); + got.sort_unstable(); + assert_eq!(got, [0, 1]); + assert!(matches!(subscriber.poll_recv_group(&waiter), Poll::Ready(Ok(None)))); + } } /// Creating a group past the next sequence says nothing about the gap: groups may be @@ -23181,13 +23278,15 @@ mod test { assert_eq!(subscriber.poll_drop(&waiter), Poll::Ready((1..3, Hole::stale(budget)))); } - /// A bounded reader is settled by a received stale run only when its range reaches where - /// the upstream's did; otherwise it still waits for the groups the run names. + /// A bounded reader is settled by a received stale run only when it reads every group the + /// upstream's ranges name above the run; otherwise it still wants the groups the run names. #[test] #[cfg(feature = "async")] - fn a_received_stale_run_settles_a_bounded_reader_only_to_the_judged_end() { + fn a_received_stale_run_settles_a_bounded_reader_only_through_the_judged_ranges() { let producer = track_producer("test", Info::timed()); producer.received(); + // As its serving session does, so an unsettled run is asked rather than given up. + let _dynamic = producer.requests(); let bounded = |max: Duration| { producer.subscribe( Subscription::default() @@ -23199,7 +23298,7 @@ mod test { let mut matching = bounded(Duration::from_millis(500)); let waiter = kio::Waiter::noop(); // The upstream judged its own range of 0..2 under 500ms. - producer.supersede_stale(Some(2)); + producer.supersede_stale(judged([0..2])); producer.give_up(0..2, Hole::stale(Duration::from_millis(500))); assert!( wider.poll_recv_group(&waiter).is_pending(), diff --git a/rs/moq-net/src/model/track/miss.rs b/rs/moq-net/src/model/track/miss.rs index a8689210634..42dbabcc797 100644 --- a/rs/moq-net/src/model/track/miss.rs +++ b/rs/moq-net/src/model/track/miss.rs @@ -627,22 +627,16 @@ impl Misses { return asks; } // A received stale run this reader cannot trust is still asked for, though nothing above - // it arrived; see `TrackState::unfed`. Only runs above the newest group need a look, and - // only while a handler may fill them: otherwise the reader waits for new terms. + // it arrived: only an update brings it back otherwise, and a reader narrower than the + // upstream's terms changes none. See `TrackState::unfed`. Only runs above the newest + // group need a look. let unknown = state - .fills - .read() - .has_handlers() - .then(|| { - state - .drops - .iter() - .rev() - .take_while(|(run, _)| state.newest_produced.is_none_or(|newest| run.end - 1 > newest)) - .find(|(run, hole)| !drift.settles(state, run.start, hole)) - .map(|(run, _)| run.end - 1) - }) - .flatten(); + .drops + .iter() + .rev() + .take_while(|(run, _)| state.newest_produced.is_none_or(|newest| run.end - 1 > newest)) + .find(|(run, hole)| !drift.settles(state, run.start, hole)) + .map(|(run, _)| run.end - 1); let Some(newest) = state.newest_produced.max(bound).max(unknown) else { return asks; }; From fff5f03acdf7b14fd884ddc2f4fffaba0eb3b332 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 6 Nov 2026 11:31:34 -0800 Subject: [PATCH 62/68] fix(net): satisfy clippy and biome on the multi-range verdict Co-Authored-By: Claude Opus 5.5 --- js/net/src/lite/subscriber.ts | 2 +- rs/moq-net/src/model/track.rs | 12 ++++++------ 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/js/net/src/lite/subscriber.ts b/js/net/src/lite/subscriber.ts index 5c710d95fff..2c86a3bc3a8 100644 --- a/js/net/src/lite/subscriber.ts +++ b/js/net/src/lite/subscriber.ts @@ -32,8 +32,8 @@ import { insertRun, maxDelayOf, minDelayOf, - removeRun, type Run, + removeRun, scopeCaptures, scopeHead, scopeOverlaps, diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index af8a6c57313..b03f158d8f5 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -22834,8 +22834,8 @@ mod test { } /// Every group a set of ranges names, as a received copy's upstream terms judge them. - fn judged(runs: impl IntoIterator>) -> Option> { - Some(runs.into_iter().collect()) + fn judged(runs: impl IntoIterator) -> Option> { + Some(runs.into_iter().map(|(start, end)| start..end).collect()) } /// A stale verdict settles a reader that reads every range the upstream judged it @@ -22848,7 +22848,7 @@ mod test { let dynamic = producer.requests(); let budget = Duration::from_millis(500); // The upstream judged 0..2 and 4..6 under 500ms, and gave up both of them as stale. - producer.supersede_stale(judged([0..2, 4..6])); + producer.supersede_stale(judged([(0, 2), (4, 6)])); producer.give_up(0..2, Hole::stale(budget)); producer.give_up(4..6, Hole::stale(budget)); let mut subscriber = producer.subscribe( @@ -22877,7 +22877,7 @@ mod test { producer.received(); let dynamic = producer.requests(); let budget = Duration::from_millis(500); - producer.supersede_stale(judged([0..2, 4..6])); + producer.supersede_stale(judged([(0, 2), (4, 6)])); producer.give_up(0..2, Hole::stale(budget)); producer.give_up(4..6, Hole::stale(budget)); let reader = |ranges: [Range; 2]| { @@ -22917,7 +22917,7 @@ mod test { producer.received(); let dynamic = producer.requests(); let budget = Duration::from_millis(500); - producer.supersede_stale(judged([0..2, 4..6])); + producer.supersede_stale(judged([(0, 2), (4, 6)])); producer.give_up(0..2, Hole::stale(budget)); producer.give_up(4..6, Hole::stale(budget)); // Stops at 2, though the upstream judged 0..2 against 4..6 too: the run is unknown to it. @@ -23298,7 +23298,7 @@ mod test { let mut matching = bounded(Duration::from_millis(500)); let waiter = kio::Waiter::noop(); // The upstream judged its own range of 0..2 under 500ms. - producer.supersede_stale(judged([0..2])); + producer.supersede_stale(judged([(0, 2)])); producer.give_up(0..2, Hole::stale(Duration::from_millis(500))); assert!( wider.poll_recv_group(&waiter).is_pending(), From 9321f3bca023e0bd60c37c32df4ed304f8182e2f Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 6 Nov 2026 11:55:00 -0800 Subject: [PATCH 63/68] quest: record decision 28 for drop-set Co-Authored-By: Claude Opus 5.5 --- quest/m1/drop-set.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/quest/m1/drop-set.md b/quest/m1/drop-set.md index c2a7e791366..3e4e3e60843 100644 --- a/quest/m1/drop-set.md +++ b/quest/m1/drop-set.md @@ -252,6 +252,14 @@ fill for every unknown stale run instead: back. Not chosen: asking a fill for every unknown stale run without the multi-range verdict; reversing the fix and tracking the hang as a follow-up. +- ✅ **28 (2026-11-06, maintainer, /quest-complete): keep the fill for + readers narrower than the verdict's ranges**, with no fill-handler gate. A + verdict judged on the union of readers' ranges is valid only for readers + that read every group it names above the run's start; a narrower reader + still needs groups the upstream already dropped, so only a fill can + deliver them. Not chosen: re-judging per reader (it would still need the + fill); separate upstream subscriptions per reader shape (they give up + aggregation); deferring the narrow case as a follow-up. Decided 2026-10-27 (maintainer): this runs after the range count cost work, which finished in [moq-dev/moq#7542](https://github.com/moq-dev/moq/pull/7542), From 54cac9621ec1e86ea7e2e76e5f7c29845cec9b71 Mon Sep 17 00:00:00 2001 From: "pullfrog[bot]" <226033991+pullfrog[bot]@users.noreply.github.com> Date: Fri, 6 Nov 2026 20:32:23 +0000 Subject: [PATCH 64/68] fix(net): set update on the lite-05/06 subscribe test helper --- rs/moq-net/src/lite/publisher.rs | 1 + 1 file changed, 1 insertion(+) diff --git a/rs/moq-net/src/lite/publisher.rs b/rs/moq-net/src/lite/publisher.rs index 7c7023b13be..7e34b814577 100644 --- a/rs/moq-net/src/lite/publisher.rs +++ b/rs/moq-net/src/lite/publisher.rs @@ -10337,6 +10337,7 @@ mod tests { max_delay: Duration::ZERO, min_delay: None, ranges: vec![lite::Range::span(None, 0, None, None)], + update: 0, } } From 3167350e481a13084402562343d4a59c837741ac Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 6 Nov 2026 13:40:20 -0800 Subject: [PATCH 65/68] fix(net): pass a NonZeroU64 to origin::Config::new in two lite subscriber tests Main's #8007 changed the argument from Hop; these two tests came from this branch. Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/src/lite/subscriber.rs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index 513a1188cfc..fe2b7315364 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -4945,7 +4945,7 @@ mod tests { let subscriber = Subscriber::new(SubscriberConfig { runtime: clock.clone(), session: crate::driver::Transport::new(session.clone()), - origin: origin::Config::new(crate::Hop::new(1).unwrap()).produce(), + origin: origin::Config::new(std::num::NonZeroU64::new(1).unwrap()).produce(), recv_bandwidth: None, version, peer_setup: crate::lite::PeerSetup::new(false, crate::transport::Address::Setup), @@ -5270,7 +5270,7 @@ mod tests { let subscriber = Subscriber::new(SubscriberConfig { runtime: crate::time::sim(), session: crate::driver::Transport::new(session.clone()), - origin: origin::Config::new(crate::Hop::new(1).unwrap()).produce(), + origin: origin::Config::new(std::num::NonZeroU64::new(1).unwrap()).produce(), recv_bandwidth: None, version, peer_setup: crate::lite::PeerSetup::new(false, crate::transport::Address::Setup), From 30472607a66285901c5cec8697cb53805c7be7f4 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 6 Nov 2026 13:56:56 -0800 Subject: [PATCH 66/68] quest: datagram-end-close notes #8000 merged first Co-Authored-By: Claude Opus 5.5 --- quest/m0/test-flakes-2/datagram-end-close.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/quest/m0/test-flakes-2/datagram-end-close.md b/quest/m0/test-flakes-2/datagram-end-close.md index e84202e7c7f..44dbeb887ac 100644 --- a/quest/m0/test-flakes-2/datagram-end-close.md +++ b/quest/m0/test-flakes-2/datagram-end-close.md @@ -49,13 +49,15 @@ retrying the test are not fixes. ## Related -- [moq-dev/moq#8000](https://github.com/moq-dev/moq/pull/8000), open, edits - this case where it still runs on `main`: it stubs `Budget.prototype.poll` so the - publisher's write budget never yields a host task, and raises the - microtask count to 10000. Stubbing the budget removes the suspected - outstanding `yieldNow`, so check whether it is the cause before choosing - this fix; the raised count alone is not one. Whichever lands second merges - the other's hunk. +- [moq-dev/moq#8000](https://github.com/moq-dev/moq/pull/8000), merged + first, edited this case: it stubs + `Budget.prototype.poll` so the publisher's write budget never yields a host + task, and raises the microtask count to 10000. The quarantined case keeps + both. Stubbing the budget removes the suspected outstanding `yieldNow`, so + check whether it is the cause before choosing this fix; the raised count + alone is not one. On 2026-11-06, ten loaded runs (six busy loops pinned to + four of 24 cores) passed with and without the stub, so that setup no longer + reproduces it and cannot tell. - [moq-dev/moq#7981](https://github.com/moq-dev/moq/pull/7981), a draft, quarantines only the lite-06 case and adds its own quest at this same path. Whichever lands second keeps one quest and this whole-case From fa97eb708ed9bbcb02514bcd2adcfd66da488d52 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 6 Nov 2026 22:53:35 -0800 Subject: [PATCH 67/68] fix(net): a re-offer dropped again before it opens counts as given up A lite-07 update that widens the budget re-offers an expired group and removes it from the served set. When a later update restores the budget before that re-offer opens, the group goes out again as a stale drop, but stayed out of the served set, so the track's end sweep named it once more as a final NOT_FOUND. Both the Rust redrop path and the JS pendingDrops path now record the dropped run as served. Co-Authored-By: Claude Opus 5.5 --- js/net/src/lite/publisher.test.ts | 58 +++++++++++++++++++++++++++++++ js/net/src/lite/publisher.ts | 2 ++ rs/moq-net/src/lite/publisher.rs | 45 ++++++++++++++++++++++++ 3 files changed, 105 insertions(+) diff --git a/js/net/src/lite/publisher.test.ts b/js/net/src/lite/publisher.test.ts index ceec354c06d..067f2c4c6d6 100644 --- a/js/net/src/lite/publisher.test.ts +++ b/js/net/src/lite/publisher.test.ts @@ -15,6 +15,7 @@ import { Front } from "../resume.ts"; import { Reader, Stream, Writer } from "../stream.ts"; import { Timescale, Timestamp } from "../time.ts"; import { Subscriber as TrackSubscriber } from "../track.ts"; +import { Budget } from "../util/yield.ts"; import { wireOf } from "../wire.ts"; import { type AnnounceBroadcast, AnnounceOk, AnnounceRequest, decodeAnnounceBroadcast } from "./announce.ts"; import { Fetch } from "./fetch.ts"; @@ -1515,6 +1516,63 @@ test("lite draft-07: a final drop replaces an expired stream", async () => { } }); +// A queued re-offer that a later update drops again as stale counts as given up: the track's end +// leaves that stale drop standing, rather than replacing it with a final one. Mirrors the Rust +// `a_redropped_reoffer_stays_stale_at_the_end_on_lite07`. +test("lite draft-07: a re-offer dropped again before it opens stays stale at the end", async () => { + const clock = new Manual(); + const installed = clock.install(); + // A write's budget yields on host CPU time, which could run the publisher's post-update task + // before the test writes update 3. + const poll = spyOn(Budget.prototype, "poll").mockReturnValue(undefined); + // Well inside the cache window, so the expired group is still cached to re-offer. + const budget = Milli(1_000); + const update = (update: bigint, maxDelay: Milli) => + new SubscribeUpdate({ + priority: 1, + startGroup: 0n, + update, + maxDelay, + ...restate(Version.DRAFT_07, update - 1n), + }).encode(sub.client.writer, Version.DRAFT_07); + const sub = await servedSubscription({ version: Version.DRAFT_07, startGroup: 0n }); + try { + sub.serve(0n); + expect(await sub.servedSequence()).toBe(0n); + expect("start" in (await decodeSubscribeResponse(sub.client.reader, Version.DRAFT_07))).toBe(true); + await update(1n, budget); + // The restated range resolves at once. + expect("started" in (await decodeSubscribeResponse(sub.client.reader, Version.DRAFT_07))).toBe(true); + + const reset = sub.stall(); + sub.serve(1n); + expect(await sub.servedHeader()).toBe(1n); + await flush(); + await clock.advance(Nano.fromMilli(Milli(budget * 10))); + sub.serve(2n); + expect(await sub.servedSequence()).toBe(2n); + await clock.advance(Nano.fromMilli(budget)); + expect(toStreamCode(fromTransport(await reset))).toBe(StreamCode.Old); + const first = await decodeSubscribeResponse(sub.client.reader, Version.DRAFT_07); + if (!("drop" in first)) throw new Error("expected SUBSCRIBE_DROP"); + expect([first.drop.start, first.drop.end, first.drop.update]).toEqual([1n, 1n, 1n]); + + // Update 2 widens the budget, so group 1 is re-offered. Update 3 restores it within the + // task the publisher yields after an update, before that re-offer opens. + await update(2n, Milli(budget * 100)); + await flush(); + await update(3n, budget); + await flush(); + sub.track.finish(); + sub.track.close(); + expect(await holesToEnd(sub.client.reader)).toEqual([["drop", 1n, 1n, 3n]]); + } finally { + await sub.close(); + installed.close(); + poll.mockRestore(); + } +}); + // A drop the reader gave up as final with OLD, as one relayed from a lite-06 upstream is, goes out as // a stale drop on lite-07, which the subscriber reopens on its next update, so each update drops it // again rather than leave it owed. Mirrors the Rust diff --git a/js/net/src/lite/publisher.ts b/js/net/src/lite/publisher.ts index 636260a9022..e76ae79f119 100644 --- a/js/net/src/lite/publisher.ts +++ b/js/net/src/lite/publisher.ts @@ -1577,6 +1577,8 @@ export class Publisher { insertRun(dropped, start, end); insertRun(droppedStale, start, end); } + // A re-offer an update owed again is given up now, so the end sweep never names it. + if (served) insertRun(served, start, end); const msg = SubscribeDrop.given(start, end - 1n, error, this.version, updateNumber); if (!(await controls.response(encodeSubscribeResponse(stream, { drop: msg }, this.version)))) return false; diff --git a/rs/moq-net/src/lite/publisher.rs b/rs/moq-net/src/lite/publisher.rs index 5feac90760f..77f06b06b1c 100644 --- a/rs/moq-net/src/lite/publisher.rs +++ b/rs/moq-net/src/lite/publisher.rs @@ -5231,6 +5231,10 @@ impl TrackRun { writer.buffer(&self.hole(run.start..=run.end - 1, &hole))?; self.expired.insert(run.clone()); self.sent_stale(run.clone()); + // A re-offer an update owed again is given up now, so the end sweep never names it. + if let Some(served) = &mut self.served { + served.insert(run.clone()); + } } return Poll::Ready(Ok(ControlFlow::Continue(()))); } @@ -8218,6 +8222,47 @@ mod serve_group_test { } } + /// A queued re-offer that a later update drops again as stale counts as given up: the + /// track's end leaves that stale drop standing, rather than replacing it with a final one. + #[kio::test] + async fn a_redropped_reoffer_stays_stale_at_the_end_on_lite07() { + let mut track = track::Producer::new( + Arc::new(broadcast::Info::default()), + "test", + crate::track::Info::timed(), + ); + let budget = Duration::from_millis(500); + let subscriber = + track.subscribe(track::Subscription::default().with_delay(track::Delay::default().with_max(budget))); + let (mut run, mut writer, log) = lite07_run(SinkSession::new(Log::default()), subscriber); + write_group(&mut track, 0, 0); + write_group(&mut track, 1, 1000); + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + write_group(&mut track, 2, 3000); + run.ctx.outcomes.dropped.lock().unwrap().push((0, StreamError::Old)); + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + // Update 1 widens the budget, so group 0 waits its turn; update 2 restores it first. + for (update, max_delay) in [(1, Duration::from_secs(10)), (2, budget)] { + run.update(lite::SubscribeUpdate { + wanted_group: None, + priority: 0, + max_delay, + min_delay: None, + ranges: vec![(0, lite::Range::span(None, 0, None, None))], + order: Order::Descending, + update, + }); + } + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + track.finish_at(track::Position::group(3), None).unwrap(); + while run.poll_step(&mut writer, &kio::Waiter::noop()).is_ready() {} + assert_eq!( + drop_updates(&mut writer, &log, Version::Lite07).await, + [(0, 0, 0), (0, 0, 2)], + "group 0 is dropped again as stale under update 2, and never named again" + ); + } + /// A drop the cursor gave up as final with OLD, as one relayed from a lite-06 upstream is, /// goes out as a stale drop on lite-07, which the subscriber reopens on its next update, so /// each update drops it again rather than leave it owed. From c1ed79c3cced8539ed38bf96c4ccd93d3e0f3747 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 7 Nov 2026 07:00:37 -0800 Subject: [PATCH 68/68] fix(moq-net): async-only helpers stay quiet without the async feature Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/src/ledger.rs | 1 + rs/moq-net/src/model/track.rs | 1 + 2 files changed, 2 insertions(+) diff --git a/rs/moq-net/src/ledger.rs b/rs/moq-net/src/ledger.rs index 80caa9ffa1f..8d8bbe2da34 100644 --- a/rs/moq-net/src/ledger.rs +++ b/rs/moq-net/src/ledger.rs @@ -158,6 +158,7 @@ impl Ledger { /// Take `groups` back out of what is accounted for, owed again from `now`: a verdict that /// accounted for them was reversed. The gap they leave opens now, so it is waited for a /// fresh grace, even where it joins an older gap. + #[cfg_attr(not(feature = "async"), allow(dead_code, reason = "async-only"))] pub fn unaccount(&mut self, groups: Range, now: Instant) { if groups.is_empty() { return; diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index 15741f4cb1a..0ee302943ad 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -22874,6 +22874,7 @@ mod test { } /// Every group a set of ranges names, as a received copy's upstream terms judge them. + #[cfg(feature = "async")] fn judged(runs: impl IntoIterator) -> Option> { Some(runs.into_iter().map(|(start, end)| start..end).collect()) }