Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 7 additions & 6 deletions .claude/skills/verify-subscribe/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
---
name: verify-subscribe
description: Model-check how a lite-07 subscription accounts for each group (`rs/moq-net/src/lite/publisher.rs` `TrackRun` and `GroupServe`, `lite/subscriber.rs` `ServeLoop::apply_response`, `src/tail.rs`) with TLA+. Use when a change touches SUBSCRIBE_DROP, SUBSCRIBE_SKIP, stale (OLD) drops, which groups a publisher serves or names once, Group Stream resets, preemption, and expiry (a lite-07 stream closed when its group expires), a GROUP's Previous Start and the skips it passes over or the publisher holds back for it, or the subscriber's tail, or when a drop bug needs every ordering of the Subscribe Stream against the Group Streams checked.
description: Model-check how a lite-07 subscription accounts for each group (`rs/moq-net/src/lite/publisher.rs` `TrackRun` and `GroupServe`, `lite/subscriber.rs` `ServeLoop::apply_response`, `src/tail.rs`) with TLA+. Use when a change touches SUBSCRIBE_DROP, SUBSCRIBE_SKIP, stale (OLD) drops, which groups a publisher serves or names once, Group Stream resets, preemption, and expiry (a stream reset with OLD when its group expires, even after its FIN), a GROUP's Previous Start and the skips it passes over or the publisher holds back for it, or the subscriber's tail, or when a drop bug needs every ordering of the Subscribe Stream against the Group Streams checked.
---

# Verify subscription accounting

`rs/moq-net/verify/subscribe/SubscribeAccounting.tla` models the groups of one answered lite-07 subscription between the publisher's `TrackRun` and the subscriber's `TrackServe`. The draft promises that every sequence in range gets a Group Stream, a SUBSCRIBE_SKIP, or a SUBSCRIBE_DROP, and that every drop is final on its subscription, a stale (OLD) one included: a SUBSCRIBE_UPDATE reopens nothing, so the model leaves updates out. A skip still goes out over a drop of a group never served. Drops and skips travel in order on the Subscribe Stream, each Group Stream travels on its own, and a reset can lose a stream's header. A Group Stream whose group expires stale is closed, not reset: a FIN without GROUP_END, so the subscriber keeps what arrived as a closed group and no drop names it. A stream that never opened is still given up with a drop, and one whose FIN went out is never reset for expiry. A stream reset while serving, evicted or preempted with TOO_FAR_BEHIND for a group that outranks it, is named in a final drop. Each GROUP's Previous Start passes over the skipped run directly below it as a SUBSCRIBE_SKIP would, so the publisher holds such a skip back for the GROUP of a group it still owes, and sends it after all, ahead of the drop or joined to the skip, if it gives that group up instead. A drop of a group whose stream opened goes out after a skip of what its GROUP passed over, since the GROUP may never have arrived. Each model step is one poll of the code that handles one event. The track cursor's verdicts (hand a group out, give it up stale or final, skip it) are free choices, and so are when a stream fails and whether the cache holds the group above a skip, which decides whether the skip is held back. With `Pressure`, so is when the cache pool's debt folds the publisher's record of a group. TLC checks every ordering under the bounds in the config, and reports a deadlock for any state short of quiet where nothing can happen.
`rs/moq-net/verify/subscribe/SubscribeAccounting.tla` models the groups of one answered lite-07 subscription between the publisher's `TrackRun` and the subscriber's `TrackServe`. The draft promises that every sequence in range gets a Group Stream, a SUBSCRIBE_SKIP, or a SUBSCRIBE_DROP, and that every drop is final on its subscription, a stale (OLD) one included: a SUBSCRIBE_UPDATE reopens nothing, so the model leaves updates out. A skip still goes out over a drop of a group never served. Drops and skips travel in order on the Subscribe Stream, each Group Stream travels on its own, and a reset can lose a stream's header. A Group Stream whose group expires stale resets with OLD and is named in a stale drop, whether it never opened, is serving, or already sent its FIN and awaits the acknowledgement, so stale bytes never hold the send window; a subscriber that read it whole before the reset keeps the group. A stream reset while serving, evicted or preempted with TOO_FAR_BEHIND for a group that outranks it, is named in a final drop. Each GROUP's Previous Start passes over the skipped run directly below it as a SUBSCRIBE_SKIP would, so the publisher holds such a skip back for the GROUP of a group it still owes, and sends it after all, ahead of the drop or joined to the skip, if it gives that group up instead. A drop of a group whose stream opened goes out after a skip of what its GROUP passed over, since the GROUP may never have arrived. Each model step is one poll of the code that handles one event. The track cursor's verdicts (hand a group out, give it up stale or final, skip it) are free choices, and so are when a stream fails and whether the cache holds the group above a skip, which decides whether the skip is held back. With `Pressure`, so is when the cache pool's debt folds the publisher's record of a group. TLC checks every ordering under the bounds in the config, and reports a deadlock for any state short of quiet where nothing can happen.

## Rules

Expand All @@ -14,7 +14,7 @@ Each rule is an INVARIANT; the harm says why it matters.
| Rule | Holds when | Harm if broken |
| --- | --- | --- |
| `ServedOnce` | The publisher opens at most one Group Stream for a group. | The group's bytes go out twice, and the subscriber reads two streams of one group. |
| `NamedOnce` | The publisher names a group in at most one drop and at most one skip, and a skip only for a group it never served. The one exception repeats a skip a GROUP passed over, once, ahead of that group's drop. | A second name contradicts a final one: the subscriber accounts the group twice or ends a reader on a group it delivered. |
| `NamedOnce` | The publisher names a group in at most one drop and at most one skip, and a skip only for a group it never served. A drop may follow the group's own Group Stream, even one sent whole, once it resets. The one exception to one skip repeats a skip a GROUP passed over, once, ahead of that group's drop. | A second name contradicts a final one: the subscriber accounts the group twice or ends a reader on a group it delivered. |
| `QuietIsSettled` | Once nothing is in flight and the publisher has nothing left to do, the subscriber's tail accounts for every group, a skip held back for a GROUP included. | The subscriber waits out its grace for a group nobody will send, and a relay holds its readers there. |

## Run
Expand All @@ -23,7 +23,7 @@ Each rule is an INVARIANT; the harm says why it matters.
quest gate just rs tla moq-net/verify/subscribe/SubscribeAccounting.tla
```

It checks every rule under `Pressure` over three groups, 1 to 3, so a skip of 1 can be held back for 2's GROUP and a run of 1 and 2 for 3's: about 1.97M distinct states, ten seconds. It passes. Nightly runs every config here and fails if a config with an `\* Open finding:` line passes.
It checks every rule under `Pressure` over three groups, 1 to 3, so a skip of 1 can be held back for 2's GROUP and a run of 1 and 2 for 3's: about 11.1M distinct states, two minutes. It passes. Nightly runs every config here and fails if a config with an `\* Open finding:` line passes.

## Reading a failure

Expand All @@ -40,7 +40,7 @@ Sync commit: `3016af623807e939c2e99571a3ec02beb2493579`. Run `git log 3016af6238
| `rs/moq-net/src/lite/publisher.rs` `poll_forward_drop`, `forward_range`, `drop_range`, `report`, `hole`; `rs/moq-net/src/model/record.rs` `Record::report`, `tell`, `untold`, `Report::blocks` | `CursorGiveUp`, `dropped` |
| `rs/moq-net/src/lite/publisher.rs` `TrackRun::elides`, `owes_group`, `send_elided` | `held`, the `hold` choice of `CursorGiveUp` |
| `rs/moq-net/src/lite/publisher.rs` `TrackRun::poll_step` (outcomes) | `PubOutcomes`, `nahead` |
| `rs/moq-net/src/lite/publisher.rs` `GroupServe::poll_serve` (`Serve`, `Closed`), `settle_failed`, `poll_stopped`, `closes_expired`, `Ending`; `rs/moq-net/src/transport/boxed.rs` `ListedState::preempted` | `StreamFin`, `StreamFail`, `StreamAck` |
| `rs/moq-net/src/lite/publisher.rs` `GroupServe::poll_serve` (`Serve`, `Closed`), `settle_failed`, `poll_given_up`; `rs/moq-net/src/transport/boxed.rs` `ListedState::preempted` | `StreamFin`, `StreamFail`, `StreamAck` |
| `rs/moq-net/src/model/record.rs` `Record::fold` | `PubFold`, `folded` |
| `rs/moq-net/src/lite/subscriber.rs` `TrackEntry::open` (with the passed-over skip), `reopen`, `place`; `GroupRecv::poll_serve` | `SubHeader`, `SubFin`, `SubReset` |
| `rs/moq-net/src/lite/subscriber.rs` `ServeLoop::apply_response` (`Drop`, `Skip`) | `OnDrop`, `OnSkip`, `SubRecv` |
Expand All @@ -52,7 +52,8 @@ What the model asks of that code, each found by TLC as a broken rule without it:
- The publisher serves no sequence its record already told, served or named (`Record::told`), and a skip goes out over a drop alone (`Report::blocks`), so a cursor that hands a group out again after giving it up, as after a route switch, sends nothing (`ServedOnce`, `NamedOnce`).
- A skip held back for a group's GROUP goes out once that group is given up instead: ahead of its drop (`drop_range`, `forward_range`), and joined to its skip, since no GROUP passes over it then (`forward_range`). A skip of the group above that kept the run held would leave it unsent for good (`QuietIsSettled`).
- A drop of a group whose stream opened goes out after a skip of what its GROUP passed over (`GroupOutcomes::dropped`), since a reset can lose the GROUP (`QuietIsSettled`).
- A stream reset for expiry after its FIN is still named in a drop (`settle_failed` from `GroupState::Closed`), since the reset can discard the header before it is read (`QuietIsSettled`).

## Not covered

The subscription starts answered and covers the groups, so SUBSCRIBE_OK, SUBSCRIBE_STARTED, ranges and Range IDs, fills (`RangeFills`) and the separate request (`Recovery`) a reader wider than a stale drop asks with, floors, SUBSCRIBE_UPDATE, and SUBSCRIBE_FINISH with `drop_unserved` are left out. The tail's grace and the ledger's folding are left out: `QuietIsSettled` asks for the publisher's answer, never the grace. So are priority and order (`Steer`), which stream a preempt picks, the Subscribe Stream's FIN and cancellation, sessions closing, versions before lite-07, relays forwarding drops downstream, the budget a relay records a stale drop under, datagrams, and the copy's readers (`poll_drop`). A route switch appears only as the cursor handing out once more a group it gave up. FETCH is left out: one request and one response, ended by a FIN or a reset, with nothing to race. GROUP_END takeover is left out, and so are frames: how much of a group an expired stream closes on, and GROUP_END's End, which only says where a whole group ends. With one Group Stream per group, the subscriber's wait for a duplicate stream (`Open::Duplicate`) is never reached. A GROUP's Previous Start is modeled as the skipped run directly below it; one naming a known group below, or nothing, passes over nothing, and a run released by an update (`release_elided`) is left out with updates. A pass covers three groups.
The subscription starts answered and covers the groups, so SUBSCRIBE_OK, SUBSCRIBE_STARTED, ranges and Range IDs, fills (`RangeFills`) and the separate request (`Recovery`) a reader wider than a stale drop asks with, floors, SUBSCRIBE_UPDATE, and SUBSCRIBE_FINISH with `drop_unserved` are left out. The tail's grace and the ledger's folding are left out: `QuietIsSettled` asks for the publisher's answer, never the grace. So are priority and order (`Steer`), which stream a preempt picks, the Subscribe Stream's FIN and cancellation, sessions closing, versions before lite-07, relays forwarding drops downstream, the budget a relay records a stale drop under, datagrams, and the copy's readers (`poll_drop`). A route switch appears only as the cursor handing out once more a group it gave up. FETCH is left out: one request and one response, ended by a FIN or a reset, with nothing to race. GROUP_END takeover is left out, and so are frames and GROUP_END's End, which only says where a whole group ends. With one Group Stream per group, the subscriber's wait for a duplicate stream (`Open::Duplicate`) is never reached. A GROUP's Previous Start is modeled as the skipped run directly below it; one naming a known group below, or nothing, passes over nothing, and a run released by an update (`release_elided`) is left out with updates. A pass covers three groups.
12 changes: 7 additions & 5 deletions drafts/draft-lcurley-moq-lite.md
Original file line number Diff line number Diff line change
Expand Up @@ -222,7 +222,7 @@ A Group can therefore be assembled from more than one publisher: each contribute

A partial Group is only ever delivered to a subscriber that asked for one with FETCH.
A publisher that holds only the later frames of a Group a subscription covers names it in a [SUBSCRIBE_DROP](#subscribe-drop) rather than deliver the part it holds.
One that holds its earlier frames, such as a relay whose upstream closed the Group or a Group expiring as it is sent, delivers them and closes the Group Stream without a GROUP_END.
One that holds its earlier frames, such as a relay whose upstream closed the Group, delivers them and closes the Group Stream without a GROUP_END.
A Group is the unit of decodability: its earlier Frames may carry a keyframe or compression state the later Frames need.

# Flow
Expand Down Expand Up @@ -838,8 +838,9 @@ A Max Age of zero follows from the same rule without a special case.
Timestamp age means a burst of groups delivered together still reads as its true age, so catching up never resets the clock.
Arrival age bounds what timestamps cannot: a publisher whose timestamps stop advancing, a successor that never presents a frame, and a run of empty groups.

A publisher stops sending an expired group: it closes the group's open Group Stream without a GROUP_END, and names a group it opened no stream for in a [SUBSCRIBE_DROP](#subscribe-drop) with OLD.
The subscriber keeps the frames that arrived, and a GROUP_END never claims an expired group whole.
A publisher stops sending an expired group: it resets the group's Group Stream with OLD and names the group in a [SUBSCRIBE_DROP](#subscribe-drop) with OLD, as it does a group it opened no stream for.
This includes a stream whose GROUP_END was sent but not yet acknowledged, whose bytes would otherwise hold flow control the streams behind it need.
A subscriber that already read such a stream whole, through its GROUP_END and FIN, keeps the group: the OLD drop changes nothing for its readers.

## Unidirectional Streams
Unidirectional streams are used for data transmission, the session's setup, and discovery and routing.
Expand Down Expand Up @@ -1996,6 +1997,7 @@ See the [Prioritization](#prioritization) section for more information.
**Subscriber Max Age**:
The optional maximum age in milliseconds that the subscriber accepts, encoded as the value plus one; 0 omits the reader bound.
The publisher still applies `Publisher Max Age`, and MUST use the tighter bound while serving and while the returned group remains open.
A response whose group expires under that bound before the subscriber acknowledges it is reset with OLD, even after its GROUP_END, as a Group Stream is (see [Expiration](#expiration)).

**Group Start**:
The start of the group to fetch.
Expand Down Expand Up @@ -2351,10 +2353,10 @@ A stream that FINs without a GROUP_END carries a closed Group, which may have he
- Replaced GROUP's `Group Sequence` with `Group Start`, added `Previous Start`, the start of the Group before it, so a subscriber sees the gap before a Group without a SUBSCRIBE_SKIP, and removed `Frame Start`: a subscription reads whole Groups, so every Group Stream starts at frame 0.
- Added GROUP_END, a `Payload Length` of 0 followed by the Group's exclusive end: a Group or Fetch Stream ends with it when its Group finished, a FIN alone ends a closed Group whose frames may not all exist, and a reset aborts. A Group's own end bounds its reach.
- Expiration also measures a group's arrival age, the time since its successor arrived at this hop.
- A publisher closes the open Group Stream of a group that expires, without a GROUP_END, rather than resetting it, and a relay closes a Group it holds only the start of.
- A relay closes a Group it holds only the start of, without a GROUP_END. A publisher resets the Group Stream of a group that expires with OLD, even one whose GROUP_END it sent but is not yet acknowledged, and names the group in a SUBSCRIBE_DROP with OLD.
- FRAME leads with `Payload Length`, the payload size plus one, in place of the trailing `Message Length`; 0 is malformed.
- A publisher SHOULD drop a datagram rather than block when the transport has no room for it, where it previously sent one only if the congestion controller could transmit it immediately.
- Added an optional `Subscriber Max Age` to FETCH, encoded as milliseconds plus one with zero meaning absent. It can tighten, but never relax, `Publisher Max Age`.
- Added an optional `Subscriber Max Age` to FETCH, encoded as milliseconds plus one with zero meaning absent. It can tighten, but never relax, `Publisher Max Age`. A response whose group expires before it is acknowledged is reset with OLD.
- A receiver drops a claim whose resolved path it cannot hold, such as one past the path limits once joined beneath its own root, rather than closing the session ([Announce](#announce)).

## moq-lite-06
Expand Down
10 changes: 9 additions & 1 deletion js/net/src/broadcast.ts
Original file line number Diff line number Diff line change
Expand Up @@ -341,7 +341,9 @@ async function fetchGroup(
// now: an in-progress group must keep receiving frames for its lifetime
// (mirrors Rust poll_fetch), and a failed group's unread frames stay in the
// cache's reach, expirable, until the caller lets them go.
void hooks.groupDone(recv.group).then(() => subscriber.close());
void (
(options as FetchFrom).held ? hooks.groupReleased(recv.group) : hooks.groupDone(recv.group)
).then(() => subscriber.close());
return recv.group;
case "drop":
// Its own reason, so a stale verdict stays OLD: a relay's fill asking under this
Expand All @@ -368,6 +370,12 @@ async function fetchGroup(
*/
export interface FetchFrom {
frameStart?: number;
/**
* Keep the fetch's subscription, which judges the group's budget, until the reader releases
* the group rather than once it reads the end: a publisher still owed an acknowledgement
* resets a response that goes stale meanwhile.
*/
held?: boolean;
}

// Settles once a read of `subscriber` may go differently.
Expand Down
9 changes: 8 additions & 1 deletion js/net/src/group.ts
Original file line number Diff line number Diff line change
Expand Up @@ -995,6 +995,8 @@ export class Consumer {
#expired?: Expired;
// Closed by its reader, which ends it whatever is left unread.
#released = false;
// Settles once this reader is released, made only when something waits on it.
#releasing?: PromiseWithResolvers<void>;
// The arrival of the copy first read, kept once a replacement copy is adopted.
#arrived?: Instant;
// The last frame handed out to stream before it completed. Cut short, it is lost, so no
Expand Down Expand Up @@ -1107,6 +1109,11 @@ export class Consumer {
makeConsumer = (state) => new Consumer(state);
hooks.releaseGroupReader = (group, abort) => group.#release(abort);
hooks.groupDone = (group) => group.#done();
hooks.groupReleased = (group) => {
if (group.#released) return Promise.resolve();
group.#releasing ??= Promise.withResolvers<void>();
return group.#releasing.promise;
};
hooks.readsMirrored = (group) => group.#state === group.#mirrored;
hooks.groupReads = (group, reads) => group.#mirrored.reads?.(reads);
hooks.groupTimestamp = (group) =>
Expand Down Expand Up @@ -1134,7 +1141,6 @@ export class Consumer {
group.#expiry = expiry;
};
hooks.groupFailed = (group) => group.#failed();
hooks.groupExpired = (group) => group.#expiry?.expired() === true;
hooks.recoverGroup = (group, recover) => {
group.#recover = recover;
// A front serving another front attaches its recovery after the inner one handed the
Expand Down Expand Up @@ -1562,6 +1568,7 @@ export class Consumer {
this.#recover = undefined;
// Closing is terminal for this reader, even with frames still unread.
this.#released = true;
this.#releasing?.resolve();
if (this.#state.closed.peek() === undefined) this.#state.closed.set(abort ?? null);
this.#refresh();
}
Expand Down
Loading