Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions .claude/skills/verify-resume/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ description: Model-check how a front's reader resumes one logical track across r

# Verify track resume

`rs/moq-net/verify/resume/ResumeReader.tla` models one `Reader` of a logical track that `resume::Producer` serves across route changes. Each route is a received track whose groups land in any order. The front serves one route at a time, switches to another live route at will, and ends the track cleanly once the serving route completes, or unroutable once no route is left. One reader poll (`poll_lowest`, through `sync` and `backfill_gaps`) is one atomic step, as the reader's lock makes it; so is a switch, which the reader follows at once as `serve_from` does. TLC checks every ordering of arrivals, route failures, switches, polls, and finished reads under the bounds in each config.
`rs/moq-net/verify/resume/ResumeReader.tla` models one `Reader` of a logical track that `resume::Producer` serves across route changes. Each route is a received track whose groups land in any order, even below an end its session already settled. The front serves one route at a time, switches to another live route at will, and ends the track cleanly once the serving route completes, or unroutable once no route is left. One reader poll (`poll_lowest`, through `sync` and `backfill_gaps`) is one atomic step, as the reader's lock makes it; so is a switch, which the reader follows at once as `serve_from` does. TLC checks every ordering of arrivals, route failures, switches, polls, and finished reads under the bounds in each config.

## Rules

Expand All @@ -14,7 +14,7 @@ Each rule is an INVARIANT; the harm says why it matters.
| Rule | Holds when | Harm if broken |
| --- | --- | --- |
| `OnceEach` | A group is handed out at most once, whichever copies deliver it. | A viewer plays a group twice after a failover. |
| `CleanEndIsComplete` | A clean end comes with every group the serving route held once complete, and every group all live routes hold. | The track ends cleanly with a hole a route could still fill; the viewer loses media silently. |
| `CleanEndIsComplete` | A clean end comes with every group the serving route held once complete, every group all live routes hold, and every group all live routes held as it ended, late ones included. | The track ends cleanly with a hole a route could still fill, or one a group in flight already filled; the viewer loses media silently. |
| `EndWaitsForReplaced` | The serving copy draining ends the track only once no copy it replaced may still deliver a group the reader lacks below that end. | A group still in flight on the old route is lost at the end. |
| `CleanEndIsDeclared` | A clean end only follows a route declaring one. | A failed route reads as the broadcast ending. |
| `FailsOnlyUnroutable` | The track fails only once the front gave up: a route failing is the front's to replace. | A single relay failure ends playback that another route could carry. |
Expand All @@ -28,7 +28,7 @@ quest gate just rs tla moq-net/verify/resume/ResumeReader.tla
quest gate just rs tla moq-net/verify/resume/ResumeReader.tla -config rs/moq-net/verify/resume/ThreeRoutes.cfg
```

The first checks every rule over two routes, three groups, two switches, and a reader holding each group until it finishes it: about 14.3M distinct states, in minutes. The second trades a group for a third route, so a route can come up after another failed: about 7.1M states, in minutes. Both pass at the sync commit. Nightly runs every config here and fails if a config with an `\* Open finding:` line passes.
The first checks every rule over two routes, three groups, two switches, and a reader holding each group until it finishes it: about 19.9M distinct states, in about five minutes. The second trades a group for a third route, so a route can come up after another failed: about 7.4M states, in about a minute. Both pass at the sync commit. Nightly runs every config here and fails if a config with an `\* Open finding:` line passes.

## Open findings

Expand All @@ -40,17 +40,17 @@ Follow Reading a failure in `rs/moq-net/verify/README.md`. Replay the path as a

## Coverage

Sync commit: `74e05ddc7039cc32f8968d9425e8224e82d4d886`. Run `git log 74e05ddc70.. -- rs/moq-net/src/model/resume.rs rs/moq-net/src/model/track.rs` to see what changed since. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function.
Sync commit: `8c939bc9d05e5e882fb4e042a9efbf2db227b721`, plus [moq-dev/moq#9103](https://github.com/moq-dev/moq/pull/9103), which reads a group landing late on a drained copy. Run `git log 8c939bc9d0.. -- rs/moq-net/src/model/resume.rs rs/moq-net/src/model/track.rs` to see what changed since. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function.

| Code | Model |
| --- | --- |
| `rs/moq-net/src/model/resume.rs` `Reader::sync` (reuse of a route's copy, new copy at `newest`) | `Sync`, `Serve` |
| `rs/moq-net/src/model/resume.rs` `Reader::backfill`, `backfill_gaps` | `Backfill`, `BackfillGaps` |
| `rs/moq-net/src/model/resume.rs` `Reader::poll_lowest`, `deliverable`, `take`, `observe` | `Poll`, `Scan`, `Peek`, `Deliverable`, `Take` |
| `rs/moq-net/src/model/resume.rs` `Reader::poll_lowest`, `deliverable`, `take`, `observe`, `Copy::done` | `Poll`, `Pass`, `Scan`, `Peek`, `Deliverable`, `Take` |
| `rs/moq-net/src/model/resume.rs` `Reader::bound_at_end`, `let_go`, `Copy::retains` | `BoundAtEnd`, `LetGo`, `Retains` |
| `rs/moq-net/src/model/resume.rs` `Reader::poll_end` | `PollEnd` |
| `rs/moq-net/src/model/resume.rs` `Producer::splice`, `serve_from`, `Producer::end`, `Phase::Finished` | `Serve`, `FrontFinish`, `FrontFail` |
| `rs/moq-net/src/model/track.rs` `TrackState::is_complete`, `may_land`, `max_key`, `Consumer::latest_key`, `track::Subscriber::has_unread_group` | `Seal`, `Held`, `MayLand`, `Latest`, `HasUnread` |
| `rs/moq-net/src/model/track.rs` `TrackState::is_complete`, `may_land`, `max_key`, `Consumer::latest_key`, `Consumer::poll_arrived`, `track::Subscriber::has_unread_group`, `Producer::finish_shared` | `Seal`, `Held`, `MayLand`, `Latest`, `Landed`, `HasUnread`, `Settle` |

## Not covered

Expand Down
5 changes: 5 additions & 0 deletions js/net/src/internal.ts
Original file line number Diff line number Diff line change
Expand Up @@ -780,6 +780,8 @@ export const hooks: {
tailGrace: (producer: Producer) => Duration;
/** Snapshot the latest group key for wire positioning; assigned by `track.ts`. */
latestKey: (subscriber: Subscriber) => bigint | undefined;
/** How many groups arrived on the subscriber's copy so far; assigned by `track.ts`. */
arrived: (subscriber: Subscriber) => number;
/**
* The source's declared start, raised to the subscriber's cursor floor, or `"pending"` while
* the upstream START that resolves it has yet to arrive; assigned by `track.ts`.
Expand Down Expand Up @@ -1182,6 +1184,9 @@ export const hooks: {
latestKey: () => {
throw new Error("track.ts not loaded");
},
arrived: () => {
throw new Error("track.ts not loaded");
},
sourceStart: () => {
throw new Error("track.ts not loaded");
},
Expand Down
29 changes: 29 additions & 0 deletions js/net/src/resume.model.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1079,6 +1079,35 @@ test("regressions", async () => {
],
finish: false,
},
// The serving route ends while a replaced copy holds the read open for a group under its
// bound: groups in range still in flight on the ended route land, and are read. Mirrors
// the Rust case.
{
ops: [
{ kind: "join", reader: 1, spec: spec(false, "hour", undefined, false) },
{ kind: "up", instance: 0, slot: 1 },
{ kind: "join", reader: 0, spec: spec(false, "hour", [6, 8], false) },
{ kind: "write", instance: 0, slot: 0, group: undefined, frames: 1 },
{ kind: "write", instance: 0, slot: 1, group: 11, frames: 1 },
{ kind: "write", instance: 0, slot: 1, group: 2, frames: 1 },
{ kind: "end", instance: 0, slot: 1 },
],
finish: false,
},
// The same for a reader without a range, once the serving copy's first group bounds the
// replaced one below the groups still in flight.
{
ops: [
{ kind: "join", reader: 1, spec: spec(false, "hour", undefined, false) },
{ kind: "up", instance: 0, slot: 1 },
{ kind: "join", reader: 0, spec: spec(false, "hour", undefined, false) },
{ kind: "write", instance: 0, slot: 0, group: undefined, frames: 1 },
{ kind: "write", instance: 0, slot: 1, group: 2, frames: 1 },
{ kind: "write", instance: 0, slot: 1, group: 11, frames: 1 },
{ kind: "end", instance: 0, slot: 1 },
],
finish: false,
},
];
const failures: string[] = [];
for (const [i, c] of regressions.entries()) {
Expand Down
75 changes: 75 additions & 0 deletions js/net/src/resume.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -420,6 +420,81 @@ test("a group below the replacement's resume point is backfilled", async () => {
}
});

test("a late group on a drained serving copy is read", async () => {
const clock = new Manual();
const installed = clock.install();
const front = new Front();
const sources = [new BroadcastProducer(), new BroadcastProducer()];
const handles = sources.map((source) => source.consume());
const [ta, tb] = sources.map(copy);
hooks.receiveTrack(ta.producer);
hooks.receiveTrack(tb.producer);
front.serve(handles[0]);
const sub = front.consumer.track("seq").subscribe({ delay: { max: Duration.millis(3_600_000) } });
try {
await clock.advance(Duration.ZERO);
ta.write(0n);
expect(drain(sub)).toEqual([0n]);
front.serve(handles[1]);
await clock.advance(Duration.ZERO);
// Its first group bounds the replaced copy at 1, which it may still deliver.
tb.write(2n);
await clock.advance(Duration.ZERO);
expect(drain(sub)).toEqual([2n]);
tb.producer.finish();
const next = sub.recvGroup();
expect(await isPending(next)).toBe(true);
tb.write(1n);
await clock.advance(Duration.ZERO);
expect((await next)?.start).toBe(1n);
} finally {
sub.close();
front.consumer.close();
for (const handle of handles) handle.close();
for (const source of sources) source.close();
installed.close();
}
});

test("a late group under the resume point is backfilled again", async () => {
const clock = new Manual();
const installed = clock.install();
const front = new Front();
const sources = [new BroadcastProducer(), new BroadcastProducer()];
const handles = sources.map((source) => source.consume());
const [ta, tb] = sources.map(copy);
hooks.receiveTrack(ta.producer);
hooks.receiveTrack(tb.producer);
front.serve(handles[0]);
const sub = front.consumer.track("seq").subscribe({ delay: { max: Duration.millis(3_600_000) } });
try {
await clock.advance(Duration.ZERO);
ta.write(2n);
ta.producer.finish();
expect(sub.tryRecvGroup()?.start).toBe(2n);
// A switch away and back resumes `ta` past 2; the backfill below it finds its track
// complete and runs out, while `tb`'s copies hold the read open.
front.serve(handles[1]);
await clock.advance(Duration.ZERO);
expect(drain(sub)).toEqual([]);
front.serve(handles[0]);
await clock.advance(Duration.ZERO);
const next = sub.recvGroup();
expect(await isPending(next)).toBe(true);
// `tb` fails, and 1 lands late on `ta` before the read looks again.
tb.producer.abort(new StreamError(StreamCode.SessionClosed));
ta.write(1n);
await clock.advance(Duration.ZERO);
expect((await next)?.start).toBe(1n);
} finally {
sub.close();
front.consumer.close();
for (const handle of handles) handle.close();
for (const source of sources) source.close();
installed.close();
}
});

// The switch's backfill waits on a fill the finished new route's handler still owes, rather
// than counting as drained while the fill is out.
test("a backfill waits on a fill the finished route still owes", async () => {
Expand Down
31 changes: 25 additions & 6 deletions js/net/src/resume.ts
Original file line number Diff line number Diff line change
Expand Up @@ -350,8 +350,12 @@ interface Copy {
* groups the newer one may never have.
*/
until?: bigint | null;
/** The group channel ran out: cleanly (`null`) or failed. */
done?: Error | null;
/**
* The group channel ran out: failed, or cleanly with how many groups had arrived on the copy
* by then, since one arriving later is still in flight below its end, which a read still open
* takes.
*/
done?: Error | number;
/**
* When a newer group superseded every group under the bound: the newer copy's first group, or
* the track's end. What the reader still lacks from it is waited for from here, for no longer
Expand Down Expand Up @@ -824,6 +828,7 @@ class Reader implements ResumeSource {
this.#learnEnd();
if (this.#closed) return { kind: "done" };
this.#groupsRead = true;
this.#reopen();
this.#backfillGaps(delivered);
if (dispositions) {
const dropped = this.#tryDrop(delivered);
Expand Down Expand Up @@ -860,10 +865,11 @@ class Reader implements ResumeSource {
// Drained: the track ended as the copy did, which is not cleanly when its
// session went away short of the declared end.
if (recv.kind === "done") {
copy.done = copy.sub.closed.peek() ?? null;
if (copy.done === null && hooks.subscriberEnd(copy.sub)) this.#settled = true;
const closed = copy.sub.closed.peek();
copy.done = closed instanceof Error ? closed : hooks.arrived(copy.sub);
if (!(closed instanceof Error) && hooks.subscriberEnd(copy.sub)) this.#settled = true;
} else if (recv.kind === "error") copy.done = recv.error;
else if (this.#drained(copy, delivered)) copy.done = null;
else if (this.#drained(copy, delivered)) copy.done = hooks.arrived(copy.sub);
break;
}
}
Expand All @@ -873,7 +879,7 @@ class Reader implements ResumeSource {
const dropped = this.#tryDrop(delivered);
if (dropped) return dropped;
}
return this.#end((copy) => copy.done);
return this.#end((copy) => (typeof copy.done === "number" ? null : copy.done));
}

/**
Expand Down Expand Up @@ -981,6 +987,19 @@ class Reader implements ResumeSource {
return hooks.spent(copy.sub, start, end, delivered);
}

// A copy that ran out cleanly still takes a group in flight below its end while the read stays
// open, as a cursor still reading it would. One landing on the serving copy may lie under its
// resume point, which is not that copy's to read, and a backfill that already ran out was let
// go: ask its route again. Mirrors the Rust `Reader::poll_lowest`.
#reopen(): void {
const serving = this.#serving();
for (const copy of this.#copies) {
if (typeof copy.done !== "number" || hooks.arrived(copy.sub) <= copy.done) continue;
copy.done = undefined;
if (copy === serving && !copy.replaced && copy.floor !== undefined) this.#resumed = true;
}
}

// Whether `copy` may still feed groups: its channel has not run out, and what it holds is
// within the reader's budget, which ends a full budget after a newer group superseded it.
#feeds(copy: Copy): boolean {
Expand Down
5 changes: 5 additions & 0 deletions js/net/src/track.ts
Original file line number Diff line number Diff line change
Expand Up @@ -816,6 +816,9 @@ class TrackState {
*/
reads?: (key: bigint) => boolean;
groups = new Signal<GroupConsumer[]>([]);
// How many groups were mirrored in, whatever this sink reads: one arriving after its reader
// ran out is still in flight below the end.
arrived = 0;
// Keys the upstream publisher gave up, with why: an incoming SUBSCRIBE_SKIP or
// SUBSCRIBE_DROP, or a producer cut. Kept apart from the cache so reclaim cannot
// erase a disposition a downstream publisher still has to forward, and as runs so a wide
Expand Down Expand Up @@ -2073,6 +2076,7 @@ export class Producer {
entry.mirrors.set(sink, dst);
if (sink.reads?.(dst.start) === false) hooks.groupReads(dst, false);
timelineInsert(sink.timeline, dst);
sink.arrived++;
void dst.readable().then(() => sink.timelineChanged.update((revision) => revision + 1));
sink.latest = max(sink.latest ?? 0n, dst.start);
sink.received.set(max(sink.received.peek(), dst.start + 1n));
Expand Down Expand Up @@ -4362,6 +4366,7 @@ export class Subscriber {
return stop;
};
hooks.latestKey = (subscriber) => (subscriber.#resume ? subscriber.#resume.latest() : subscriber.#state.latest);
hooks.arrived = (subscriber) => (subscriber.#resume ? 0 : subscriber.#state.arrived);
hooks.bufferedLargest = (subscriber) => subscriber.#bufferedLargest();
hooks.objectLargest = (subscriber) => {
if (subscriber.#resume) return subscriber.#resume.objectLargest();
Expand Down
36 changes: 36 additions & 0 deletions quest/m0/nightly/resume-window-mutants.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# [M] No mutant of the resume reader's window survives its tests

## Goal

Nightly's mutants job never flags `rs/moq-net/src/model/resume.rs`
`Reader::window`: each of its two surviving mutants is killed by a test, or
shown equivalent and excluded in `.cargo/mutants.toml` with the reason.

## Plan

[moq-dev/moq#9019](https://github.com/moq-dev/moq/pull/9019) left both
unexcluded, expecting the resume model's seed-3 failure to be range logic in
`Reader::window`. It was not: [moq-dev/moq#9103](https://github.com/moq-dev/moq/pull/9103)
found a cleanly drained copy dropping groups that landed late, and its fix
leaves `Reader::window` as it was. The mutants:

- `> → >=` in the capped-copy filter (`resolved.end.is_none_or(|end| end > resolved.start)`):
differs only when a capped copy resolves a range to an empty one ending at
its start, which the original treats as saying nothing and the mutant
places as an empty window.
- `< → >` on the cap on the end (`cap.is_none_or(|cap| *end < cap)`): a capped
copy's own wire ends at its cap, so its resolved end never passes it, and the
mutant drops every end the original keeps. The window then waits for another
copy to place its end. This shows only when a ranges change leaves the
window's time end unsettled on the serving copy while a replaced, capped copy
already holds a group past it. A bounded read then ends at its range's end
on the original, and stays open on the mutant until the track ends.
`resume_model.rs` accepts both, an open read and one ended at its range,
which is why 60k sweep cases never killed it: a unit test in `resume.rs`'s
`mod test` that asserts the read ends is the likely kill.

Public API: none. Wire: none.

## Related

- [Mutants of the other modelled code](/quest/m4/mutants-modelled.md) - the same Nightly job, widened to more files
37 changes: 37 additions & 0 deletions quest/m1/precut-fixes/resume-js-window.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# [M] A JS resumed reader hands out no group outside its window

## Goal

A `@moq/net` reader that resumes a track across routes (`js/net/src/resume.ts`)
never hands out a group outside the window its ranges resolved to, as the Rust
reader never does. `MOQ_RESUME_SEED=3 MOQ_RESUME_CASES=50000` and seeds 1 and 7
pass on `js/net/src/resume.model.test.ts`, so Nightly's `test/resume-model.sh`
passes on its JS side.

## Plan

Found by [moq-dev/moq#9103](https://github.com/moq-dev/moq/pull/9103) while
sweeping its fix: on `main` at [4f3ba3317f](https://github.com/moq-dev/moq/commit/4f3ba3317f),
the JS model fails fast with `group 0 outside ranges` (seed 3 at 89 cases,
seed 1 at 434, seed 7 at 196), with or without that PR.

```sh
MOQ_RESUME_SEED=3 MOQ_RESUME_CASES=50000 bun test ./js/net/src/resume.model.test.ts
```

Counterexample: join route 0 with range `[1, 3]`; route 0/1 comes up; write
group 1 (two frames), then group 0, then the next group on 0/0. The serving
copy resolves the time anchor against its own cache, which holds only group 0,
and hands group 0 out, though the reader's window starts at 1.

Introduced by [moq-dev/moq#9027](https://github.com/moq-dev/moq/pull/9027)
(`0c14c87c49`, "@moq/net addresses groups by start"): the same seeds pass on
its parent. Rust's `Reader::poll_lowest` filters each peeked group with
`self.window().contains(key)` ("the window decides"), leaving an ordered
cursor in place when `owes_below` still owes a lower group. JS `#lowest` and
the serving loop in `tryGroup` check only `#deliverable` (the copy's cap and
`delivered`). Mirror that filter, with the window built from `#placed` /
`resolved()`, and add the counterexample to `resume.model.test.ts`'s
regressions.

Public API: none. Wire: none.
Loading