diff --git a/.claude/skills/verify-resume/SKILL.md b/.claude/skills/verify-resume/SKILL.md index ebe336bc4f..c1ca9f6b80 100644 --- a/.claude/skills/verify-resume/SKILL.md +++ b/.claude/skills/verify-resume/SKILL.md @@ -5,7 +5,7 @@ description: Model-check how a front's reader resumes one logical track across r # Verify track resume -`rs/moq-net/verify/resume/ResumeReader.tla` models one `Reader` of a logical track that `resume::Producer` serves across route changes. Each route is a received track whose groups land in any order. The front serves one route at a time, switches to another live route at will, and ends the track cleanly once the serving route completes, or unroutable once no route is left. One reader poll (`poll_lowest`, through `sync` and `backfill_gaps`) is one atomic step, as the reader's lock makes it; so is a switch, which the reader follows at once as `serve_from` does. TLC checks every ordering of arrivals, route failures, switches, polls, and finished reads under the bounds in each config. +`rs/moq-net/verify/resume/ResumeReader.tla` models one `Reader` of a logical track that `resume::Producer` serves across route changes. Each route is a received track whose groups land in any order, even below an end its session already settled. The front serves one route at a time, switches to another live route at will, and ends the track cleanly once the serving route completes, or unroutable once no route is left. One reader poll (`poll_lowest`, through `sync` and `backfill_gaps`) is one atomic step, as the reader's lock makes it; so is a switch, which the reader follows at once as `serve_from` does. TLC checks every ordering of arrivals, route failures, switches, polls, and finished reads under the bounds in each config. ## Rules @@ -14,7 +14,7 @@ Each rule is an INVARIANT; the harm says why it matters. | Rule | Holds when | Harm if broken | | --- | --- | --- | | `OnceEach` | A group is handed out at most once, whichever copies deliver it. | A viewer plays a group twice after a failover. | -| `CleanEndIsComplete` | A clean end comes with every group the serving route held once complete, and every group all live routes hold. | The track ends cleanly with a hole a route could still fill; the viewer loses media silently. | +| `CleanEndIsComplete` | A clean end comes with every group the serving route held once complete, every group all live routes hold, and every group all live routes held as it ended, late ones included. | The track ends cleanly with a hole a route could still fill, or one a group in flight already filled; the viewer loses media silently. | | `EndWaitsForReplaced` | The serving copy draining ends the track only once no copy it replaced may still deliver a group the reader lacks below that end. | A group still in flight on the old route is lost at the end. | | `CleanEndIsDeclared` | A clean end only follows a route declaring one. | A failed route reads as the broadcast ending. | | `FailsOnlyUnroutable` | The track fails only once the front gave up: a route failing is the front's to replace. | A single relay failure ends playback that another route could carry. | @@ -28,7 +28,7 @@ quest gate just rs tla moq-net/verify/resume/ResumeReader.tla quest gate just rs tla moq-net/verify/resume/ResumeReader.tla -config rs/moq-net/verify/resume/ThreeRoutes.cfg ``` -The first checks every rule over two routes, three groups, two switches, and a reader holding each group until it finishes it: about 14.3M distinct states, in minutes. The second trades a group for a third route, so a route can come up after another failed: about 7.1M states, in minutes. Both pass at the sync commit. Nightly runs every config here and fails if a config with an `\* Open finding:` line passes. +The first checks every rule over two routes, three groups, two switches, and a reader holding each group until it finishes it: about 19.9M distinct states, in about five minutes. The second trades a group for a third route, so a route can come up after another failed: about 7.4M states, in about a minute. Both pass at the sync commit. Nightly runs every config here and fails if a config with an `\* Open finding:` line passes. ## Open findings @@ -40,17 +40,17 @@ Follow Reading a failure in `rs/moq-net/verify/README.md`. Replay the path as a ## Coverage -Sync commit: `74e05ddc7039cc32f8968d9425e8224e82d4d886`. Run `git log 74e05ddc70.. -- rs/moq-net/src/model/resume.rs rs/moq-net/src/model/track.rs` to see what changed since. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function. +Sync commit: `8c939bc9d05e5e882fb4e042a9efbf2db227b721`, plus [moq-dev/moq#9103](https://github.com/moq-dev/moq/pull/9103), which reads a group landing late on a drained copy. Run `git log 8c939bc9d0.. -- rs/moq-net/src/model/resume.rs rs/moq-net/src/model/track.rs` to see what changed since. Follow Coverage in `rs/moq-net/verify/README.md` after a change to a covered function. | Code | Model | | --- | --- | | `rs/moq-net/src/model/resume.rs` `Reader::sync` (reuse of a route's copy, new copy at `newest`) | `Sync`, `Serve` | | `rs/moq-net/src/model/resume.rs` `Reader::backfill`, `backfill_gaps` | `Backfill`, `BackfillGaps` | -| `rs/moq-net/src/model/resume.rs` `Reader::poll_lowest`, `deliverable`, `take`, `observe` | `Poll`, `Scan`, `Peek`, `Deliverable`, `Take` | +| `rs/moq-net/src/model/resume.rs` `Reader::poll_lowest`, `deliverable`, `take`, `observe`, `Copy::done` | `Poll`, `Pass`, `Scan`, `Peek`, `Deliverable`, `Take` | | `rs/moq-net/src/model/resume.rs` `Reader::bound_at_end`, `let_go`, `Copy::retains` | `BoundAtEnd`, `LetGo`, `Retains` | | `rs/moq-net/src/model/resume.rs` `Reader::poll_end` | `PollEnd` | | `rs/moq-net/src/model/resume.rs` `Producer::splice`, `serve_from`, `Producer::end`, `Phase::Finished` | `Serve`, `FrontFinish`, `FrontFail` | -| `rs/moq-net/src/model/track.rs` `TrackState::is_complete`, `may_land`, `max_key`, `Consumer::latest_key`, `track::Subscriber::has_unread_group` | `Seal`, `Held`, `MayLand`, `Latest`, `HasUnread` | +| `rs/moq-net/src/model/track.rs` `TrackState::is_complete`, `may_land`, `max_key`, `Consumer::latest_key`, `Consumer::poll_arrived`, `track::Subscriber::has_unread_group`, `Producer::finish_shared` | `Seal`, `Held`, `MayLand`, `Latest`, `Landed`, `HasUnread`, `Settle` | ## Not covered diff --git a/js/net/src/internal.ts b/js/net/src/internal.ts index abe2f53e57..193870c6ce 100644 --- a/js/net/src/internal.ts +++ b/js/net/src/internal.ts @@ -780,6 +780,8 @@ export const hooks: { tailGrace: (producer: Producer) => Duration; /** Snapshot the latest group key for wire positioning; assigned by `track.ts`. */ latestKey: (subscriber: Subscriber) => bigint | undefined; + /** How many groups arrived on the subscriber's copy so far; assigned by `track.ts`. */ + arrived: (subscriber: Subscriber) => number; /** * The source's declared start, raised to the subscriber's cursor floor, or `"pending"` while * the upstream START that resolves it has yet to arrive; assigned by `track.ts`. @@ -1182,6 +1184,9 @@ export const hooks: { latestKey: () => { throw new Error("track.ts not loaded"); }, + arrived: () => { + throw new Error("track.ts not loaded"); + }, sourceStart: () => { throw new Error("track.ts not loaded"); }, diff --git a/js/net/src/resume.model.test.ts b/js/net/src/resume.model.test.ts index bc33e637c6..819a01bc78 100644 --- a/js/net/src/resume.model.test.ts +++ b/js/net/src/resume.model.test.ts @@ -1079,6 +1079,35 @@ test("regressions", async () => { ], finish: false, }, + // The serving route ends while a replaced copy holds the read open for a group under its + // bound: groups in range still in flight on the ended route land, and are read. Mirrors + // the Rust case. + { + ops: [ + { kind: "join", reader: 1, spec: spec(false, "hour", undefined, false) }, + { kind: "up", instance: 0, slot: 1 }, + { kind: "join", reader: 0, spec: spec(false, "hour", [6, 8], false) }, + { kind: "write", instance: 0, slot: 0, group: undefined, frames: 1 }, + { kind: "write", instance: 0, slot: 1, group: 11, frames: 1 }, + { kind: "write", instance: 0, slot: 1, group: 2, frames: 1 }, + { kind: "end", instance: 0, slot: 1 }, + ], + finish: false, + }, + // The same for a reader without a range, once the serving copy's first group bounds the + // replaced one below the groups still in flight. + { + ops: [ + { kind: "join", reader: 1, spec: spec(false, "hour", undefined, false) }, + { kind: "up", instance: 0, slot: 1 }, + { kind: "join", reader: 0, spec: spec(false, "hour", undefined, false) }, + { kind: "write", instance: 0, slot: 0, group: undefined, frames: 1 }, + { kind: "write", instance: 0, slot: 1, group: 2, frames: 1 }, + { kind: "write", instance: 0, slot: 1, group: 11, frames: 1 }, + { kind: "end", instance: 0, slot: 1 }, + ], + finish: false, + }, ]; const failures: string[] = []; for (const [i, c] of regressions.entries()) { diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts index 6980f1a560..65b2abeb16 100644 --- a/js/net/src/resume.test.ts +++ b/js/net/src/resume.test.ts @@ -420,6 +420,81 @@ test("a group below the replacement's resume point is backfilled", async () => { } }); +test("a late group on a drained serving copy is read", async () => { + const clock = new Manual(); + const installed = clock.install(); + const front = new Front(); + const sources = [new BroadcastProducer(), new BroadcastProducer()]; + const handles = sources.map((source) => source.consume()); + const [ta, tb] = sources.map(copy); + hooks.receiveTrack(ta.producer); + hooks.receiveTrack(tb.producer); + front.serve(handles[0]); + const sub = front.consumer.track("seq").subscribe({ delay: { max: Duration.millis(3_600_000) } }); + try { + await clock.advance(Duration.ZERO); + ta.write(0n); + expect(drain(sub)).toEqual([0n]); + front.serve(handles[1]); + await clock.advance(Duration.ZERO); + // Its first group bounds the replaced copy at 1, which it may still deliver. + tb.write(2n); + await clock.advance(Duration.ZERO); + expect(drain(sub)).toEqual([2n]); + tb.producer.finish(); + const next = sub.recvGroup(); + expect(await isPending(next)).toBe(true); + tb.write(1n); + await clock.advance(Duration.ZERO); + expect((await next)?.start).toBe(1n); + } finally { + sub.close(); + front.consumer.close(); + for (const handle of handles) handle.close(); + for (const source of sources) source.close(); + installed.close(); + } +}); + +test("a late group under the resume point is backfilled again", async () => { + const clock = new Manual(); + const installed = clock.install(); + const front = new Front(); + const sources = [new BroadcastProducer(), new BroadcastProducer()]; + const handles = sources.map((source) => source.consume()); + const [ta, tb] = sources.map(copy); + hooks.receiveTrack(ta.producer); + hooks.receiveTrack(tb.producer); + front.serve(handles[0]); + const sub = front.consumer.track("seq").subscribe({ delay: { max: Duration.millis(3_600_000) } }); + try { + await clock.advance(Duration.ZERO); + ta.write(2n); + ta.producer.finish(); + expect(sub.tryRecvGroup()?.start).toBe(2n); + // A switch away and back resumes `ta` past 2; the backfill below it finds its track + // complete and runs out, while `tb`'s copies hold the read open. + front.serve(handles[1]); + await clock.advance(Duration.ZERO); + expect(drain(sub)).toEqual([]); + front.serve(handles[0]); + await clock.advance(Duration.ZERO); + const next = sub.recvGroup(); + expect(await isPending(next)).toBe(true); + // `tb` fails, and 1 lands late on `ta` before the read looks again. + tb.producer.abort(new StreamError(StreamCode.SessionClosed)); + ta.write(1n); + await clock.advance(Duration.ZERO); + expect((await next)?.start).toBe(1n); + } finally { + sub.close(); + front.consumer.close(); + for (const handle of handles) handle.close(); + for (const source of sources) source.close(); + installed.close(); + } +}); + // The switch's backfill waits on a fill the finished new route's handler still owes, rather // than counting as drained while the fill is out. test("a backfill waits on a fill the finished route still owes", async () => { diff --git a/js/net/src/resume.ts b/js/net/src/resume.ts index c39540c21b..b3495b1815 100644 --- a/js/net/src/resume.ts +++ b/js/net/src/resume.ts @@ -350,8 +350,12 @@ interface Copy { * groups the newer one may never have. */ until?: bigint | null; - /** The group channel ran out: cleanly (`null`) or failed. */ - done?: Error | null; + /** + * The group channel ran out: failed, or cleanly with how many groups had arrived on the copy + * by then, since one arriving later is still in flight below its end, which a read still open + * takes. + */ + done?: Error | number; /** * When a newer group superseded every group under the bound: the newer copy's first group, or * the track's end. What the reader still lacks from it is waited for from here, for no longer @@ -824,6 +828,7 @@ class Reader implements ResumeSource { this.#learnEnd(); if (this.#closed) return { kind: "done" }; this.#groupsRead = true; + this.#reopen(); this.#backfillGaps(delivered); if (dispositions) { const dropped = this.#tryDrop(delivered); @@ -860,10 +865,11 @@ class Reader implements ResumeSource { // Drained: the track ended as the copy did, which is not cleanly when its // session went away short of the declared end. if (recv.kind === "done") { - copy.done = copy.sub.closed.peek() ?? null; - if (copy.done === null && hooks.subscriberEnd(copy.sub)) this.#settled = true; + const closed = copy.sub.closed.peek(); + copy.done = closed instanceof Error ? closed : hooks.arrived(copy.sub); + if (!(closed instanceof Error) && hooks.subscriberEnd(copy.sub)) this.#settled = true; } else if (recv.kind === "error") copy.done = recv.error; - else if (this.#drained(copy, delivered)) copy.done = null; + else if (this.#drained(copy, delivered)) copy.done = hooks.arrived(copy.sub); break; } } @@ -873,7 +879,7 @@ class Reader implements ResumeSource { const dropped = this.#tryDrop(delivered); if (dropped) return dropped; } - return this.#end((copy) => copy.done); + return this.#end((copy) => (typeof copy.done === "number" ? null : copy.done)); } /** @@ -981,6 +987,19 @@ class Reader implements ResumeSource { return hooks.spent(copy.sub, start, end, delivered); } + // A copy that ran out cleanly still takes a group in flight below its end while the read stays + // open, as a cursor still reading it would. One landing on the serving copy may lie under its + // resume point, which is not that copy's to read, and a backfill that already ran out was let + // go: ask its route again. Mirrors the Rust `Reader::poll_lowest`. + #reopen(): void { + const serving = this.#serving(); + for (const copy of this.#copies) { + if (typeof copy.done !== "number" || hooks.arrived(copy.sub) <= copy.done) continue; + copy.done = undefined; + if (copy === serving && !copy.replaced && copy.floor !== undefined) this.#resumed = true; + } + } + // Whether `copy` may still feed groups: its channel has not run out, and what it holds is // within the reader's budget, which ends a full budget after a newer group superseded it. #feeds(copy: Copy): boolean { diff --git a/js/net/src/track.ts b/js/net/src/track.ts index e0d90d6547..9b48b8041f 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -816,6 +816,9 @@ class TrackState { */ reads?: (key: bigint) => boolean; groups = new Signal([]); + // How many groups were mirrored in, whatever this sink reads: one arriving after its reader + // ran out is still in flight below the end. + arrived = 0; // Keys the upstream publisher gave up, with why: an incoming SUBSCRIBE_SKIP or // SUBSCRIBE_DROP, or a producer cut. Kept apart from the cache so reclaim cannot // erase a disposition a downstream publisher still has to forward, and as runs so a wide @@ -2073,6 +2076,7 @@ export class Producer { entry.mirrors.set(sink, dst); if (sink.reads?.(dst.start) === false) hooks.groupReads(dst, false); timelineInsert(sink.timeline, dst); + sink.arrived++; void dst.readable().then(() => sink.timelineChanged.update((revision) => revision + 1)); sink.latest = max(sink.latest ?? 0n, dst.start); sink.received.set(max(sink.received.peek(), dst.start + 1n)); @@ -4362,6 +4366,7 @@ export class Subscriber { return stop; }; hooks.latestKey = (subscriber) => (subscriber.#resume ? subscriber.#resume.latest() : subscriber.#state.latest); + hooks.arrived = (subscriber) => (subscriber.#resume ? 0 : subscriber.#state.arrived); hooks.bufferedLargest = (subscriber) => subscriber.#bufferedLargest(); hooks.objectLargest = (subscriber) => { if (subscriber.#resume) return subscriber.#resume.objectLargest(); diff --git a/quest/m0/nightly/resume-window-mutants.md b/quest/m0/nightly/resume-window-mutants.md new file mode 100644 index 0000000000..9536c99a39 --- /dev/null +++ b/quest/m0/nightly/resume-window-mutants.md @@ -0,0 +1,36 @@ +# [M] No mutant of the resume reader's window survives its tests + +## Goal + +Nightly's mutants job never flags `rs/moq-net/src/model/resume.rs` +`Reader::window`: each of its two surviving mutants is killed by a test, or +shown equivalent and excluded in `.cargo/mutants.toml` with the reason. + +## Plan + +[moq-dev/moq#9019](https://github.com/moq-dev/moq/pull/9019) left both +unexcluded, expecting the resume model's seed-3 failure to be range logic in +`Reader::window`. It was not: [moq-dev/moq#9103](https://github.com/moq-dev/moq/pull/9103) +found a cleanly drained copy dropping groups that landed late, and its fix +leaves `Reader::window` as it was. The mutants: + +- `> → >=` in the capped-copy filter (`resolved.end.is_none_or(|end| end > resolved.start)`): + differs only when a capped copy resolves a range to an empty one ending at + its start, which the original treats as saying nothing and the mutant + places as an empty window. +- `< → >` on the cap on the end (`cap.is_none_or(|cap| *end < cap)`): a capped + copy's own wire ends at its cap, so its resolved end never passes it, and the + mutant drops every end the original keeps. The window then waits for another + copy to place its end. This shows only when a ranges change leaves the + window's time end unsettled on the serving copy while a replaced, capped copy + already holds a group past it. A bounded read then ends at its range's end + on the original, and stays open on the mutant until the track ends. + `resume_model.rs` accepts both, an open read and one ended at its range, + which is why 60k sweep cases never killed it: a unit test in `resume.rs`'s + `mod test` that asserts the read ends is the likely kill. + +Public API: none. Wire: none. + +## Related + +- [Mutants of the other modelled code](/quest/m4/mutants-modelled.md) - the same Nightly job, widened to more files diff --git a/quest/m1/precut-fixes/resume-js-window.md b/quest/m1/precut-fixes/resume-js-window.md new file mode 100644 index 0000000000..2fae2e329d --- /dev/null +++ b/quest/m1/precut-fixes/resume-js-window.md @@ -0,0 +1,37 @@ +# [M] A JS resumed reader hands out no group outside its window + +## Goal + +A `@moq/net` reader that resumes a track across routes (`js/net/src/resume.ts`) +never hands out a group outside the window its ranges resolved to, as the Rust +reader never does. `MOQ_RESUME_SEED=3 MOQ_RESUME_CASES=50000` and seeds 1 and 7 +pass on `js/net/src/resume.model.test.ts`, so Nightly's `test/resume-model.sh` +passes on its JS side. + +## Plan + +Found by [moq-dev/moq#9103](https://github.com/moq-dev/moq/pull/9103) while +sweeping its fix: on `main` at [4f3ba3317f](https://github.com/moq-dev/moq/commit/4f3ba3317f), +the JS model fails fast with `group 0 outside ranges` (seed 3 at 89 cases, +seed 1 at 434, seed 7 at 196), with or without that PR. + +```sh +MOQ_RESUME_SEED=3 MOQ_RESUME_CASES=50000 bun test ./js/net/src/resume.model.test.ts +``` + +Counterexample: join route 0 with range `[1, 3]`; route 0/1 comes up; write +group 1 (two frames), then group 0, then the next group on 0/0. The serving +copy resolves the time anchor against its own cache, which holds only group 0, +and hands group 0 out, though the reader's window starts at 1. + +Introduced by [moq-dev/moq#9027](https://github.com/moq-dev/moq/pull/9027) +(`0c14c87c49`, "@moq/net addresses groups by start"): the same seeds pass on +its parent. Rust's `Reader::poll_lowest` filters each peeked group with +`self.window().contains(key)` ("the window decides"), leaving an ordered +cursor in place when `owes_below` still owes a lower group. JS `#lowest` and +the serving loop in `tryGroup` check only `#deliverable` (the copy's cap and +`delivered`). Mirror that filter, with the window built from `#placed` / +`resolved()`, and add the counterexample to `resume.model.test.ts`'s +regressions. + +Public API: none. Wire: none. diff --git a/quest/m1/precut-fixes/resume-range-reader.md b/quest/m1/precut-fixes/resume-range-reader.md deleted file mode 100644 index 7cae000fe5..0000000000 --- a/quest/m1/precut-fixes/resume-range-reader.md +++ /dev/null @@ -1,43 +0,0 @@ -# [M] A ranged reader resumed across routes gets exactly the groups in its range - -## Goal - -A reader that asks a resumed track (`rs/moq-net/src/model/resume.rs`) for a -range of groups is handed every group of the logical track in that range, and -none outside it, whichever routes come up, write, and end while it reads. -The resume model's random sweep (`model/resume_model.rs`) passes on that -seed, and the resume TLA+ model (`/verify-resume`) checks the fix. - -## Plan - -Found by [moq-dev/moq#9019](https://github.com/moq-dev/moq/pull/9019) while -killing `Reader::window` mutants: the resume model fails on unmutated `main`. -Reproduced on `main` at [2dcb5f8994](https://github.com/moq-dev/moq/commit/2dcb5f8994): - -```sh -MOQ_RESUME_SEED=3 MOQ_RESUME_CASES=50000 cargo test -p moq-net --all-features --lib readers_match_the_logical_track -``` - -> reader 0: missing groups [6, 7]; handed out [0, 2, 1], dropped {}, floor None, -> spec Spec { ordered: false, budget: Hour, range: Some((6, 8)), drops: false } - -Shrunk ops: `Join r1 live; Up 0/1; Join r0 (6,8); Write 0/0 next; Write 0/1 g11; Write 0/1 g2; End 0/1`. - -Decided 2026-11-17 (maintainer, [moq-dev/moq#9019](https://github.com/moq-dev/moq/pull/9019#issuecomment-6961416403)): -✅ a quest in this epic, sized M, so it gates the `0.22.0` cut (rejected: a -standalone m1 quest outside the gate; dropping it). - -- Reproduce with that seed, then pin the shrunk case as a unit test in - `resume_model.rs` next to `unchanged_routes_keep_groups_owed`. -- Fix at the root in `resume.rs`, wherever the reader's range stops bounding - what it picks and what it waits for (`Reader::window`, `poll_lowest`, - `deliverable`, `poll_end` are the likely places). No widening of the model's - expectations. -- Check the fix with `/verify-resume` (`rs/moq-net/verify/resume/`), extending - `ResumeReader.tla` with a ranged reader if it has none, so every ordering is - checked rather than one seed. -- Settles the two `Reader::window` mutants [moq-dev/moq#9019](https://github.com/moq-dev/moq/pull/9019) left unexcluded (`> → >=` on - the capped-copy filter and `< → >` on the cap on the end): once the range - logic is fixed, kill each with a test or show it equivalent and exclude it - in `.cargo/mutants.toml` with the reason. -- Find the culprit for `## Caused by` by blaming the fixed lines. diff --git a/rs/moq-net/src/model/resume.rs b/rs/moq-net/src/model/resume.rs index 448aead006..d9bf17d5ef 100644 --- a/rs/moq-net/src/model/resume.rs +++ b/rs/moq-net/src/model/resume.rs @@ -14,8 +14,9 @@ //! grace a passed group is waited for), and while a group still read is read from it or may //! continue from it (it holds that group, or may yet get it). Nothing under the bound is //! waited for past the reader's budget. The serving copy draining cleanly ends the track -//! only once nothing it replaced may still deliver below that end. A reader that never asked -//! for a group keeps no copy. +//! only once nothing it replaced may still deliver below that end. Until then, a copy that +//! ran out still hands out a group landing on it late, one still in flight below the end its +//! session settled. A reader that never asked for a group keeps no copy. //! //! A new copy is asked from where the reader got to, so it never has the groups below the //! reader's local floor that the copies it replaced cached. Lowering that floor asks the @@ -596,8 +597,10 @@ struct Copy { /// Bounded once the newer copy delivered its first group: nothing past this group is /// taken from it. Unset while it may still feed a group the newer copy never has. until: Option>, - /// The group channel ran out. - done: Option>, + /// The group channel ran out: cleanly with how many groups had arrived on the copy by + /// then, since one arriving later is still in flight below its end, which a read still + /// open takes. + done: Option>, /// When a newer group superseded every group under the bound: the newer copy's first /// group, or the track's end. What the reader still lacks from it is waited for from /// here, for no longer than the reader's budget. @@ -665,7 +668,7 @@ impl Copy { /// bounds may hold a group the new ones admit, and runs out again at once otherwise. A /// copy that failed stays failed. fn reopen(&mut self) { - if matches!(self.done, Some(Ok(()))) { + if matches!(self.done, Some(Ok(_))) { self.done = None; } } @@ -1122,7 +1125,7 @@ impl Subscriber { .copies .last() .filter(|copy| Some(copy.generation) == reader.generation)?; - if !matches!(serving.done, Some(Ok(()))) + if !matches!(serving.done, Some(Ok(_))) || !matches!(serving.track.poll_complete(&kio::Waiter::noop()), Poll::Ready(Ok(()))) { return None; @@ -1555,7 +1558,7 @@ impl Reader { else { return false; }; - if !matches!(serving.done, Some(Ok(()))) { + if !matches!(serving.done, Some(Ok(_))) { return false; } let until = match self @@ -1686,10 +1689,22 @@ impl Reader { // superseded it a full budget ago. // A fully handed-out bounded copy still backs groups being read, but cannot // supply another one. Keep its recovery lease without polling its empty range. - let spent = copy.done.is_some() || copy.end().is_some_and(|end| delivered.covers(groups.0..end)); + let spent = copy.end().is_some_and(|end| delivered.covers(groups.0..end)); + // A copy that ran out cleanly still takes a group in flight below its end while the + // read stays open, as a cursor still reading it would. + let ran_out = match ©.done { + None => false, + Some(Ok(drained)) if !spent && copy.track.poll_arrived(*drained, waiter).is_ready() => true, + Some(_) => continue, + }; if spent || copy.expired(budget, waiter) { continue; } + if ran_out { + copy.done = None; + } + let mut late = ran_out; + let mut arrivals = copy.track.arrived(); loop { let copy = &mut self.copies[index]; let sub = match copy.poll_ready(groups, waiter) { @@ -1752,10 +1767,17 @@ impl Reader { // Drained: the track ended as the copy did, which is not cleanly when // its session went away short of the declared end. Poll::Ready(Ok(None)) => { + // Counted before the read, so a group landing during it is read again. copy.done = Some(match copy.track.poll_complete(&kio::Waiter::noop()) { - Poll::Ready(end) => end, - Poll::Pending => Ok(()), + Poll::Ready(end) => end.map(|()| arrivals), + Poll::Pending => Ok(arrivals), }); + if matches!(copy.done, Some(Ok(_))) && copy.track.poll_arrived(arrivals, waiter).is_ready() { + copy.done = None; + late = true; + arrivals = copy.track.arrived(); + continue; + } break; } Poll::Ready(Err(err)) => { @@ -1764,6 +1786,12 @@ impl Reader { } } } + // A late group under the serving copy's resume point is not this copy's to read, and + // a backfill that already ran out was let go: ask its route again. + let copy = &self.copies[index]; + if late && !copy.replaced && copy.floor.is_some() && Some(copy.generation) == self.generation { + self.resumed.get_or_insert_with(|| (None, Vec::new())); + } } if let Some(lowest) = lowest { return Some(Poll::Ready(Ok(Some(lowest)))); @@ -1778,7 +1806,7 @@ impl Reader { if !parks && !waits && self.exhausted(&[delivered], waiter) { return Some(Poll::Ready(Ok(None))); } - match self.poll_end(waiter, |copy| copy.done.clone()) { + match self.poll_end(waiter, |copy| copy.done.clone().map(|done| done.map(|_| ()))) { // The serving copy drained, but a copy it replaced may still deliver a group below // its end: the track ends once that copy has. Some(Poll::Ready(Ok(None))) if drained && waits => Some(Poll::Pending), @@ -6904,6 +6932,66 @@ mod test { assert_eq!(drain(&mut sub, false), [2, 6]); } + /// A serving copy that ran out at its session's settled end still hands out a group in + /// flight below that end while a copy it replaced holds the read open, and wakes the read. + #[test] + fn a_late_group_on_a_drained_serving_copy_is_read() { + let routes = Producer::new(); + let logical = logical(&routes); + let a = received_copy(); + routes.serve(a.consume()); + let mut sub = subscribe(&logical, HOUR); + write(&a, 0); + assert_eq!(drain(&mut sub, false), [0]); + + let b = received_copy(); + routes.serve(b.consume()); + // Its first group bounds the replaced copy at 1, which it may still deliver. + write(&b, 2); + assert_eq!(drain(&mut sub, false), [2]); + b.finish_shared().unwrap(); + let wake = Arc::new(Wake::default()); + let waiter = kio::Waiter::new(wake.clone().into()); + assert!( + sub.poll_recv_group(&waiter).is_pending(), + "the replaced copy may still deliver 1" + ); + + write(&b, 1); + assert!( + wake.0.load(std::sync::atomic::Ordering::Relaxed), + "the late group wakes the read" + ); + assert_eq!(drain(&mut sub, false), [1]); + } + + /// A late group under the serving copy's resume point is asked of its route again, once + /// the backfill that asked for it ran out and was let go. + #[test] + fn a_late_group_under_the_resume_point_is_backfilled_again() { + let routes = Producer::new(); + let logical = logical(&routes); + let a = received_copy(); + routes.serve(a.consume()); + let mut sub = subscribe(&logical, HOUR); + write(&a, 2); + a.finish_shared().unwrap(); + assert_eq!(recv(&mut sub).start, 2); + + // A switch away and back resumes `a` past 2; the backfill below it finds its track + // complete and runs out, while `b`'s copies hold the read open. + let b = received_copy(); + routes.serve(b.consume()); + assert!(sub.poll_recv_group(&kio::Waiter::noop()).is_pending()); + routes.serve(a.consume()); + assert!(sub.poll_recv_group(&kio::Waiter::noop()).is_pending()); + // `b` fails, and 1 lands late on `a` before the read looks again. + b.abort(Error::Stream(StreamError::Session(SessionError::Cancel))) + .unwrap(); + write(&a, 1); + assert_eq!(drain(&mut sub, false), [1]); + } + /// A replacement copy keeps each range where the serving copy resolved it: the floor /// raises a later time range's start rather than lowering it to the floor, and capping /// the replaced copy never widens a time end. diff --git a/rs/moq-net/src/model/resume_model.rs b/rs/moq-net/src/model/resume_model.rs index d111a099ff..52d2f71696 100644 --- a/rs/moq-net/src/model/resume_model.rs +++ b/rs/moq-net/src/model/resume_model.rs @@ -1445,6 +1445,76 @@ fn regressions() { ], finish: false, }, + // The serving route ends while a replaced copy holds the read open for a group under + // its bound: groups in range still in flight on the ended route land, and are read. + Case { + ops: vec![ + Op::Join { + reader: 1, + spec: spec(false, Budget::Hour, None, false), + }, + Op::Up { instance: 0, slot: 1 }, + Op::Join { + reader: 0, + spec: spec(false, Budget::Hour, Some((6, 8)), false), + }, + Op::Write { + instance: 0, + slot: 0, + group: None, + frames: 1, + }, + Op::Write { + instance: 0, + slot: 1, + group: Some(11), + frames: 1, + }, + Op::Write { + instance: 0, + slot: 1, + group: Some(2), + frames: 1, + }, + Op::End { instance: 0, slot: 1 }, + ], + finish: false, + }, + // The same for a reader without a range, once the serving copy's first group bounds + // the replaced one below the groups still in flight. + Case { + ops: vec![ + Op::Join { + reader: 1, + spec: spec(false, Budget::Hour, None, false), + }, + Op::Up { instance: 0, slot: 1 }, + Op::Join { + reader: 0, + spec: spec(false, Budget::Hour, None, false), + }, + Op::Write { + instance: 0, + slot: 0, + group: None, + frames: 1, + }, + Op::Write { + instance: 0, + slot: 1, + group: Some(2), + frames: 1, + }, + Op::Write { + instance: 0, + slot: 1, + group: Some(11), + frames: 1, + }, + Op::End { instance: 0, slot: 1 }, + ], + finish: false, + }, ]; for case in cases { if let Err(err) = run(&case) { diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index 21fdffca76..01b416416d 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -6583,6 +6583,28 @@ impl Consumer { }) } + /// How many groups arrived here so far; see [`Self::poll_arrived`]. + pub(super) fn arrived(&self) -> usize { + self.with_state(|state| state.map_or(0, |state| state.read().next_arrival)) + } + + /// Ready once more than `seen` groups arrived here. A received copy still takes the groups + /// in flight below an end its readers stopped waiting at. Pending for good once the track closes. + pub(super) fn poll_arrived(&self, seen: usize, waiter: &kio::Waiter) -> Poll<()> { + self.with_state(|state| { + let Some(state) = state else { + return Poll::Pending; + }; + match state.poll(waiter, |state| match state.next_arrival > seen { + true => Poll::Ready(()), + false => Poll::Pending, + }) { + Poll::Ready(Ok(())) => Poll::Ready(()), + _ => Poll::Pending, + } + }) + } + /// Whether a group in `groups` may still land here; see `TrackState::may_land`. Never /// once no producer is left to write one. Parks `waiter` until its gap folds. pub(super) fn may_land( diff --git a/rs/moq-net/verify/resume/ResumeReader.tla b/rs/moq-net/verify/resume/ResumeReader.tla index fd9d8596c6..24c0d41e52 100644 --- a/rs/moq-net/verify/resume/ResumeReader.tla +++ b/rs/moq-net/verify/resume/ResumeReader.tla @@ -17,7 +17,8 @@ (* stale or expires), arrival order, no local range, copies that answer at *) (* once, and a reader that joins before any group lands. A route declares *) (* its end once it holds the last group, a gap below it may still land, *) -(* and a route that declared its end never fails. *) +(* even once its session settled that end, and a route that declared its *) +(* end never fails. *) (***************************************************************************) EXTENDS Integers, Sequences, FiniteSets @@ -82,7 +83,7 @@ Admits(floor, until, g) == NewCopy(id, t, gen, floor, rep, until) == [id |-> id, t |-> t, gen |-> gen, floor |-> floor, rep |-> rep, - until |-> until, done |-> "no", since |-> rep, idx |-> 1, + until |-> until, done |-> "no", drained |-> 0, since |-> rep, idx |-> 1, took |-> {}, owed |-> {}] \* The wire asks for [floor, until] instead (`Copy::bound`, or `Copy::send` on a @@ -121,10 +122,11 @@ Peek(c, dlv, fuel) == THEN [c |-> c, g |-> g, src |-> "owed"] ELSE Peek([c EXCEPT !.took = @ \cup {g}, !.owed = @ \ {g}], dlv, fuel - 1) \* `TrackState::poll_recv_group`: past the last arrival, a complete track - \* ends the walk, and an aborted one fails it. + \* ends the walk, and an aborted one fails it. A clean end records the + \* arrivals it saw (`Copy::drained`). ELSE IF c.idx > Len(arr) THEN IF trk[c.t].whole - THEN [c |-> [c EXCEPT !.done = "ok"], g |-> None, src |-> "none"] + THEN [c |-> [c EXCEPT !.done = "ok", !.drained = Len(arr)], g |-> None, src |-> "none"] ELSE IF trk[c.t].failed THEN [c |-> [c EXCEPT !.done = "err"], g |-> None, src |-> "none"] ELSE [c |-> c, g |-> None, src |-> "none"] @@ -230,20 +232,28 @@ BackfillGaps(r) == IN IF gaps = {} THEN r1 ELSE Backfill(r1, Min(gaps), at) \* The copy loop of `poll_lowest`: peek each copy, keeping the lowest group; -\* a tie goes to the newer copy. -RECURSIVE Scan(_, _, _, _) -Scan(cs, dlv, i, best) == - IF i > Len(cs) THEN [cs |-> cs, best |-> best] +\* a tie goes to the newer copy. A copy that ran out cleanly is read again +\* once a group arrived on it since (`Copy::drained`); one arriving on the +\* serving copy, which resumed at a floor, may be under it (`late`). +RECURSIVE Scan(_, _, _, _, _, _) +Scan(cs, dlv, gen, i, best, late) == + IF i > Len(cs) THEN [cs |-> cs, best |-> best, late |-> late] ELSE - LET c == cs[i] IN - IF c.done # "no" \/ (Bounded(c) /\ Covers(dlv, 0, End(c))) - THEN Scan(cs, dlv, i + 1, best) + LET c0 == cs[i] + spent == Bounded(c0) /\ Covers(dlv, 0, End(c0)) + reopens == c0.done = "ok" /\ ~spent /\ Len(trk[c0.t].arr) > c0.drained + c == IF reopens THEN [c0 EXCEPT !.done = "no"] ELSE c0 + serves == ~c.rep /\ c.gen = gen /\ c.floor # None + IN + IF c.done # "no" \/ spent + THEN Scan(cs, dlv, gen, i + 1, best, late) ELSE LET p == Peek(c, dlv, Fuel) cs2 == [cs EXCEPT ![i] = p.c] better == p.g # None /\ (best.g = None \/ p.g <= best.g) - IN Scan(cs2, dlv, i + 1, - IF better THEN [i |-> i, g |-> p.g, src |-> p.src] ELSE best) + IN Scan(cs2, dlv, gen, i + 1, + IF better THEN [i |-> i, g |-> p.g, src |-> p.src] ELSE best, + late \/ (reopens /\ serves)) NoBest == [i |-> 0, g |-> None, src |-> "none"] @@ -316,13 +326,16 @@ PollEnd(r, drained, waits) == ELSE IF front.end = "ok" /\ (s = {} \/ \E c \in s : c.done # "no") THEN ok ELSE "no" -\* One `poll_recv_group` on the logical track (`Reader::poll_lowest`). -Poll(r) == - LET r1 == BackfillGaps(Sync(r, front)) - sc == Scan(r1.copies, r1.dlv, 1, NoBest) - r2 == [r1 EXCEPT !.copies = sc.cs] +\* One `poll_recv_group` on the logical track (`Reader::poll_lowest`, looked +\* at again while it returns `None`: a late group on the serving copy sets +\* `resumed`, so the next pass asks its route for what lies under its floor). +RECURSIVE Pass(_) +Pass(r1) == + LET sc == Scan(r1.copies, r1.dlv, r1.gen, 1, NoBest, FALSE) + r2 == [r1 EXCEPT !.copies = sc.cs, !.resumed = @ \/ sc.late] IN IF sc.best.g # None THEN [r |-> Take(r2, sc.best), cid |-> r2.copies[sc.best.i].id, g |-> sc.best.g] + ELSE IF sc.late THEN Pass(BackfillGaps(r2)) ELSE LET drained == \E c \in Serving(r2) : c.done = "ok" r3 == IF drained THEN BoundAtEnd(r2) ELSE r2 @@ -330,6 +343,8 @@ Poll(r) == r4 == [r3 EXCEPT !.copies = lg.copies] IN [r |-> [r4 EXCEPT !.ended = PollEnd(r4, drained, lg.waits)], cid |-> 0, g |-> None] +Poll(r) == Pass(BackfillGaps(Sync(r, front))) + \* What a clean end owes the reader: every group its serving route held once \* complete, and, as `resume_model`'s oracle owes, every group all live routes \* hold. @@ -338,6 +353,12 @@ Owed(r) == \/ Len(r.copies) > 0 /\ g \in Held(r.copies[Len(r.copies)].t) \/ \A t \in Tracks : Live(t) => g \in Held(t)} +\* What `resume_model`'s oracle owes a read that ends cleanly: every group +\* all live routes hold as it ends, late ones included. One landing after it +\* is not owed. +Carried == + {g \in Groups : \A t \in Tracks : Live(t) => g \in Landed(t)} + \* The groups the reader lacks that a replaced copy it still holds has unread, \* or may still get by the rule `may_land` applies, within what it asks for. Late(r) == @@ -360,7 +381,7 @@ Init == /\ front = [cur |-> 1, gen |-> 1, end |-> "open", switches |-> 0] \* Subscribed to the serving copy at once (`Consumer::subscribe`). /\ rd = [gen |-> 1, next |-> 2, newest |-> None, dlv |-> {}, resumed |-> TRUE, - ended |-> "no", dup |-> FALSE, late |-> {}, + ended |-> "no", dup |-> FALSE, late |-> {}, carried |-> {}, copies |-> <>] /\ reading = {} @@ -391,6 +412,15 @@ Finish(t) == /\ trk' = [trk EXCEPT ![t] = Seal([@ EXCEPT !.fin = TRUE])] /\ UNCHANGED <> +\* The route's session settles its declared end (`Producer::finish_shared` on a +\* received copy): it has every group stream it will get, so the track is +\* complete with what it holds, gaps and all. A group still in flight below +\* the end may land afterwards, and is not owed by the route. +Settle(t) == + /\ Live(t) /\ trk[t].fin /\ ~trk[t].whole + /\ trk' = [trk EXCEPT ![t].whole = TRUE, ![t].held = Landed(t)] + /\ UNCHANGED <> + \* The front serves another live route (`Producer::splice`), and every reader \* follows at once (`serve_from`). Leaving a failed route is not counted. A \* copy the front serves after a clean end (`Action::Reroute`) is for group @@ -425,7 +455,7 @@ FrontFail == Read == /\ rd.ended = "no" /\ LET p == Poll(rd) - r == IF p.r.ended = "ok" THEN [p.r EXCEPT !.late = Late(p.r)] ELSE p.r + r == IF p.r.ended = "ok" THEN [p.r EXCEPT !.late = Late(p.r), !.carried = Carried] ELSE p.r IN /\ rd' = r /\ reading' = IF Hold /\ p.g # None THEN reading \cup {[g |-> p.g, cid |-> p.cid]} ELSE reading /\ rd' # rd @@ -443,7 +473,7 @@ Done == /\ UNCHANGED vars Next == - \/ \E t \in Tracks : Up(t) \/ Down(t) \/ Finish(t) \/ Serve(t) + \/ \E t \in Tracks : Up(t) \/ Down(t) \/ Finish(t) \/ Settle(t) \/ Serve(t) \/ \E t \in Tracks, g \in Groups : Land(t, g) \/ FrontFinish \/ FrontFail \/ Read @@ -459,8 +489,9 @@ Spec == Init /\ [][Next]_vars \* A group is handed out at most once, whichever copies deliver it. OnceEach == ~rd.dup -\* A clean end comes with every group it owes (`Owed`), judged as routes go on. -CleanEndIsComplete == rd.ended = "ok" => Owed(rd) \subseteq rd.dlv +\* A clean end comes with every group it owes (`Owed`), judged as routes go on, +\* and every group all live routes held as it ended (`Carried`). +CleanEndIsComplete == rd.ended = "ok" => (Owed(rd) \cup rd.carried) \subseteq rd.dlv \* The serving copy draining ends the track only once no copy it replaced may \* still deliver a group the reader lacks below that end (`Late`, as it ended).