Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion js/net/src/fill.ts
Original file line number Diff line number Diff line change
Expand Up @@ -520,7 +520,9 @@ export function requestsOf(state: FillState): Requests {
* Most tracks never serve old content, so this lives on its own handle rather than the producer:
* a publisher backed by storage opens one to serve groups its live cache no longer holds, or
* never held. While at least one is open, a reader missing a run of groups queues a {@link Fill}
* here; with none open, the miss is given up. {@link close} it once done.
* here; with none open, the miss is given up. {@link close} it once done, and keep taking fills
* until then: a reader waits on its {@link FillLookup} until it is answered or closed, so a
* handle left open but unread stalls that reader.
*
* @public
*/
Expand Down
142 changes: 142 additions & 0 deletions js/net/src/resume.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4324,3 +4324,145 @@ test("a group an inner front hands out already failed still continues on the out
a.close();
b.close();
});

// Mirrors `an_ended_copy_leaves_a_time_start_to_the_serving_copy` in rs/moq-net/src/model/resume.rs:
// a copy that failed holds nothing past its last group, so it reaches a time start past that group
// back to its oldest. The copy continuing the track places it where the track holds it.
test("a failed copy leaves a time start to the serving copy", async () => {
const a = new BroadcastProducer();
const b = new BroadcastProducer();
const ta = copy(a);
const tb = copy(b);
hooks.receiveTrack(ta.producer);
hooks.receiveTrack(tb.producer);
const front = new Front();
front.serve(a.consume());
ta.write(64n);
const waiting = front.consumer.track("seq").subscribe({ delay: { max: null }, ranges: [{ start: { time: 66n } }] });
try {
const read: bigint[] = [];
const reading = (async () => {
for (let group = await waiting.recvGroup(); group; group = await waiting.recvGroup()) {
read.push(group.start);
group.close();
if (read.length === 2) return;
}
})();
await settle();
// The source fails before the group holding the start arrives.
ta.producer.abort(new StreamError(StreamCode.SessionClosed));
await settle();
expect(read).toEqual([]);
// Nor does the reader report a start the failed copy placed.
expect(hooks.servingResolved(waiting)[0]).toBeUndefined();
expect(hooks.servingStarts(waiting, 0n)[0]).toBeUndefined();
front.serve(b.consume());
await settle();
// Fetched ahead of the live feed's 66.
tb.write(67n);
tb.write(66n);
await reading;
expect(read.sort()).toEqual([66n, 67n]);
expect(hooks.servingResolved(waiting)[0]?.start).toBe(66n);
} finally {
waiting.close();
front.consumer.close();
a.close();
b.close();
}
});

// A range from the oldest group needs no copy to place it, so a failed copy's buffered group goes
// out without waiting for a replacement, as the Rust reader's window starting unheld places it.
test("a failed copy hands out its buffered group under ranges no copy places", async () => {
const a = new BroadcastProducer();
const ta = copy(a);
hooks.receiveTrack(ta.producer);
const front = new Front();
front.serve(a.consume());
const sub = front.consumer.track("seq").subscribe({ delay: { max: null } });
try {
ta.write(64n);
ta.producer.abort(new StreamError(StreamCode.SessionClosed));
await settle();
const group = sub.tryRecvGroup();
expect(group?.start).toBe(64n);
group?.close();
} finally {
sub.close();
front.consumer.close();
a.close();
}
});

// A failed copy holding a group back for a placement still reports its failure: a route still
// serving is asked afresh, and that copy places the start, rather than the read waiting on a copy
// that can deliver nothing more. It got a group, so its route did not refuse the track.
test("a failed copy holding a group back is still subscribed afresh", async () => {
const { a, active, close } = await setup();
const ta = copy(a);
hooks.receiveTrack(ta.producer);
ta.write(64n);
const sub = active.track("seq").subscribe({ delay: { max: null }, ranges: [{ start: { time: 66n } }] });
await settle();
expect(sub.tryRecvGroup()).toBeUndefined();

ta.producer.abort(new StreamError(StreamCode.Internal));
const again = copy(a);
hooks.receiveTrack(again.producer);
again.write(66n);
again.write(67n);
const read: bigint[] = [];
for (let attempt = 0; attempt < 10 && read.length < 2; attempt++) {
await settle();
for (let group = sub.tryRecvGroup(); group; group = sub.tryRecvGroup()) {
read.push(group.start);
group.close();
}
}
expect(read).toEqual([66n, 67n]);
expect(sub.closed.peek()).toBeUndefined();

sub.close();
close();
});

// As above, with the fresh copy failing too while it holds a group back: the read asks again.
test("failed copies holding groups back are each subscribed afresh", async () => {
const { a, active, close } = await setup();
const ta = copy(a);
hooks.receiveTrack(ta.producer);
ta.write(64n);
const sub = active.track("seq").subscribe({ delay: { max: null }, ranges: [{ start: { time: 66n } }] });
const read: bigint[] = [];
const poll = async () => {
for (let attempt = 0; attempt < 10 && read.length < 2; attempt++) {
await settle();
for (let group = sub.tryRecvGroup(); group; group = sub.tryRecvGroup()) {
read.push(group.start);
group.close();
}
}
};
await poll();
expect(read).toEqual([]);

ta.producer.abort(new StreamError(StreamCode.Internal));
const retry = copy(a);
hooks.receiveTrack(retry.producer);
retry.write(65n);
await poll();
expect(read).toEqual([]);

retry.producer.abort(new StreamError(StreamCode.Internal));
const again = copy(a);
hooks.receiveTrack(again.producer);
again.write(66n);
again.write(67n);
await poll();
expect(read).toEqual([66n, 67n]);
expect(sub.closed.peek()).toBeUndefined();

sub.close();
close();
});
73 changes: 66 additions & 7 deletions js/net/src/resume.ts
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ import { grace } from "./ledger.ts";
import { keyOf, type Lister, type Tracks, type TracksEvent } from "./listing.ts";
import * as track from "./track.ts";
import { max, min, UNBOUNDED } from "./util/key.ts";
import { clamped, type Resolved, sameRanges, type Window } from "./window.ts";
import { clamped, type Resolved, sameRanges, Window } from "./window.ts";
import { type Broadcast as BroadcastWire, registerWire, wireOf } from "./wire.ts";

/** A copy claiming the same content with different metadata: refused rather than reinterpreted. */
Expand Down Expand Up @@ -364,7 +364,10 @@ interface Copy {
since?: Instant;
/** Groups still read that came from this copy, which keep a replaced copy subscribed. */
lease: number;
/** It handed out a group, so a failure is its route failing mid-serve rather than refusing. */
/**
* It handed out a group, or got one it holds back for a placement, so a failure is its route
* failing mid-serve rather than refusing.
*/
delivered?: boolean;
}

Expand Down Expand Up @@ -657,6 +660,10 @@ class Reader implements ResumeSource {
const placed = this.#resolution;
for (const copy of this.#copies) {
if (copy.floor !== undefined || copy.until !== undefined) continue;
// A copy whose delivery failed places a time start past its last group as if nothing
// followed, but the front's next copy continues the track: only that copy knows which
// group holds it, until the track concludes. Mirrors the Rust `Reader::window`.
if (!this.route.concluded && copyFailed(copy)) continue;
Comment thread
kixelated marked this conversation as resolved.
Resolved
if (!copy.from || !sameRanges(copy.from, ranges)) continue;
const own = hooks.servingResolved(copy.sub);
placed.resolved = ranges.map((_, index) => placed.resolved[index] ?? own[index]);
Expand Down Expand Up @@ -819,7 +826,31 @@ class Reader implements ResumeSource {

#deliverable(copy: Copy, key: bigint, delivered: Delivered): boolean {
const end = copyEnd(copy);
return (end === undefined || key < end) && !delivered.has(key);
if ((end !== undefined && key >= end) || delivered.has(key)) return false;
// A failed copy resolved the reader's ranges as if nothing followed its last group: the
// placement a copy continuing the track made decides. Mirrors the Rust `Reader::window`.
if (this.route.concluded || !copyFailed(copy)) return true;
return this.#placement().some(
(range) => range !== undefined && key >= range.start && (range.end === undefined || key < range.end),
);
}

/**
* The reader's ranges as {@link #placed} resolves them now, or as they resolve with no copy at
* all: a range from the oldest group needs no copy to place it. Mirrors the Rust reader's
* window starting unheld.
*/
#placement(): (Resolved | undefined)[] {
const ranges = (this.#shell.subscription.peek() ?? track.Subscription.default()).ranges ?? [{}];
const placed = this.#placed(ranges);
const unheld = new Window(ranges, undefined).resolved();
return ranges.map((_, index) => placed[index] ?? unheld[index]);
}

// Whether a failed copy's next group waits for a copy continuing the track to place the
// reader's ranges, which decides whether it goes out. Mirrors the Rust `Reader::poll_lowest`.
#waits(copy: Copy): boolean {
return !this.route.concluded && copyFailed(copy) && this.#placement().includes(undefined);
}

tryGroup(ordered: boolean, delivered: Delivered, dispositions: boolean): Recv {
Expand Down Expand Up @@ -848,12 +879,22 @@ class Reader implements ResumeSource {
// A newer copy must not pass it and permanently discard it from sequence order.
if (ordered && recv.kind === "idle") return recv;
}
// The failed copies holding a group back for a placement, which still report their failure.
const held = new Set<Copy>();
// The serving copy first: it is the one that keeps delivering.
for (let index = this.#copies.length - 1; index >= 0; index--) {
const copy = this.#copies[index];
if (!this.#feeds(copy)) continue;
for (;;) {
const recv = take(copy, ordered, this.#newest === undefined ? undefined : this.#newest + 1n);
const newest = this.#newest === undefined ? undefined : this.#newest + 1n;
// Its group stays there until placed, but how it ran out is still judged below. It got
// a group, so its route did not refuse the track.
if (this.#waits(copy) && hooks.peekGroup(copy.sub, ordered, newest) !== undefined) {
copy.delivered = true;
held.add(copy);
break;
}
const recv = take(copy, ordered, newest);
if (recv.kind === "group") {
if (!this.#deliverable(copy, recv.group.start, delivered)) {
recv.group.close();
Expand All @@ -879,7 +920,11 @@ class Reader implements ResumeSource {
const dropped = this.#tryDrop(delivered);
if (dropped) return dropped;
}
return this.#end((copy) => (typeof copy.done === "number" ? null : copy.done));
return this.#end((copy) => {
if (typeof copy.done === "number") return null;
const failed = held.has(copy) ? copy.sub.closed.peek() : undefined;
return copy.done ?? (failed instanceof Error ? failed : undefined);
});
}

/**
Expand Down Expand Up @@ -1018,7 +1063,7 @@ class Reader implements ResumeSource {
ordered,
this.#newest === undefined ? undefined : this.#newest + 1n,
);
if (key === undefined) break;
if (key === undefined || this.#waits(copy)) break;
if (this.#deliverable(copy, key, delivered)) {
// A tie goes to the newer copy, so the serving one hands out its first group and
// bounds the copies it replaced.
Expand Down Expand Up @@ -1415,18 +1460,32 @@ class Reader implements ResumeSource {
return serving && hooks.holeEnd(serving.sub, key, dropped);
}

// A failed serving copy's own resolution is no placement while the front may still replace it:
// the reader's ranges stay as placed, pending until a copy continuing the track places them.
// Mirrors the Rust reader answering from its window.
#placesNothing(serving: Copy): boolean {
return !this.route.concluded && copyFailed(serving);
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
}

// Each copy asks for the reader's ranges in turn, clamped to its floor and replacement.
starts(from: bigint): (bigint | undefined)[] | undefined {
this.sync();
const serving = this.#serving();
return serving && hooks.servingStarts(serving.sub, from);
if (!serving) return undefined;
if (!this.#placesNothing(serving)) return hooks.servingStarts(serving.sub, from);
return this.#placement().map((range) => {
if (!range) return undefined;
const start = max(range.start, from);
return range.end === undefined || range.end > start ? start : undefined;
});
}

// Each copy resolves the reader's ranges in turn, clamped to its floor and replacement.
resolved(): (Resolved | undefined)[] | undefined {
this.sync();
const serving = this.#serving();
if (!serving) return undefined;
if (this.#placesNothing(serving)) return this.#placement();
// As the first copy to resolve them placed them, then as the serving copy does.
const ranges = (this.#shell.subscription.peek() ?? track.Subscription.default()).ranges ?? [{}];
const placed = this.#placed(ranges);
Expand Down
29 changes: 0 additions & 29 deletions quest/m0/test-flakes-3/track-replacement-bun.md

This file was deleted.

31 changes: 31 additions & 0 deletions quest/m1/waiting-time-start-reads.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# [M] A waiting time start reads its owed group, and a fill reaches subscriptions

## Goal

A Rust reader whose time start is still waiting on the groups held never hands
out a later group ahead of the one the start resolves to, and a group a
publisher accepts from a fill is delivered on its live subscriptions the way
`@moq/net` delivers it, or the difference is documented as intended.

## Plan

Both were seen while chasing the interop hang that
[moq-dev/moq#9127](https://github.com/moq-dev/moq/pull/9127) fixed, through a
lite-07 publisher, a relay, and two viewers, all in one simulated test:

- A viewer that fetched group 67, then subscribed from `Time(66)` before 66
was written, read 67 from an `Ordered` reader at once, then reported 66 as an
`Old` gap once 66 landed, though its budget was unlimited. Its own copy held
67 and placed nothing for the time, so the waiting span read from 66, and the
ordered read did not wait for the group the start would resolve to.
- With the fetch on the other viewer instead, the subscribing viewer got 66 and
never 67: neither the Rust publisher nor the relay served the fill-accepted
group on a subscription, while the `@moq/net` publisher serves one there
(the interop relay received 69 over its subscription in every passing run).

Start from a test in `rs/moq-net/tests/integration` with that topology
(`MockConnectOptions` on `moq-lite-07-wip`, the publisher's `Requests` serving
67 only). Decide the second from the draft's Subscribe section: a fill below
the live edge is history, but one at or past it is the live feed.

Public API: none expected. Wire: none.
Loading