diff --git a/doc/concept/moq-lite.md b/doc/concept/moq-lite.md index ca6ad8db3bd..a926879de9a 100644 --- a/doc/concept/moq-lite.md +++ b/doc/concept/moq-lite.md @@ -449,7 +449,8 @@ datagrams and `audio` as streams are two independent tracks. Nothing on the wire says which kind a name is: the application knows from its catalog or by convention, so a broadcast's track listing never includes a datagram track. In Rust, `broadcast.create_datagrams(name, info)` publishes one and -`broadcast.datagrams(name)` reads it. +`broadcast.datagrams(name)` reads it. In JavaScript use +`broadcast.createDatagrams(name, info)` and `broadcast.datagrams(name)`. A datagram subscription is live from when it starts: it has no ranges, order, or max delay, only a priority, which a subscriber can change. A datagram still diff --git a/doc/concept/standard.md b/doc/concept/standard.md index 7aaa1c31718..472e84e7dcb 100644 --- a/doc/concept/standard.md +++ b/doc/concept/standard.md @@ -36,7 +36,7 @@ subset drops. What a peer actually observes against this implementation: - **One `SETUP` credential**, forwarded to the [auth server](/bin/relay/auth#the-contract) unverified. A second `SETUP` token closes the session. We advertise no token cache, so an alias reference in `SETUP` closes it with `PROTOCOL_VIOLATION`, one on a request with `UNKNOWN_AUTH_TOKEN_ALIAS`, and a registration on a request with `AUTH_TOKEN_CACHE_OVERFLOW`. Where the Rust side authorizes the peer or verifies tokens, each distinct request token is verified once while any open request on that session names it, and its grant authorizes only requests naming it, even where that authorization already covers it; it never widens the session. It is refused with `NOT_SUPPORTED` when nothing verifies it, when it is empty, or when its Token Type is anything but 0 (out of band). A side that does neither, JavaScript included, ignores the token. Rust refuses a request carrying more than one distinct token with `NOT_SUPPORTED`, since it verifies one; JavaScript, which verifies none, ignores them all. A token-authorized `SUBSCRIBE` still stays within the grant the peer gave us. A token-authorized `PUBLISH_NAMESPACE` needs a grant covering everything beneath the namespace, and the tracks we subscribe under it rely on that announce instead of our limit, ending with it. An update on a token-authorized request moves it to the token it carries; the old token ends with its last naming request. A request is admitted by the session or by a token for good, so an update naming a token on one the session admitted is refused with `NOT_SUPPORTED`. A token-less `PUBLISH_NAMESPACE` our limit does not cover is refused `UNAUTHORIZED`, and one the limit stops covering ends; the publisher re-announces once it widens. Expiry or revocation ends every request naming that token. The relay reports the lease end to its auth server, and a later request naming the token verifies it again. A refused renewal ends only that request. Rust and JavaScript send the token a route names (`Route::auth`) on its `PUBLISH_NAMESPACE`, and swapping it renews the request with `REQUEST_UPDATE`; the inline `NAMESPACE` that answers a solicited `SUBSCRIBE_NAMESPACE` carries no token. A token added to a reconnecting connection's auth handle rides every session it makes, so the announcement keeps it across a reconnect; one added to a single session's handle ends with that session. Rust honors a token on any supported request; discovery reveals only paths it grants, and a joining `FETCH` keeps following its subscription. A request repeating a token is refused with `MALFORMED_AUTH_TOKEN`. The negotiated [MoQ Auth extension](/concept/moq-lite#authorization) accepts additional session tokens in band. - **Pause delivery.** `FORWARD=0` on a `SUBSCRIBE` or its update drops the subscription, closing its data streams and holding nothing upstream, while the request and its slot stay. `FORWARD=1` subscribes again as a new `SUBSCRIBE` with the same filter would, so nothing carries across the pause: its `REQUEST_OK` names the live edge it joined at. A paused update may change only priority and max age. - **Refused, not fatal.** A legal request this stack does not serve is rejected on its own and the session stays up: `SUBSCRIBE_TRACKS`, multi-group `FETCH`, and joining requests reaching before the joined group get `NOT_SUPPORTED`. Range Filters get `INVALID_FILTER` from draft 20 because the advertised filter limit is zero. From draft 20, namespaces whose first tuple field is exactly `.` or `.session` get `DOES_NOT_EXIST` before application lookup; other dot prefixes still reach the application. Subscription updates accept priority, negotiated subscriber max age, and delivery timeouts. Rust also carries range changes, keeps bounded subscriptions updatable, renews request tokens, and judges `NEW_GROUP_REQUEST` as a join (see dynamic groups below). From draft 20, a wider or historical fill fails on its separate fetch stream while its subscription stays open; a largest-group fill hands its head to the live subgroup tail. Other supported-but-unimplemented preferences end that subscription; a `TRACK_PROPERTY_FILTER` there closes the session with `PROTOCOL_VIOLATION`, since it belongs only to `SUBSCRIBE_TRACKS`. A namespace update may change its prefix; suffixes after the acknowledgement are relative to that new prefix. -- **Objects and datagrams.** Only subgroup 0 is accepted: a stream for any other subgroup, whether its header gives the ID or takes it from the first Object ID, is stopped and the session stays up. Rust relays preserve Object IDs, gaps, object properties, FIRST\_OBJECT, and END\_OF\_GROUP status objects between moq-transport peers for subgroup 0, including across a moq-lite-07 hop between two relays that negotiate the [ietf](/draft/moq-ietf) extension. Plain Rust readers and moq-lite peers without the extension see consecutive frames. Rust drops every OBJECT\_DATAGRAM, since a moq-lite track takes either streams or datagrams and Rust does not yet map a [datagram track](/concept/moq-lite#datagrams) onto moq-transport; JavaScript supports only normal datagrams at Object ID 0. A malformed datagram closes the session. A relayed Timestamp leaves in the units the downstream response declares, and is dropped where it declares none, as for a plain frame; an immutable one that would change or vanish is refused. On a timed track the frame holds each object's time, so a mutable Timestamp that only repeats the immutable one is not restated; a status object has no frame, so it keeps its Timestamp among its properties. Only property 0x10 is read as a Timestamp. Property 0x06 stays opaque and keeps its value regardless of timestamp units; it is dropped between draft 18 and later drafts, or refused when immutable, because draft 20 assigns it to SUBGROUP\_DELIVERY\_TIMEOUT. A status object carrying properties is refused. A `FETCH` has no status objects, so its response ends at the last frame. A group cannot resume across route changes by frame ordinal once Object IDs skip ahead of the ordinals on either route; recovery refuses it instead of splicing different objects. +- **Objects and datagrams.** Only subgroup 0 is accepted: a stream for any other subgroup, whether its header gives the ID or takes it from the first Object ID, is stopped and the session stays up. Rust relays preserve Object IDs, gaps, object properties, FIRST\_OBJECT, and END\_OF\_GROUP status objects between moq-transport peers for subgroup 0, including across a moq-lite-07 hop between two relays that negotiate the [ietf](/draft/moq-ietf) extension. Plain Rust readers and moq-lite peers without the extension see consecutive frames. Rust and JavaScript drop every OBJECT\_DATAGRAM until they map a [datagram track](/concept/moq-lite#datagrams) onto moq-transport. A malformed datagram closes the session. A relayed Timestamp leaves in the units the downstream response declares, and is dropped where it declares none, as for a plain frame; an immutable one that would change or vanish is refused. On a timed track the frame holds each object's time, so a mutable Timestamp that only repeats the immutable one is not restated; a status object has no frame, so it keeps its Timestamp among its properties. Only property 0x10 is read as a Timestamp. Property 0x06 stays opaque and keeps its value regardless of timestamp units; it is dropped between draft 18 and later drafts, or refused when immutable, because draft 20 assigns it to SUBGROUP\_DELIVERY\_TIMEOUT. A status object carrying properties is refused. A `FETCH` has no status objects, so its response ends at the last frame. A group cannot resume across route changes by frame ordinal once Object IDs skip ahead of the ordinals on either route; recovery refuses it instead of splicing different objects. - **Dynamic groups.** A track advertises `DYNAMIC_GROUPS` when its publisher sets a refresh ratio, and a track received with `DYNAMIC_GROUPS` gets a ratio of 0. A `NEW_GROUP_REQUEST` counts as one more subscriber joining, which a `SUBSCRIBE` already is: it asks for a new group only when the newest group's bytes after its first frame exceed the ratio times the first frame. Joins coalesce until the asked-for group arrives. A Rust relay sends `NEW_GROUP_REQUEST` upstream, on the `SUBSCRIBE` or in a `REQUEST_UPDATE`, once a join wants a group past the newest it holds; a relay holding nothing yet asks for none. On a track without a ratio, an update carrying one ends the subscription with `UPDATE_FAILED`. A moq-lite-07 hop (`moq-lite-07-wip`, opt-in) carries the same demand natively: the ratio in `TRACK_INFO` and the asked-for group in `SUBSCRIBE_UPDATE`. Earlier moq-lite versions carry neither, so a track received over one never refreshes. - **Priority.** Higher is served first. This stack's priorities run from -128 to 127 with 0 unset, and go out on the IETF wire, which ranks lower first, as `127 - priority`: an unset priority is IETF 127. An omitted subscriber or publisher priority reads as the drafts' default of IETF 128, one step below unset, so a relay forwards it unchanged. A relay's upstream subscription carries the highest priority its subscribers ask for and updates it as they come and go; its group order stays as the `SUBSCRIBE` asked, since no draft lets an update change it. - **Size.** An object extension block larger than 64 KiB ends that subgroup stream. The session stays up. This cap is ours, not the draft's. diff --git a/drafts/draft-lcurley-moq-flate.md b/drafts/draft-lcurley-moq-flate.md index 106119889d8..fafd8408985 100644 --- a/drafts/draft-lcurley-moq-flate.md +++ b/drafts/draft-lcurley-moq-flate.md @@ -53,7 +53,7 @@ Each subgroup is a separate DEFLATE stream: its object payloads share one window The application decides which subgroups a track uses; a consumer knows which to expect and decodes each one's objects in order. -A datagram carries a group of one object, so it is a stream of its own: compressed alone, starting cold. +Datagram tracks are never compressed. An object with no payload is skipped, neither advancing nor resetting the window. @@ -77,7 +77,7 @@ The compression level is a publisher's choice; any conformant stream decodes. # Declaring Compression {#declare} The transport declares a track compressed, and gives each object's inflated size beside its compressed one. -{{moql}} does both from moq-lite-07: a Compression codec id in TRACK_INFO, and an Inflated Size on each FRAME and datagram. +{{moql}} does both from moq-lite-07: a Compression codec id in TRACK_INFO, and an Inflated Size on each FRAME. A consumer MUST NOT infer compression from the payload: raw DEFLATE has no magic number, so a wrong guess yields plausible garbage. A consumer MUST treat an object that inflates to more or less than its declared size as malformed, and abandon the subgroup. @@ -109,7 +109,7 @@ Every Normal Object on a compressed track MUST carry it, including an empty payl A plain track or a status object MUST NOT carry it. It can appear in the mutable list or Immutable Properties, but MUST NOT appear more than once across them. The base Object Payload Length remains the size of the compressed bytes. -The same rules apply to subgroup streams, FETCH streams, and datagrams. +The same rules apply to subgroup streams and FETCH streams. A receiver MUST validate the declared size against the same resource limits as an uncompressed payload before admitting it, and MUST NOT allocate a buffer solely from that declaration. Incremental inflate output is charged to the receiver's cache budget and shared across its plain readers. @@ -150,9 +150,9 @@ It also requests these entries in the MOQ Properties registry, outside the appli ## Since draft-lcurley-moq-flate-00 (in progress) - Negotiated FLATE signaling adds COMPRESSION and INFLATED_SIZE properties to MoQ Transport draft-18 and later; sealed declarations are forwarded unchanged or refused when incompatible. -- Compression is declared by the transport instead of the application: {{moql}} carries a Compression codec in TRACK_INFO and each object's Inflated Size on FRAME and datagrams. +- Compression is declared by the transport instead of the application: {{moql}} carries a Compression codec in TRACK_INFO and each object's Inflated Size on FRAME. - A relay inflates for a peer whose transport does not declare compression. -- A datagram is compressed alone, as a stream of its own. +- Datagram tracks are not compressed and carry no Inflated Size. - A consumer rejects an object that inflates to other than its declared size, or that ends in a final block. # Acknowledgments diff --git a/js/crypto/MIGRATE.md b/js/crypto/MIGRATE.md new file mode 100644 index 00000000000..9836a6b42ab --- /dev/null +++ b/js/crypto/MIGRATE.md @@ -0,0 +1,14 @@ +# Migrating @moq/crypto + +## Unreleased + +### Protected datagrams use their own tracks ([#7846](https://github.com/moq-dev/moq/pull/7846)) + +Call `E2EE.Broadcast.Producer.createDatagrams(semantic, info)` and +`E2EE.Broadcast.Consumer.datagrams(semantic)`. Their `E2EE.Datagram.Producer` and +`Consumer` use `append`, `insert`, and `recv` in place of the stream track's +`appendDatagram`, `insertDatagram`, and `recvDatagram`. A datagram's sequence is +independent of same-name groups; the existing separate cryptographic key domains +keep their nonces distinct. Stream authentication failures and lifecycle operations +no longer end the datagram track. Finish or abort its producer, and close its consumer, +independently. diff --git a/js/crypto/src/e2ee/broadcast.ts b/js/crypto/src/e2ee/broadcast.ts index db64a175eb5..d1c86bec62f 100644 --- a/js/crypto/src/e2ee/broadcast.ts +++ b/js/crypto/src/e2ee/broadcast.ts @@ -7,7 +7,8 @@ * * @module */ -import type { Broadcast as NetBroadcast, Track as NetTrack } from "@moq/net"; +import type { Broadcast as NetBroadcast, Datagram as NetDatagram, Track as NetTrack } from "@moq/net"; +import type * as Datagram from "./datagram.ts"; import type * as Epoch from "./epoch.ts"; import type * as Generation from "./generation.ts"; import type * as Track from "./track.ts"; @@ -47,6 +48,12 @@ export class Producer { return this.generation.produce(this.#inner.createTrack(name, info), semantic); } + /** Create a protected datagram track in its own namespace and sequence space. */ + async createDatagrams(semantic: string, info: Partial = {}): Promise { + const name = await this.generation.name(semantic); + return this.generation.produceDatagrams(this.#inner.createDatagrams(name, info), semantic); + } + /** End the wrapped broadcast, as `@moq/net`'s `close` does. Idempotent. */ close(): void { this.#inner.close(); @@ -80,6 +87,12 @@ export class Consumer { return this.generation.consume(this.#inner.track(name).subscribe(subscription), semantic); } + /** Read the protected datagram track under its semantic name, live from when that name resolves to its physical one. */ + async datagrams(semantic: string): Promise { + const name = await this.generation.name(semantic); + return this.generation.consumeDatagrams(this.#inner.datagrams(name), semantic); + } + /** Release the wrapped broadcast handle, as `@moq/net`'s `close` does. Idempotent. */ close(): void { this.#inner.close(); diff --git a/js/crypto/src/e2ee/datagram.ts b/js/crypto/src/e2ee/datagram.ts index 0cad9698911..49e37bad909 100644 --- a/js/crypto/src/e2ee/datagram.ts +++ b/js/crypto/src/e2ee/datagram.ts @@ -3,11 +3,17 @@ * * @module */ -import type { Time } from "@moq/net"; +import { Datagram as NetDatagram, type Time } from "@moq/net"; +import { Pump } from "../pump.ts"; +import { checkGroup } from "./derive.ts"; +import { Failure } from "./failure.ts"; +import type { TrackKey } from "./key.ts"; +import { MAX_DATAGRAM_PAYLOAD } from "./limits.ts"; +import { DatagramWindow } from "./window.ts"; /** A decrypted datagram. */ export interface Datagram { - /** Per-track sequence, shared with the group namespace. */ + /** The datagram track's own sequence. */ sequence: bigint; /** Presentation timestamp, or `undefined` when untimed. Not authenticated. */ timestamp?: Time.Timestamp; @@ -30,3 +36,138 @@ export type Event = | { kind: "duplicate"; sequence: bigint } | { kind: "identity"; sequence: bigint } | { kind: "oversize"; sequence: bigint }; + +/** Encrypts a datagram track under its name's datagram key and independent sequence space. */ +export class Producer { + /** The physical track name. */ + readonly name: string; + #inner: NetDatagram.Producer; + #info: NetDatagram.Info; + #key: TrackKey; + #pump = new Pump(); + #next = 0n; + #finished = false; + #aborted?: Error; + + private constructor(inner: NetDatagram.Producer, info: NetDatagram.Info, key: TrackKey) { + this.#inner = inner; + this.#info = info; + this.#key = key; + this.name = inner.name; + } + /** @internal */ + static make(inner: NetDatagram.Producer, info: NetDatagram.Info, key: TrackKey): Producer { + return new Producer(inner, info, key); + } + /** The highest priority among the datagram track's subscribers. */ + get priority(): NetDatagram.Producer["priority"] { + return this.#inner.priority; + } + /** The datagram track's terminal result. */ + get closed(): NetDatagram.Producer["closed"] { + return this.#inner.closed; + } + /** Encrypt a payload at the next datagram sequence, returning that sequence. */ + async append(timestamp: Time.Timestamp | undefined, plaintext: Uint8Array): Promise { + const sequence = this.#next; + await this.insert(sequence, timestamp, plaintext); + return sequence; + } + /** Encrypt at an explicit unused sequence, refusing predictable failures before allocating it. */ + async insert(sequence: bigint, timestamp: Time.Timestamp | undefined, plaintext: Uint8Array): Promise { + if (this.#finished || this.#aborted) throw this.#aborted ?? new Error("datagram track is closed"); + checkGroup(sequence); + if (sequence < this.#next) throw new Failure("reuse"); + const at = NetDatagram.rescale(this.#info, timestamp); + const iv = this.#key.admitSeal(sequence, 0, plaintext.byteLength, MAX_DATAGRAM_PAYLOAD); + this.#next = sequence + 1n; + const owned = plaintext.slice(); + return this.#pump.run(async () => { + if (this.#aborted) throw this.#aborted; + const payload = await this.#key.encrypt(iv, owned); + if (this.#aborted) throw this.#aborted; + this.#inner.insert({ sequence, timestamp: at, payload }); + }); + } + /** Finish after every queued write, refusing new writes immediately. */ + finish(): void { + if (this.#finished || this.#aborted) return; + this.#finished = true; + void this.#pump.run(() => { + if (!this.#aborted) this.#inner.finish(); + }); + } + /** Abort immediately, rejecting queued writes. */ + abort(error: Error): void { + if (this.#aborted) return; + this.#aborted = error; + this.#inner.abort(error); + } +} + +/** Decrypts a datagram track, reporting malformed and replayed datagrams as events. */ +export class Consumer { + /** The physical track name. */ + readonly name: string; + #inner: NetDatagram.Consumer; + #key: TrackKey; + #window = new DatagramWindow(); + #pump = new Pump(); + #reads: Promise = Promise.resolve(); + #closed = false; + + private constructor(inner: NetDatagram.Consumer, key: TrackKey) { + this.#inner = inner; + this.#key = key; + this.name = inner.name; + } + /** @internal */ + static make(inner: NetDatagram.Consumer, key: TrackKey): Consumer { + return new Consumer(inner, key); + } + /** This consumer's subscriber priority. */ + get priority(): number { + return this.#inner.priority; + } + /** The datagram track's publisher properties, once its request is answered. */ + info(): Promise { + return this.#inner.info(); + } + /** Change this consumer's priority, higher first. */ + setPriority(priority: number): void { + this.#inner.setPriority(priority); + } + /** Read the next datagram event, or undefined after a clean finish. */ + recv(): Promise { + const result = this.#reads.then(() => this.#recv()); + this.#reads = result.catch(() => {}); + return result; + } + async #recv(): Promise { + if (this.#closed) return; + const datagram = await this.#inner.recv(); + if (!datagram || this.#closed) return; + const { sequence, timestamp, payload } = datagram; + if (this.#window.isDuplicate(sequence)) return { kind: "duplicate", sequence }; + let plaintext: Uint8Array; + try { + const iv = this.#key.admitOpen(sequence, 0, payload.byteLength, MAX_DATAGRAM_PAYLOAD); + plaintext = await this.#pump.run(() => this.#key.decrypt(iv, payload)); + } catch (err) { + if (this.#closed) return; + if (!(err instanceof Failure)) throw err; + const kind = err.code; + if (kind === "authentication" || kind === "identity" || kind === "oversize") return { kind, sequence }; + throw err; + } + if (this.#closed) return; + this.#window.mark(sequence); + return { kind: "datagram", datagram: { sequence, timestamp, plaintext } }; + } + /** Release this consumer's demand and wake its pending reads. */ + close(): void { + if (this.#closed) return; + this.#closed = true; + this.#inner.close(); + } +} diff --git a/js/crypto/src/e2ee/e2ee.test.ts b/js/crypto/src/e2ee/e2ee.test.ts index 14f17fd6c36..24e601f0147 100644 --- a/js/crypto/src/e2ee/e2ee.test.ts +++ b/js/crypto/src/e2ee/e2ee.test.ts @@ -1,6 +1,14 @@ import { expect, spyOn, test } from "bun:test"; import { Milli } from "@moq/clock"; -import { Broadcast, Error as NetError, Group as NetGroup, Track as NetTrack, StreamCode, Time } from "@moq/net"; +import { + Broadcast, + Datagram as NetDatagram, + Error as NetError, + Group as NetGroup, + Track as NetTrack, + StreamCode, + Time, +} from "@moq/net"; import { Pump } from "../pump.ts"; import * as Protected from "./broadcast.ts"; import { Credential } from "./credential.ts"; @@ -50,6 +58,7 @@ interface Pair { generation: Generation.Producer; consuming: Generation.Consumer; net: NetTrack.Producer; + dg: { net: NetDatagram.Producer; producer: Datagram.Producer; consumer: Datagram.Consumer }; producer: Track.Producer; consumer: Track.Consumer; } @@ -60,7 +69,13 @@ async function pair(semantic: string): Promise { const producer = await generation.produce(net, semantic); const consuming = generation.credential.consume(generation.epoch); const consumer = consuming.consume(subscribe(net), semantic); - return { semantic, generation, consuming, net, producer, consumer }; + const raw = new NetDatagram.Producer(net.name).accept({ timescale: Time.Timescale.MILLI }); + const dg = { + net: raw, + producer: await generation.produceDatagrams(raw, semantic), + consumer: consuming.consumeDatagrams(raw.consume(), semantic), + }; + return { semantic, generation, consuming, net, producer, consumer, dg }; } // Ciphertext written straight to the net track, as a relay or attacker would. @@ -186,8 +201,8 @@ test("grouped roundtrip", async () => { test("datagram roundtrip", async () => { const p = await pair("audio"); - expect(await p.producer.appendDatagram(ms(5), text("opus"))).toBe(0n); - const event = await p.consumer.recvDatagram(); + expect(await p.dg.producer.append(ms(5), text("opus"))).toBe(0n); + const event = await p.dg.consumer.recv(); expect(event?.kind).toBe("datagram"); if (event?.kind !== "datagram") return; expect(event.datagram.sequence).toBe(0n); @@ -197,31 +212,29 @@ test("datagram roundtrip", async () => { test("sequences allocate monotonically", async () => { const p = await pair("video"); - const subscriber = subscribe(p.net); + const subscriber = p.dg.net.consume(); expect(p.producer.appendGroup().sequence).toBe(0n); expect(p.producer.createGroup(5n).sequence).toBe(5n); - expect(await p.producer.appendDatagram(ms(1), text("x"))).toBe(6n); - expect(p.producer.appendGroup().sequence).toBe(7n); - expect(thrown(() => p.producer.createGroup(7n))).toBe("reuse"); + expect(await p.dg.producer.append(ms(1), text("x"))).toBe(0n); + expect(p.producer.appendGroup().sequence).toBe(6n); + expect(thrown(() => p.producer.createGroup(6n))).toBe("reuse"); expect(thrown(() => p.producer.createGroup(3n))).toBe("reuse"); - expect(await failure(p.producer.insertDatagram(6n, ms(1), text("x")))).toBe("reuse"); + expect(await failure(p.dg.producer.insert(0n, ms(1), text("x")))).toBe("reuse"); expect(thrown(() => p.producer.createGroup(BigInt(MAX_U53) + 1n))).toBe("identity"); expect(thrown(() => p.producer.createGroup(-1n))).toBe("identity"); // An untyped caller's fractional sequence is refused as an identity too. expect(thrown(() => p.producer.createGroup(1.5 as unknown as bigint))).toBe("identity"); // Only the one admitted datagram reached the net track: a later sentinel is next. - expect((await subscriber.recvDatagram())?.sequence).toBe(6n); - p.net.insertDatagram(8n, ms(2), new Uint8Array()); - expect((await subscriber.recvDatagram())?.sequence).toBe(8n); + expect((await subscriber.recv())?.sequence).toBe(0n); + p.dg.net.insert({ sequence: 8n, timestamp: ms(2), payload: new Uint8Array() }); + expect((await subscriber.recv())?.sequence).toBe(8n); }); test("oversize payloads are refused before an identity is spent", async () => { const p = await pair("audio"); - expect(await failure(p.producer.appendDatagram(ms(1), new Uint8Array(MAX_DATAGRAM_PLAINTEXT + 1)))).toBe( - "oversize", - ); - expect(await p.producer.appendDatagram(ms(1), new Uint8Array(MAX_DATAGRAM_PLAINTEXT))).toBe(0n); + expect(await failure(p.dg.producer.append(ms(1), new Uint8Array(MAX_DATAGRAM_PLAINTEXT + 1)))).toBe("oversize"); + expect(await p.dg.producer.append(ms(1), new Uint8Array(MAX_DATAGRAM_PLAINTEXT))).toBe(0n); const group = p.producer.appendGroup(); expect(await failure(group.writeFrame(ms(1), new Uint8Array(MAX_GROUPED_PLAINTEXT + 1)))).toBe("oversize"); @@ -415,7 +428,8 @@ test("grouped authentication failure ends the track", async () => { const forged = await recvGroup(p.consumer); expect(await failure(forged.readFrame())).toBe("authentication"); expect(await failure(p.consumer.recvGroup())).toBe("authentication"); - expect(await failure(p.consumer.recvDatagram())).toBe("authentication"); + await p.dg.producer.append(ms(3), text("independent")); + expect(datagramText(await p.dg.consumer.recv())).toBe("independent"); }); test("grouped failure wakes pending reads and releases demand", async () => { @@ -438,12 +452,13 @@ test("grouped failure wakes pending reads and releases demand", async () => { // Sibling frame, next group, and datagram reads have nothing to return until the failure. const frame = good.readFrame(); const nextGroup = p.consumer.recvGroup(); - const datagram = p.consumer.recvDatagram(); + const datagram = p.dg.consumer.recv(); expect(await failure(forged.readFrame())).toBe("authentication"); expect(await failure(frame)).toBe("authentication"); expect(await failure(nextGroup)).toBe("authentication"); - expect(await failure(datagram)).toBe("authentication"); + await p.dg.producer.append(ms(3), text("independent")); + expect(datagramText(await datagram)).toBe("independent"); expect(demand.wanted.peek()).toBeUndefined(); expect(badDemand.wanted.peek()).toBe(false); @@ -451,11 +466,11 @@ test("grouped failure wakes pending reads and releases demand", async () => { expect(p.net.closed.peek()).toBeUndefined(); }); -test("a grouped failure ahead of a datagram open rejects the datagram", async () => { +test("a grouped failure leaves the independent datagram track readable", async () => { const p = await pair("audio"); const bad = p.net.createGroup(0n); bad.writeFrame({ timestamp: ms(1), payload: await sealed(p, GROUP, 0n, 0, "nope", true) }); - p.net.insertDatagram(0n, ms(2), await sealed(p, DATAGRAM, 0n, 0, "late")); + p.dg.net.insert({ sequence: 0n, timestamp: ms(2), payload: await sealed(p, DATAGRAM, 0n, 0, "late") }); const forged = await recvGroup(p.consumer); const hold = holdFirstOpen(); @@ -464,10 +479,11 @@ test("a grouped failure ahead of a datagram open rejects the datagram", async () const read = forged.readFrame(); await hold.opening; // The datagram passes its terminal check, then queues behind the forged open. - const datagram = p.consumer.recvDatagram(); + const datagram = p.dg.consumer.recv(); await admit.done; hold.release(); - expect(await Promise.all([failure(read), failure(datagram)])).toEqual(["authentication", "authentication"]); + expect(await failure(read)).toBe("authentication"); + expect(datagramText(await datagram)).toBe("late"); } finally { hold.restore(); admit.restore(); @@ -523,18 +539,18 @@ test("a bad datagram stays an event and keeps demand", async () => { const demand = p.net.demand(); const open = p.producer.appendGroup(); await open.writeFrame(ms(1), text("ok")); - p.net.insertDatagram(1n, ms(2), await sealed(p, DATAGRAM, 1n, 0, "nope", true)); + p.dg.net.insert({ sequence: 1n, timestamp: ms(2), payload: await sealed(p, DATAGRAM, 1n, 0, "nope", true) }); const good = await recvGroup(p.consumer); expect(await readText(good)).toBe("ok"); // The parked grouped read survives the bad datagram: it later resolves with a newer frame. const frame = good.readFrame(); - expect(await p.consumer.recvDatagram()).toEqual({ kind: "authentication", sequence: 1n }); + expect(await p.dg.consumer.recv()).toEqual({ kind: "authentication", sequence: 1n }); expect(demand.wanted.peek()).toBeDefined(); - await p.producer.insertDatagram(2n, ms(3), text("next")); - expect(datagramText(await p.consumer.recvDatagram())).toBe("next"); + await p.dg.producer.insert(2n, ms(3), text("next")); + expect(datagramText(await p.dg.consumer.recv())).toBe("next"); expect(demand.wanted.peek()).toBeDefined(); await open.writeFrame(ms(4), text("still")); @@ -543,31 +559,31 @@ test("a bad datagram stays an event and keeps demand", async () => { test("a forged datagram does not burn the identity", async () => { const p = await pair("audio"); - await p.producer.appendDatagram(ms(1), text("one")); - p.net.insertDatagram(1n, ms(2), await sealed(p, DATAGRAM, 1n, 0, "two", true)); - await p.producer.insertDatagram(1n, ms(2), text("two")); + await p.dg.producer.append(ms(1), text("one")); + p.dg.net.insert({ sequence: 1n, timestamp: ms(2), payload: await sealed(p, DATAGRAM, 1n, 0, "two", true) }); + await p.dg.producer.insert(1n, ms(2), text("two")); - expect(datagramText(await p.consumer.recvDatagram())).toBe("one"); - expect(await p.consumer.recvDatagram()).toEqual({ kind: "authentication", sequence: 1n }); - const event = await p.consumer.recvDatagram(); + expect(datagramText(await p.dg.consumer.recv())).toBe("one"); + expect(await p.dg.consumer.recv()).toEqual({ kind: "authentication", sequence: 1n }); + const event = await p.dg.consumer.recv(); expect(datagramText(event)).toBe("two"); }); test("a datagram no publisher could seal is dropped with its code", async () => { const p = await pair("audio"); const sealedAt = await sealed(p, DATAGRAM, 0n, 0, "x"); - p.net.insertDatagram(1n, ms(1), new Uint8Array(MAX_DATAGRAM_PAYLOAD + 1)); - p.net.insertDatagram(2n, ms(2), sealedAt.subarray(0, 3)); - p.net.insertDatagram(2n ** 53n, ms(3), sealedAt); - p.net.insertDatagram(2n ** 53n + 2n, ms(3), sealedAt.subarray(0, 3)); - await p.producer.insertDatagram(2n ** 53n - 1n, ms(4), text("last")); - - expect(await p.consumer.recvDatagram()).toEqual({ kind: "oversize", sequence: 1n }); - expect(await p.consumer.recvDatagram()).toEqual({ kind: "oversize", sequence: 2n }); - expect(await p.consumer.recvDatagram()).toEqual({ kind: "identity", sequence: 2n ** 53n }); + p.dg.net.insert({ sequence: 1n, timestamp: ms(1), payload: new Uint8Array(MAX_DATAGRAM_PAYLOAD + 1) }); + p.dg.net.insert({ sequence: 2n, timestamp: ms(2), payload: sealedAt.subarray(0, 3) }); + p.dg.net.insert({ sequence: 2n ** 53n, timestamp: ms(3), payload: sealedAt }); + p.dg.net.insert({ sequence: 2n ** 53n + 2n, timestamp: ms(3), payload: sealedAt.subarray(0, 3) }); + await p.dg.producer.insert(2n ** 53n - 1n, ms(4), text("last")); + + expect(await p.dg.consumer.recv()).toEqual({ kind: "oversize", sequence: 1n }); + expect(await p.dg.consumer.recv()).toEqual({ kind: "oversize", sequence: 2n }); + expect(await p.dg.consumer.recv()).toEqual({ kind: "identity", sequence: 2n ** 53n }); // Identity is checked before size, as in every core. - expect(await p.consumer.recvDatagram()).toEqual({ kind: "identity", sequence: 2n ** 53n + 2n }); - expect(datagramText(await p.consumer.recvDatagram())).toBe("last"); + expect(await p.dg.consumer.recv()).toEqual({ kind: "identity", sequence: 2n ** 53n + 2n }); + expect(datagramText(await p.dg.consumer.recv())).toBe("last"); }); test("resubscribing shares a key's budget", async () => { @@ -579,14 +595,14 @@ test("resubscribing shares a key's budget", async () => { return admitOpen.apply(this, args); }; try { - await p.producer.appendDatagram(ms(1), text("one")); - expect(datagramText(await p.consumer.recvDatagram())).toBe("one"); + await p.dg.producer.append(ms(1), text("one")); + expect(datagramText(await p.dg.consumer.recv())).toBe("one"); p.consumer.close(); opened[0].invocations = MAX_INVOCATIONS; - const again = p.consuming.consume(subscribe(p.net), p.semantic); - await p.producer.appendDatagram(ms(2), text("two")); - expect(await failure(again.recvDatagram())).toBe("exhausted"); + const again = p.consuming.consumeDatagrams(p.dg.net.consume(), p.semantic); + await p.dg.producer.append(ms(2), text("two")); + expect(await failure(again.recv())).toBe("exhausted"); } finally { TrackKey.prototype.admitOpen = admitOpen; } @@ -595,18 +611,22 @@ test("resubscribing shares a key's budget", async () => { test("duplicate datagrams are suppressed inside a bounded window", async () => { const p = await pair("audio"); const replay = await sealed(p, DATAGRAM, 0n, 0, "once"); - p.net.insertDatagram(0n, ms(1), replay); - p.net.insertDatagram(0n, ms(1), replay); - expect(datagramText(await p.consumer.recvDatagram())).toBe("once"); - expect(await p.consumer.recvDatagram()).toEqual({ kind: "duplicate", sequence: 0n }); + p.dg.net.insert({ sequence: 0n, timestamp: ms(1), payload: replay }); + p.dg.net.insert({ sequence: 0n, timestamp: ms(1), payload: replay }); + expect(datagramText(await p.dg.consumer.recv())).toBe("once"); + expect(await p.dg.consumer.recv()).toEqual({ kind: "duplicate", sequence: 0n }); // Once the window slides past it, the same ciphertext opens again. for (let sequence = 1n; sequence <= BigInt(DATAGRAM_WINDOW); sequence++) { - p.net.insertDatagram(sequence, ms(Number(sequence)), await sealed(p, DATAGRAM, sequence, 0, "x")); - expect((await p.consumer.recvDatagram())?.kind).toBe("datagram"); + p.dg.net.insert({ + sequence, + timestamp: ms(Number(sequence)), + payload: await sealed(p, DATAGRAM, sequence, 0, "x"), + }); + expect((await p.dg.consumer.recv())?.kind).toBe("datagram"); } - p.net.insertDatagram(0n, ms(1), replay); - expect(datagramText(await p.consumer.recvDatagram())).toBe("once"); + p.dg.net.insert({ sequence: 0n, timestamp: ms(1), payload: replay }); + expect(datagramText(await p.dg.consumer.recv())).toBe("once"); }); test("releasing an unfinished producer cancels queued writes and held groups", async () => { @@ -616,7 +636,7 @@ test("releasing an unfinished producer cancels queued writes and held groups", a const received = await recvGroup(p.consumer); expect(await readText(received)).toBe("first"); const tail = received.readFrame().catch((error) => error); - const datagram = p.producer.appendDatagram(ms(2), text("queued")); + const datagram = group.writeFrame(ms(2), text("queued")); const pending = Promise.allSettled([datagram]); p.producer.close(); const [result] = await pending; @@ -660,7 +680,6 @@ test("a declared track end leaves existing groups writable and abortable", async expect(await readText(received)).toBe("last"); expect(p.producer.closed.peek()).toBeUndefined(); expect(() => p.producer.appendGroup()).toThrow("track is closed"); - await expect(p.producer.appendDatagram(ms(3), text("late"))).rejects.toThrow("track is closed"); const aborted = new Error("unfinished group"); p.producer.abort(aborted); expect(await p.producer.closed).toBe(aborted); @@ -690,7 +709,7 @@ test("abort after declaring an end stops an in-flight seal and its queued writes try { const frame = group.writeFrame(ms(1), text("in flight")); await sealing; - const datagram = p.producer.appendDatagram(ms(2), text("queued")); + const datagram = group.writeFrame(ms(2), text("queued")); const pending = Promise.allSettled([frame, datagram]); p.producer.finish(); expect(p.producer.closed.peek()).toBeUndefined(); @@ -717,18 +736,18 @@ test("the pump keeps frame ordinals in submit order", async () => { const datagrams: Promise[] = []; for (let i = 0; i < 200; i++) { writes.push(groups[i % 2].writeFrame(ms(i), text(`frame-${i}`))); - if (i % 50 === 0) datagrams.push(p.producer.appendDatagram(ms(i), text(`datagram-${i}`))); + if (i % 50 === 0) datagrams.push(p.dg.producer.append(ms(i), text(`datagram-${i}`))); } for (const group of groups) group.finish(); await Promise.all(writes); - expect(await Promise.all(datagrams)).toEqual([2n, 3n, 4n, 5n]); + expect(await Promise.all(datagrams)).toEqual([0n, 1n, 2n, 3n]); for (let g = 0; g < 2; g++) { const received = await recvGroup(p.consumer); for (let i = g; i < 200; i += 2) expect(await readText(received)).toBe(`frame-${i}`); expect(await received.readFrame()).toBeUndefined(); } - for (let i = 0; i < 200; i += 50) expect(datagramText(await p.consumer.recvDatagram())).toBe(`datagram-${i}`); + for (let i = 0; i < 200; i += 50) expect(datagramText(await p.dg.consumer.recv())).toBe(`datagram-${i}`); }); test("concurrent reads open in transport order", async () => { @@ -772,7 +791,7 @@ test("aborting a track rejects pending writes on it and its groups", async () => const p = await pair("video"); const group = p.producer.appendGroup(); const frame = group.writeFrame(ms(1), text("a")); - const datagram = p.producer.appendDatagram(ms(1), text("d")); + const datagram = group.writeFrame(ms(2), text("queued")); const abort = new Error("cancelled"); p.producer.abort(abort); expect(await Promise.allSettled([frame, datagram])).toEqual([ @@ -793,8 +812,8 @@ test("an abort stops queued datagrams before they seal", async () => { }; try { const abort = new Error("abort"); - const write = p.producer.insertDatagram(0n, ms(1), text("x")); - p.producer.abort(abort); + const write = p.dg.producer.insert(0n, ms(1), text("x")); + p.dg.producer.abort(abort); await expect(write).rejects.toBe(abort); expect(seals).toBe(0); } finally { @@ -808,7 +827,7 @@ test("a finished group and a released declared track keep their promised drain", const frame = group.writeFrame(ms(1), text("frame")); group.finish(); group.abort(new Error("late")); - const datagram = p.producer.insertDatagram(1n, ms(2), text("datagram")); + const datagram = p.dg.producer.insert(1n, ms(2), text("datagram")); p.producer.finish(); p.producer.close(); p.producer.abort(new Error("late")); @@ -818,7 +837,7 @@ test("a finished group and a released declared track keep their promised drain", const received = await recvGroup(p.consumer); expect(await readText(received)).toBe("frame"); expect(await received.readFrame()).toBeUndefined(); - expect(datagramText(await p.consumer.recvDatagram())).toBe("datagram"); + expect(datagramText(await p.dg.consumer.recv())).toBe("datagram"); }); test("a refused timestamp does not burn the frame", async () => { @@ -840,13 +859,10 @@ test("produce waits for an unaccepted track's timescale", async () => { const producer = await producing; // The timescale is known from the start, so mismatches spend neither a sequence nor a frame. - await expect(producer.insertDatagram(0n, undefined, text("untimed"))).rejects.toThrow("timestamp"); - await producer.insertDatagram(0n, ms(1), text("datagram")); const group = producer.createGroup(1n); await expect(group.writeFrame(undefined, text("untimed"))).rejects.toThrow("timestamp"); await group.writeFrame(ms(2), text("frame")); group.finish(); - expect(datagramText(await consumer.recvDatagram())).toBe("datagram"); expect(await readText(await recvGroup(consumer))).toBe("frame"); // A track closed before it is accepted never produces, and leaves its name unclaimed. @@ -868,13 +884,10 @@ test("a timestamp out of range at the track's timescale spends nothing", async ( const producer = await generation.produce(net, "video"); const huge = new Time.Timestamp(Number.MAX_SAFE_INTEGER, Time.Timescale.MILLI); - await expect(producer.insertDatagram(0n, huge, text("x"))).rejects.toBeInstanceOf(NetError.TimestampMismatch); - await producer.insertDatagram(0n, ms(1), text("datagram")); const group = producer.createGroup(1n); await expect(group.writeFrame(huge, text("x"))).rejects.toBeInstanceOf(NetError.TimestampMismatch); await group.writeFrame(ms(2), text("frame")); group.finish(); - expect(datagramText(await consumer.recvDatagram())).toBe("datagram"); expect(await readText(await recvGroup(consumer))).toBe("frame"); }); @@ -884,17 +897,17 @@ test("a grouped failure skips opens queued behind it", async () => { bad.writeFrame({ timestamp: ms(1), payload: await sealed(p, GROUP, 0n, 0, "nope", true) }); const good = p.net.createGroup(1n); good.writeFrame({ timestamp: ms(2), payload: await sealed(p, GROUP, 1n, 0, "ok") }); - p.net.insertDatagram(0n, ms(3), await sealed(p, DATAGRAM, 0n, 0, "late")); + p.dg.net.insert({ sequence: 0n, timestamp: ms(3), payload: await sealed(p, DATAGRAM, 0n, 0, "late") }); const forged = await recvGroup(p.consumer); const held = await recvGroup(p.consumer); const hold = holdFirstOpen(); - const admit = admitted(3); + const admit = admitted(2); try { - const reads = [forged.readFrame(), held.readFrame(), p.consumer.recvDatagram()]; + const reads = [forged.readFrame(), held.readFrame()]; await admit.done; hold.release(); - expect(await Promise.all(reads.map(failure))).toEqual(["authentication", "authentication", "authentication"]); + expect(await Promise.all(reads.map(failure))).toEqual(["authentication", "authentication"]); expect(hold.calls).toBe(1); } finally { hold.restore(); diff --git a/js/crypto/src/e2ee/generation-internal.ts b/js/crypto/src/e2ee/generation-internal.ts index aad83278fc0..f8a6936ddd6 100644 --- a/js/crypto/src/e2ee/generation-internal.ts +++ b/js/crypto/src/e2ee/generation-internal.ts @@ -8,8 +8,9 @@ * * @module */ -import type { Track as NetTrack } from "@moq/net"; +import type { Datagram as NetDatagram, Track as NetTrack } from "@moq/net"; import type { Credential } from "./credential.ts"; +import * as Datagram from "./datagram.ts"; import { DATAGRAM, GROUP, type Name, pad } from "./derive.ts"; import * as Epoch from "./epoch.ts"; import { Failure } from "./failure.ts"; @@ -81,7 +82,16 @@ export class Consumer { shared = keys(this.credential, this.epoch, semantic); this.#keys.set(semantic, shared); } - return Track.Consumer.make(track, shared); + return Track.Consumer.make(track, shared.group); + } + /** Decrypt a net datagram track under the semantic name's datagram key. @internal */ + consumeDatagrams(track: NetDatagram.Consumer, semantic: string): Datagram.Consumer { + let shared = this.#keys.get(semantic); + if (!shared) { + shared = keys(this.credential, this.epoch, semantic); + this.#keys.set(semantic, shared); + } + return Datagram.Consumer.make(track, shared.datagram); } } @@ -99,6 +109,7 @@ export class Producer { #consumer: Consumer; // Keyed by semantic name, and so by physical name. #claims = new Set(); + #datagramClaims = new Set(); private constructor(credential: Credential) { this.credential = credential; @@ -157,6 +168,25 @@ export class Producer { this.#claims.delete(semantic); throw err; } - return Track.Producer.make(track, info, keys(this.credential, this.epoch, semantic)); + return Track.Producer.make(track, info, keys(this.credential, this.epoch, semantic).group); + } + /** Protect a net datagram track, claiming its semantic name in the datagram namespace. @internal */ + async produceDatagrams(track: NetDatagram.Producer, semantic: string): Promise { + try { + pad(semantic); + if (this.#datagramClaims.has(semantic)) throw new Failure("reuse"); + } catch (err) { + track.abort(err as Failure); + throw err; + } + this.#datagramClaims.add(semantic); + let info: NetDatagram.Info; + try { + info = await track.info(); + } catch (err) { + this.#datagramClaims.delete(semantic); + throw err; + } + return Datagram.Producer.make(track, info, keys(this.credential, this.epoch, semantic).datagram); } } diff --git a/js/crypto/src/e2ee/index.ts b/js/crypto/src/e2ee/index.ts index d58d299accc..949c58e758e 100644 --- a/js/crypto/src/e2ee/index.ts +++ b/js/crypto/src/e2ee/index.ts @@ -35,7 +35,7 @@ export * as Broadcast from "./broadcast.ts"; export { Credential } from "./credential.ts"; -/** Decrypted datagrams and the events a protected datagram read yields. */ +/** Protected datagram track producers, consumers, and read events. */ export * as Datagram from "./datagram.ts"; export type { Name } from "./derive.ts"; /** Generation epochs: the UUIDv7 text a publisher instance announces as its route's instance. */ @@ -46,5 +46,5 @@ export type * as Generation from "./generation.ts"; /** Grouped-frame producer and consumer for one group identity. */ export * as Group from "./group.ts"; export { MAX_DATAGRAM_PLAINTEXT, MAX_GROUPED_PLAINTEXT, SECRET_LEN } from "./limits.ts"; -/** Grouped-frame and datagram writers and readers for one physical track. */ +/** Grouped-frame writers and readers for one physical track. */ export * as Track from "./track.ts"; diff --git a/js/crypto/src/e2ee/internal.ts b/js/crypto/src/e2ee/internal.ts index 18104a494cd..5375dbfa9ed 100644 --- a/js/crypto/src/e2ee/internal.ts +++ b/js/crypto/src/e2ee/internal.ts @@ -23,7 +23,7 @@ export interface Writer { /** State one track consumer shares with its groups. */ export interface Reader { - // One open in flight across the track's groups and datagrams; net reads wait outside it. + // One open in flight across the stream track's groups; net reads wait outside it. pump: Pump; key: TrackKey; terminal: Terminal; diff --git a/js/crypto/src/e2ee/track.ts b/js/crypto/src/e2ee/track.ts index 4823b804e0a..961952ad9ea 100644 --- a/js/crypto/src/e2ee/track.ts +++ b/js/crypto/src/e2ee/track.ts @@ -1,24 +1,21 @@ /** - * Grouped-frame and datagram writers and readers for one physical track. + * Grouped-frame writers and readers for one physical track. * * @module */ -import { type Group as NetGroup, Track as NetTrack, type Time } from "@moq/net"; +import type { Group as NetGroup, Track as NetTrack } from "@moq/net"; import { Pump } from "../pump.ts"; -import type { Event } from "./datagram.ts"; import { checkGroup } from "./derive.ts"; import { Failure } from "./failure.ts"; import * as Group from "./group.ts"; -import type { Keys, Reader, Writer } from "./internal.ts"; +import type { Reader, Writer } from "./internal.ts"; import type { TrackKey } from "./key.ts"; -import { MAX_DATAGRAM_PAYLOAD } from "./limits.ts"; import { Terminal } from "./terminal.ts"; -import { DatagramWindow } from "./window.ts"; /** * Exclusive producer for one physical track. * - * Owns the grouped-frame and datagram keys and the shared sequence namespace. Sequences are + * Owns the grouped-frame key and sequence namespace. Sequences are * allocated monotonically; reuse and exhaustion are refused before encryption. Every write * on the track and its groups runs through one serial pump, so net writes land in call order. */ @@ -28,20 +25,18 @@ export class Producer { #inner: NetTrack.Producer; #writer: Writer; - #datagramKey: TrackKey; #next = 0n; #finished = false; - private constructor(inner: NetTrack.Producer, info: NetTrack.Info, keys: Keys) { + private constructor(inner: NetTrack.Producer, info: NetTrack.Info, key: TrackKey) { this.#inner = inner; this.name = inner.name; - this.#writer = { pump: new Pump(), key: keys.group, track: info, closed: false }; - this.#datagramKey = keys.datagram; + this.#writer = { pump: new Pump(), key, track: info, closed: false }; } /** @internal */ - static make(inner: NetTrack.Producer, info: NetTrack.Info, keys: Keys): Producer { - return new Producer(inner, info, keys); + static make(inner: NetTrack.Producer, info: NetTrack.Info, key: TrackKey): Producer { + return new Producer(inner, info, key); } /** The track's final outcome, once its declared end is reached or it aborts. */ @@ -68,43 +63,9 @@ export class Producer { return Group.Producer.make(inner, this.#writer); } - /** Encrypt `plaintext` at the next sequence and insert the datagram, resolving to that sequence. */ - async appendDatagram(timestamp: Time.Timestamp | undefined, plaintext: Uint8Array): Promise { - const sequence = this.#next; - await this.insertDatagram(sequence, timestamp, plaintext); - return sequence; - } - - /** - * Encrypt `plaintext` at an explicit sequence and insert the datagram. - * - * Plaintext is capped at `MAX_DATAGRAM_PLAINTEXT` and copied, so the caller may reuse its - * buffer. A predictable failure, including a mismatched timestamp, is refused before the - * sequence is allocated; once admitted, the identity is spent even if the net write fails. - * Rejects with {@link Failure} `reuse`, `identity`, `exhausted`, or `oversize`, or a net error. - */ - async insertDatagram( - sequence: bigint, - timestamp: Time.Timestamp | undefined, - plaintext: Uint8Array, - ): Promise { - this.#writable(); - this.#reserve(sequence); - const at = NetTrack.rescale(this.#writer.track, timestamp); - const iv = this.#datagramKey.admitSeal(sequence, 0, plaintext.byteLength, MAX_DATAGRAM_PAYLOAD); - this.#next = sequence + 1n; - const owned = plaintext.slice(); - return this.#writer.pump.run(async () => { - if (this.#writer.aborted) throw this.#writer.aborted; - const payload = await this.#datagramKey.encrypt(iv, owned); - if (this.#writer.aborted) throw this.#writer.aborted; - this.#inner.insertDatagram(sequence, at, payload); - }); - } - /** * Declare the end after queued writes. Existing groups may still be written; new groups - * and datagrams are refused. Idempotent. + * are refused. Idempotent. */ finish(): void { if (this.#finished || this.#writer.closed) return; @@ -149,8 +110,7 @@ export class Producer { * Subscriber for one physical track. * * Grouped authentication failure ends the track and rejects pending reads with {@link Failure} - * `authentication`. A datagram that fails to open is dropped as an `identity`, `oversize`, or - * `authentication` {@link Event} until that key is exhausted. + * `authentication`. */ export class Consumer { /** The physical track name. */ @@ -158,21 +118,15 @@ export class Consumer { #inner: NetTrack.Subscriber; #reader: Reader; - #datagramKey: TrackKey; - #window = new DatagramWindow(); - // Datagram reads run one at a time, so the window and key see them in arrival order. - #datagrams: Promise = Promise.resolve(); - - private constructor(inner: NetTrack.Subscriber, keys: Keys) { + private constructor(inner: NetTrack.Subscriber, key: TrackKey) { this.#inner = inner; this.name = inner.name; - this.#reader = { pump: new Pump(), key: keys.group, terminal: new Terminal(inner) }; - this.#datagramKey = keys.datagram; + this.#reader = { pump: new Pump(), key, terminal: new Terminal(inner) }; } /** @internal */ - static make(inner: NetTrack.Subscriber, keys: Keys): Consumer { - return new Consumer(inner, keys); + static make(inner: NetTrack.Subscriber, key: TrackKey): Consumer { + return new Consumer(inner, key); } /** @@ -196,56 +150,6 @@ export class Consumer { return group && Group.Consumer.make(group, this.#reader); } - /** - * Receive the next datagram event, or `undefined` once the track ends. - * - * A datagram that fails to open (a sequence out of range, a bad size, or a forged tag) and a - * duplicate are events, not track-ending errors. Rejects with {@link Failure} - * `authentication` only if a grouped frame already failed, `exhausted` from opening, or a - * net error. - */ - recvDatagram(): Promise { - const result = this.#datagrams.then(() => this.#recvDatagram()); - this.#datagrams = result.catch(() => {}); - return result; - } - - async #recvDatagram(): Promise { - this.#reader.terminal.check(); - let datagram: NetTrack.Datagram | undefined; - try { - datagram = await this.#inner.recvDatagram(); - } catch (err) { - this.#reader.terminal.check(); - throw err; - } - this.#reader.terminal.check(); - if (!datagram) return undefined; - - const { sequence, timestamp, payload } = datagram; - if (this.#window.isDuplicate(sequence)) return { kind: "duplicate", sequence }; - let plaintext: Uint8Array; - try { - const iv = this.#datagramKey.admitOpen(sequence, 0, payload.byteLength, MAX_DATAGRAM_PAYLOAD); - plaintext = await this.#reader.pump.run(() => { - this.#reader.terminal.check(); - return this.#datagramKey.decrypt(iv, payload); - }); - } catch (err) { - // A grouped failure during the open still ends this read. - this.#reader.terminal.check(); - // A bad datagram is not grouped failure: the track, its demand, and any parked grouped - // read stay up. `exhausted` is this key's own limit. - if (!(err instanceof Failure)) throw err; - const kind = err.code; - if (kind === "authentication" || kind === "identity" || kind === "oversize") return { kind, sequence }; - throw err; - } - this.#reader.terminal.check(); - this.#window.mark(sequence); - return { kind: "datagram", datagram: { sequence, timestamp, plaintext } }; - } - /** * Unsubscribe, releasing track demand. As in `@moq/net`, groups already received end * after the frames they hold. Idempotent. diff --git a/js/crypto/src/e2ee/transport.test.ts b/js/crypto/src/e2ee/transport.test.ts index 93393471fcd..176690027de 100644 --- a/js/crypto/src/e2ee/transport.test.ts +++ b/js/crypto/src/e2ee/transport.test.ts @@ -112,7 +112,7 @@ for (const [name, protocol, routed, timed] of transports) { for (const [name, protocol] of [ ["moq-lite-07", "moq-lite-07-wip"], - ["moq-transport-18", "moqt-18"], + ["moq-lite-05", "moq-lite-05"], ] as const) test(`${name}: datagrams roundtrip encrypted`, async () => { const { publisher, subscriber, close } = await session(protocol); @@ -121,36 +121,31 @@ for (const [name, protocol] of [ const path = await cred.path("meeting.hang"); const broadcastPublication = publisher.createBroadcast(path); const broadcast = broadcastPublication.broadcast; - const audio = await new Protected.Producer(broadcast, generation).createTrack("audio", { + const audio = await new Protected.Producer(broadcast, generation).createDatagrams("audio", { timescale: Time.Timescale.MILLI, }); broadcastPublication.announce({ instance: Epoch.instance(generation.epoch) }); - // A group first, so the subscription is serving before datagrams, which are never cached. - const group = audio.appendGroup(); - await group.writeFrame(ms(0), encoder.encode("init")); - group.finish(); - const remote = await active(subscriber, path); const consumer = await new Protected.Consumer( remote, cred.consume(Epoch.of(remote) ?? generation.epoch), - ).subscribe("audio", { delay: { max: Milli(60_000) } }); - expect(decoder.decode((await (await consumer.recvGroup())?.readFrame())?.plaintext)).toBe("init"); - - // Sequences continue past the group, preserved hop to hop by insertDatagram. - const sequence = await audio.appendDatagram(ms(20), encoder.encode("opus")); - expect(sequence).toBe(1n); - const event = await consumer.recvDatagram(); + ).datagrams("audio"); + // Once the request is answered the peer routes datagrams to it, so the first append cannot + // arrive before the track is registered. + await consumer.info(); + while (audio.priority.peek() === undefined) await audio.priority.changed(); + const sequence = await audio.append(ms(20), encoder.encode("opus")); + expect(sequence).toBe(0n); + const event = await consumer.recv(); expect(event?.kind).toBe("datagram"); if (event?.kind !== "datagram") return; - expect(event.datagram.sequence).toBe(1n); + expect(event.datagram.sequence).toBe(0n); expect(event.datagram.timestamp?.as(Time.Timescale.MILLI)).toBe(20); expect(decoder.decode(event.datagram.plaintext)).toBe("opus"); consumer.close(); audio.finish(); - audio.close(); broadcast.close(); close(); }); diff --git a/js/json/src/snapshot/snapshot.test.ts b/js/json/src/snapshot/snapshot.test.ts index 2d037330da0..a6c49574b10 100644 --- a/js/json/src/snapshot/snapshot.test.ts +++ b/js/json/src/snapshot/snapshot.test.ts @@ -156,15 +156,13 @@ test("latest abandons a held group when a newer snapshot group rolls", async () expect(await consumer.latest()).toBeUndefined(); }); -// Skipping to the head takes groups, not the track's largest position, so a datagram past every -// group does not strand it. -test("latest ignores a datagram past every group", async () => { +// Skipping to the head reads the newest snapshot group. +test("latest reads the newest snapshot group", async () => { const track = new Track.Producer("test"); const producer = new Producer({ track, deltas: false }); const consumer = new Consumer({ track }); for (let n = 0; n < 4; n++) producer.update({ value: { a: n } }); - track.appendDatagram(Time.Timestamp.now(), new Uint8Array([0])); expect((await consumer.latest())?.value).toEqual({ a: 3 }); producer.finish(); }); diff --git a/js/justfile b/js/justfile index a73a5642a38..0b8f122f610 100644 --- a/js/justfile +++ b/js/justfile @@ -61,7 +61,7 @@ test-changed list: coverage package="": python3 ../sh/js/coverage.py '{{ package }}' -# Run one net benchmark: broadcasts, forward, requests, fills, fetch, reader, frames, track, resume, observed, drift, ledger, bandwidth, replay, varint, lite-varint, priority, subscribe-drop, refresh, or compression. +# Run one net benchmark: broadcasts, forward, requests, fills, fetch, reader, frames, track, resume, observed, drift, ledger, bandwidth, replay, varint, lite-varint, priority, subscribe-drop, refresh, compression, or datagrams. bench name: bun install --frozen-lockfile bun net/bench/{{ name }}.ts diff --git a/js/net/MIGRATE.md b/js/net/MIGRATE.md index 7116046186f..e88a594fd64 100644 --- a/js/net/MIGRATE.md +++ b/js/net/MIGRATE.md @@ -4,6 +4,29 @@ How to update your code for each breaking change, newest first. ## Unreleased +### Datagrams use separate track types ([#7846](https://github.com/moq-dev/moq/pull/7846)) + +Create a datagram track with `broadcast.createDatagrams(name, info)` and read it with +`broadcast.datagrams(name)`. Replace `Track.Producer.appendDatagram(timestamp, payload)` +with `Datagram.Producer.append(timestamp, payload)`, `insertDatagram(sequence, timestamp, payload)` +with `insert({ sequence, timestamp, payload })`, and `Subscriber.recvDatagram()` or +`Ordered.recvDatagram()` with `Datagram.Consumer.recv()`. The payload type is now +`Datagram.Datagram`, replacing `Track.Datagram`. + +Stream and datagram tracks may share a name, with separate sequence counters. Datagram +consumers start live, have only a priority (`setPriority`), and have no range, order, +compression, FETCH, or cache API. `finish()` drains existing readers; `abort(error)` +fails them immediately. Close consumers when done. + +A datagram consumer binds to the broadcast's serving source when opened. It keeps no +stream resume state and does not move to a replacement route. + +Lite-07-wip uses DATAGRAMS on stream type 0x9 and carries no Inflated Size. Lite-05/06 +keep their wire format and carry both kinds of a name on one SUBSCRIBE, without deduplication. +Create a datagram track before or with its same-name stream track on those versions. +Lite-01..04 and transports without datagrams refuse datagram requests. moq-transport +currently refuses them while its separate datagram-track adapter is implemented. + ### Announcements are a listing ([#7678](https://github.com/moq-dev/moq/pull/7678)) `Announce.Event` is `{ kind: "present" | "absent" } & Announce`. Another instance taking a path over keeps it present, and a route's cost or hops changing is not reported, so `start`, `update`, `restart` and `end` are gone. diff --git a/js/net/api/index.api.md b/js/net/api/index.api.md index dba751e5a23..31794468923 100644 --- a/js/net/api/index.api.md +++ b/js/net/api/index.api.md @@ -95,6 +95,7 @@ export namespace Broadcast { clone(): Broadcast.Consumer; close(): void; get closed(): Signals.GetPromise; + datagrams(name: string): Datagram.Consumer; get instance(): Instance.Valid | undefined; get path(): Path.Valid; protected shareState(): never; @@ -106,6 +107,7 @@ export namespace Broadcast { close(): void; get closed(): Signals.GetPromise; consume(): Broadcast.Consumer; + createDatagrams(name: string, info?: Partial): Datagram.Producer; createTrack(name: string, info?: Track.InfoInit): Track.Producer; removeTrack(name: string): void; track(name: string): Track.Consumer; @@ -262,6 +264,50 @@ export class Connection { readonly viewers: Signals.Getter | undefined>; } +export namespace Datagram { + export class Consumer { + clone(): Datagram.Consumer; + close(error?: Error): void; + info(): Promise; + readonly name: string; + get priority(): number; + recv(): Promise; + setPriority(value: number): void; + } + export interface Datagram { + payload: Uint8Array; + sequence: bigint; + timestamp?: Time.Timestamp; + } + export interface Info { + priority: number; + timescale?: Time.Timescale | null; + } + export class Producer { + constructor(name: string); + abort(error: Error): void; + accept(properties?: Partial): this; + append(timestamp: Time.Timestamp | undefined, payload: Uint8Array): bigint; + get closed(): Signals.GetPromise; + consume(): Datagram.Consumer; + finish(): void; + info(): Promise; + insert(datagram: Datagram.Datagram): void; + readonly name: string; + get priority(): Signals.Getter; + } + export class Request { + accept(properties?: Partial): Datagram.Producer; + get closed(): Signals.GetPromise; + get name(): string; + get priority(): Signals.Getter; + reject(error: Error): void; + } + export function rescale(info: { + timescale?: Time.Timescale | null; + }, timestamp: Time.Timestamp | undefined): Time.Timestamp | undefined; +} + export namespace Error { export class ConnectTimeout extends Error { constructor(timeout: Milli); @@ -626,11 +672,6 @@ export namespace Track { readonly name: string; subscribe(options?: Track.Subscription): Track.Subscriber; } - export interface Datagram { - payload: Uint8Array; - sequence: bigint; - timestamp?: Time.Timestamp; - } export interface Delay { max?: Milli | null; min?: Milli | null; @@ -691,7 +732,6 @@ export namespace Track { } & Group.Frame | undefined>; readJson(): Promise; readString(): Promise; - recvDatagram(): Promise; get subscription(): Signals.Getter; tryNextGroup(): Group.Consumer | undefined; update(options: Track.Subscription): void; @@ -705,7 +745,6 @@ export namespace Track { constructor(name: string); abort(error: Error): void; accept(info?: Track.InfoInit): this; - appendDatagram(timestamp: Time.Timestamp | undefined, payload: Uint8Array): bigint; appendGroup(): Group.Producer; close(): void; get closed(): Signals.GetPromise; @@ -714,7 +753,6 @@ export namespace Track { finish(): void; finishAt(end: Track.Position, error?: Error): void; info(): Promise; - insertDatagram(sequence: bigint, timestamp: Time.Timestamp | undefined, payload: Uint8Array): void; largest(): Track.Position | undefined; readonly name: string; get priority(): number; @@ -749,7 +787,6 @@ export namespace Track { largest(): Promise; readonly name: string; ordered(): Track.Ordered; - recvDatagram(): Promise; recvGroup(): Promise; get subscription(): Signals.Getter; tryRecvGroup(): Group.Consumer | undefined; @@ -766,7 +803,7 @@ export namespace Track { ranges?: Track.Range[]; } export function infoDefaults(info?: Partial): Track.Info; - export function rescale(info: Pick, timestamp: Time.Timestamp | undefined): Time.Timestamp | undefined; + export import rescale = Datagram.rescale; } export namespace Varint { diff --git a/js/net/bench/compression.ts b/js/net/bench/compression.ts index 0e1f2cfc674..cde70da91fa 100644 --- a/js/net/bench/compression.ts +++ b/js/net/bench/compression.ts @@ -1,51 +1,11 @@ -/** Sweep compressed publishers x plain readers, reporting one shared inflated copy per group, then compressed datagram retention. */ +/** Sweep compressed publishers x plain readers, reporting one shared inflated copy per group. */ import { gzipSync } from "node:zlib"; -import { Encoder } from "../src/flate/encoder.ts"; -import { hooks } from "../src/internal.ts"; -import { Producer as Track } from "../src/track.ts"; import { fanout, tails } from "./compression-fanout.ts"; console.log("publishers,readers,phase,us,size_bytes"); for (const row of fanout()) console.log([row.publishers, row.readers, row.phase, row.us.toFixed(1), row.size_bytes].join(",")); -// Received compressed datagrams of zeros, each inflating to the largest payload, swept over -// tracks x queued datagrams. A fast reader takes each one while a slow reader never reads, so -// the heap the slow queues retain shows whether they hold the slices or inflated copies. -const zeros = new Uint8Array(65535); -const slice = new Encoder().frame(zeros); -async function datagrams(count: number, queued: number): Promise<[number, number]> { - Bun.gc(true); - const before = process.memoryUsage().heapUsed; - const tracks = Array.from({ length: count }, () => new Track("bench").accept({ compress: true })); - const slow = tracks.map((track) => track.subscribe()); - const fast = tracks.map((track) => track.subscribe()); - const started = performance.now(); - for (let sequence = 0n; sequence < BigInt(queued); sequence++) { - for (const [index, track] of tracks.entries()) { - // Its own copy, as each received datagram carries its own wire bytes. - hooks.writeDatagram(track, { sequence, payload: slice.slice() }, zeros.length); - const datagram = await fast[index].recvDatagram(); - if (datagram?.payload.length !== zeros.length) throw new Error("missing payload"); - } - } - const deliver = ((performance.now() - started) * 1000) / (count * queued); - Bun.gc(true); - const retained = process.memoryUsage().heapUsed - before; - for (const track of tracks) track.finish(); - for (const track of tracks) track.close(); - slow.length = 0; - return [deliver, retained]; -} -await datagrams(16, 64); -console.log("tracks,queued,deliver_us,retained_bytes"); -for (const count of [1, 16, 64]) { - for (const queued of [1, 8, 64]) { - const [deliver, retained] = await datagrams(count, queued); - console.log([count, queued, deliver.toFixed(1), retained].join(",")); - } -} - // This entry is an application importing only its track consumer. Synchronous model writes // share the module, so its bundle includes the encoder as well as the decoder. const bundle = await Bun.build({ diff --git a/js/net/bench/datagrams.ts b/js/net/bench/datagrams.ts new file mode 100644 index 00000000000..8ca94efc37b --- /dev/null +++ b/js/net/bench/datagrams.ts @@ -0,0 +1,34 @@ +/** Sweep datagram tracks x readers, with a stalled reader sharing each bounded buffer. */ +import { Producer } from "../src/datagram.ts"; + +const payload = new Uint8Array(1024); +async function measure(tracks: number, readers: number): Promise<{ join: number; delivery: number }> { + const joining = performance.now(); + const producers = Array.from({ length: tracks }, () => new Producer("bench").accept()); + const consumers = producers.map((producer) => Array.from({ length: readers }, () => producer.consume())); + const stalled = producers.map((producer) => producer.consume()); + const join = ((performance.now() - joining) * 1e6) / (tracks * (readers + 1)); + const start = performance.now(); + for (let i = 0; i < 256; i++) { + for (const [index, producer] of producers.entries()) { + producer.append(undefined, payload); + for (const consumer of consumers[index]) { + if ((await consumer.recv())?.sequence !== BigInt(i)) throw new Error("missing datagram"); + } + } + } + const elapsed = performance.now() - start; + for (const producer of producers) producer.finish(); + for (const row of consumers) for (const consumer of row) consumer.close(); + for (const consumer of stalled) consumer.close(); + return { join, delivery: (elapsed * 1e6) / (tracks * readers * 256) }; +} +await measure(4, 4); +console.log("tracks,readers,ns_per_join,ns_per_delivery"); +for (const tracks of [1, 16, 64]) + for (const readers of [1, 16, 64]) { + const runs = []; + for (let run = 0; run < 3; run++) runs.push(await measure(tracks, readers)); + runs.sort((a, b) => a.delivery - b.delivery); + console.log(`${tracks},${readers},${runs[1].join.toFixed(1)},${runs[1].delivery.toFixed(1)}`); + } diff --git a/js/net/bench/resume.ts b/js/net/bench/resume.ts index 0b66e2a7ec7..de26949e05b 100644 --- a/js/net/bench/resume.ts +++ b/js/net/bench/resume.ts @@ -238,44 +238,3 @@ for (const replayed of [0, 1, 64, 1024]) { console.log(`${readers},${replayed},${Math.min(...samples).toFixed(2)}`); } } - -// Sweep audience and delivered history: a read must touch only the delivered identity, -// whether the history is under, at, or past the replay-buffer cap on retained identities. -async function measureDatagrams(readers: number, retained: number): Promise { - const front = new Front(); - const broadcast = new BroadcastProducer(); - const producer = broadcast.createTrack("seq"); - front.serve(broadcast.consume()); - const subscribers = Array.from({ length: readers }, () => - front.consumer.track("seq").subscribe({ delay: { max: Milli(3_600_000) } }), - ); - const deliver = (sequence: bigint) => { - producer.insertDatagram(sequence, undefined, payload); - for (const subscriber of subscribers) { - const recv = hooks.tryRecvDatagram(subscriber); - if (recv.kind !== "datagram") throw new Error("datagram was not delivered"); - checksum += Number(recv.datagram.sequence); - } - }; - const base = BigInt(retained); - for (let sequence = 0n; sequence < base; sequence++) deliver(sequence); - const end = base + BigInt(groups); - const start = performance.now(); - for (let sequence = base; sequence < end; sequence++) deliver(sequence); - const elapsed = performance.now() - start; - for (const subscriber of subscribers) subscriber.close(); - front.consumer.close(); - broadcast.close(); - await flush(); - return (elapsed * 1000) / groups / readers; -} - -await measureDatagrams(4, 64); -console.log("datagram_readers,retained,us_per_datagram"); -for (const retained of [1, 64, 4096]) { - for (const readers of readerCounts) { - const runs: number[] = []; - for (let rep = 0; rep < reps; rep++) runs.push(await measureDatagrams(readers, retained)); - console.log(`${readers},${retained},${Math.min(...runs).toFixed(2)}`); - } -} diff --git a/js/net/src/broadcast.ts b/js/net/src/broadcast.ts index 8962412ccbf..e07d002d8ca 100644 --- a/js/net/src/broadcast.ts +++ b/js/net/src/broadcast.ts @@ -4,6 +4,7 @@ * @module */ import { type GetPromise, Once, Signal } from "@moq/signals"; +import * as datagram from "./datagram.ts"; import { NotFound, StreamCode, StreamError, toStreamCode } from "./error.ts"; import type { Consumer as GroupConsumer } from "./group.ts"; import type { Route } from "./hop.ts"; @@ -13,7 +14,7 @@ import { Listing, listTracks, makeLister, type Tracks } from "./listing.ts"; import * as Path from "./path.ts"; import * as track from "./track.ts"; import { untilAborted } from "./util/abort.ts"; -import { registerWire, trackOf } from "./wire.ts"; +import { registerWire, trackOf, wireOf } from "./wire.ts"; export { Tracks, type TracksEvent } from "./listing.ts"; @@ -40,6 +41,10 @@ class Lookups { /** Reactive backing state shared by broadcast producers and consumers. */ class BroadcastState { requested = new Signal([]); + datagramRequested = new Signal([]); + datagramPending = new Set(); + datagrams = new Map(); + datagramServed = false; pending = new Set(); closed = new Once(); // The tracks the broadcast lists, and the readers listing them. @@ -63,6 +68,74 @@ class BroadcastState { closers = new Set<() => void>(); } +// Datagram tracks never enter the stream listing or cache. +function liveDatagrams(state: BroadcastState, name: string): datagram.Consumer | undefined { + const producer = state.datagrams.get(name); + if (!producer) return; + if (producer.closed.peek() !== undefined) { + state.datagrams.delete(name); + return; + } + return producer.consume(); +} + +function createDatagrams(state: BroadcastState, name: string, info: Partial): datagram.Producer { + if (state.closed.peek() !== undefined) throw new Error("broadcast is closed"); + // A request its readers closed waits for async cleanup; serve the name fresh instead. + const queued = state.datagramRequested + .peek() + .find((request) => request.name === name && request.closed.peek() === undefined); + if (queued) { + state.datagramRequested.mutate((requests) => requests.splice(requests.indexOf(queued), 1)); + return queued.accept(info); + } + const existing = state.datagrams.get(name); + if (existing?.closed.peek() === undefined && existing) throw new Error(`duplicate datagram track: ${name}`); + const producer = new datagram.Producer(name).accept(info); + state.datagrams.set(name, producer); + void producer.closed.then(() => { + if (state.datagrams.get(name) === producer) state.datagrams.delete(name); + }); + return producer; +} + +function datagrams(state: BroadcastState, name: string): datagram.Consumer { + if (state.closed.peek() !== undefined) throw new Error("broadcast is closed"); + const existing = liveDatagrams(state, name); + if (existing) return existing; + const producer = new datagram.Producer(name); + const consumer = producer.consume(); + if (!state.datagramServed) { + producer.abort(new NotFound(`datagram track ${name}`)); + return consumer; + } + const request = hooks.datagramRequest(producer, state.datagramPending); + state.datagrams.set(name, producer); + state.datagramRequested.mutate((requests) => requests.push(request)); + void producer.closed.then(() => { + state.datagramPending.delete(request); + if (state.datagrams.get(name) === producer) state.datagrams.delete(name); + state.datagramRequested.mutate((requests) => { + const index = requests.indexOf(request); + if (index !== -1) requests.splice(index, 1); + }); + }); + return consumer; +} + +async function requestedDatagrams(state: BroadcastState): Promise { + state.datagramServed = true; + for (;;) { + const requests = state.datagramRequested.peek(); + requests.sort((a, b) => (a.priority.peek() ?? -128) - (b.priority.peek() ?? -128)); + const request = requests.pop(); + if (request && request.closed.peek() === undefined) return request; + if (request) continue; + if (state.closed.peek() !== undefined) return; + await Signal.race(state.datagramRequested, state.closed); + } +} + function dequeueRequest(state: BroadcastState): track.Request | undefined { const requested = state.requested.peek(); requested.sort((a, b) => a.priority - b.priority); @@ -96,6 +169,8 @@ function closeState(state: BroadcastState) { if (state.closed.peek() !== undefined) return; state.closed.set(null); state.listing.close(); + for (const request of state.datagramPending) request.reject(new StreamError(StreamCode.Cancel)); + state.datagramRequested.set([]); for (const request of state.pending) request.reject( new StreamError(StreamCode.Internal, { @@ -321,6 +396,10 @@ export class Producer { largest: (name) => largest(this.#state, name), fetchGroup: (name, sequence, options) => fetchGroup(this.#state, name, sequence, options), requested: () => requested(this.#state), + datagrams: (name) => datagrams(this.#state, name), + liveDatagrams: (name) => liveDatagrams(this.#state, name), + createDatagrams: (name, info) => createDatagrams(this.#state, name, info), + requestedDatagrams: () => requestedDatagrams(this.#state), tracks: (owner) => listTracks(this.#state.listing, owner), lister: () => makeLister(this.#state.listing), }); @@ -393,6 +472,11 @@ export class Producer { return producer; } + /** Create a live datagram track in a namespace separate from stream tracks. */ + createDatagrams(name: string, info: Partial = {}): datagram.Producer { + return createDatagrams(this.#state, name, info); + } + /** * Remove a statically inserted track, including a finished cached one, and stop listing it * at once. @@ -458,6 +542,7 @@ export class Consumer { // A standalone consumer is a wire layer's, which serves every track on demand. this.#state = new BroadcastState(); this.#state.served = true; + this.#state.datagramServed = true; this.#state.unheld = true; } this.#state.consumers++; @@ -467,6 +552,10 @@ export class Consumer { largest: (name) => largest(this.#state, name), fetchGroup: (name, sequence, options) => fetchGroup(this.#state, name, sequence, options), requested: () => requested(this.#state), + datagrams: (name) => datagrams(this.#state, name), + liveDatagrams: (name) => liveDatagrams(this.#state, name), + createDatagrams: (name, info) => createDatagrams(this.#state, name, info), + requestedDatagrams: () => requestedDatagrams(this.#state), tracks: (owner) => this.#tracks(owner), lister: () => makeLister(this.#state.listing), }); @@ -547,6 +636,11 @@ export class Consumer { } satisfies Shared as never; } + /** Read a datagram track live from now, independently of any same-name stream track. */ + datagrams(name: string): datagram.Consumer { + return wireOf(this).datagrams(name); + } + /** Get a lazy handle for a track on this broadcast. Repeat subscriptions dedupe onto one upstream subscription. */ track(name: string): track.Consumer { return trackOf(name, this); diff --git a/js/net/src/datagram.test.ts b/js/net/src/datagram.test.ts new file mode 100644 index 00000000000..a2d6ab801cf --- /dev/null +++ b/js/net/src/datagram.test.ts @@ -0,0 +1,187 @@ +import { expect, test } from "bun:test"; +import * as Broadcast from "./broadcast.ts"; + +test("a broadcast keeps same-name datagram and stream tracks independent", async () => { + const broadcast = new Broadcast.Producer(); + const streams = broadcast.createTrack("audio"); + const datagrams = broadcast.createDatagrams("audio"); + const reader = broadcast.consume().datagrams("audio"); + expect(streams.appendGroup().sequence).toBe(0n); + expect(datagrams.append(undefined, new Uint8Array([1]))).toBe(0n); + expect(streams.appendGroup().sequence).toBe(1n); + expect((await reader.recv())?.sequence).toBe(0n); + reader.close(); + datagrams.finish(); + streams.finish(); + broadcast.close(); +}); + +import * as Datagram from "./datagram.ts"; +import { StreamCode, StreamError, TimestampMismatch } from "./error.ts"; +import { hooks } from "./internal.ts"; +import { Timescale, Timestamp } from "./time.ts"; +import { wireOf } from "./wire.ts"; + +test("datagram readers start live, clone their cursor, and drain only an existing finish", async () => { + const track = new Datagram.Producer("test").accept(); + track.append(undefined, new Uint8Array([0])); + const reader = track.consume(); + track.append(undefined, new Uint8Array([1])); + track.finish(); + const cloned = reader.clone(); + expect((await reader.recv())?.sequence).toBe(1n); + reader.close(); + expect((await cloned.recv())?.sequence).toBe(1n); + expect(await reader.recv()).toBeUndefined(); + expect(await track.consume().recv()).toBeUndefined(); + reader.close(); + cloned.close(); +}); + +test("a shared datagram buffer bounds both count and bytes for stalled readers", async () => { + for (const size of [1, 16384, 65535]) { + const track = new Datagram.Producer("test").accept(); + const slow = track.consume(); + const fast = track.consume(); + for (let i = 0; i < 70; i++) { + track.append(undefined, new Uint8Array(size)); + expect((await fast.recv())?.sequence).toBe(BigInt(i)); + } + track.finish(); + const kept = Math.min(64, Math.floor(65536 / size)); + expect((await slow.recv())?.sequence).toBe(BigInt(70 - kept)); + slow.close(); + fast.close(); + } +}); + +test("datagram inserts preserve repeats and reordering, and append past the high water mark", async () => { + const track = new Datagram.Producer("test").accept(); + const reader = track.consume(); + for (const sequence of [10n, 3n, 10n]) track.insert({ sequence, payload: new Uint8Array() }); + expect(track.append(undefined, new Uint8Array())).toBe(11n); + for (const sequence of [10n, 3n, 10n, 11n]) expect((await reader.recv())?.sequence).toBe(sequence); + track.insert({ sequence: (1n << 64n) - 1n, payload: new Uint8Array() }); + expect(() => track.append(undefined, new Uint8Array())).toThrow(RangeError); + track.finish(); + reader.close(); +}); + +test("refused datagram writes spend neither sequences nor buffer space", async () => { + const track = new Datagram.Producer("test").accept({ timescale: Timescale.MILLI }); + const reader = track.consume(); + expect(() => track.append(undefined, new Uint8Array())).toThrow(TimestampMismatch); + expect(() => track.append(Timestamp.fromMillis(1), new Uint8Array(65536))).toThrow(); + expect(track.append(Timestamp.fromMicros(1234), new Uint8Array([7]))).toBe(0n); + expect((await reader.recv())?.timestamp).toEqual(Timestamp.fromMillis(1)); + track.finish(); + reader.close(); +}); + +test("an abort or local close wakes datagram reads with its exact result", async () => { + const track = new Datagram.Producer("test").accept(); + const reader = track.consume(); + track.append(undefined, new Uint8Array([1])); + const cause = new StreamError(StreamCode.Unauthorized); + track.abort(cause); + await expect(reader.recv()).rejects.toBe(cause); + const live = new Datagram.Producer("live").accept(); + const local = live.consume(); + const read = local.recv(); + local.close(cause); + await expect(read).rejects.toBe(cause); + reader.close(); + live.finish(); +}); + +test("each datagram reader owns its copy, and edits never reach the shared byte bound", async () => { + const track = new Datagram.Producer("test").accept(); + const first = track.consume(); + const second = track.consume(); + // Two of these fit the 64 KiB bound; a third evicts exactly one. + const size = 30_000; + track.append(undefined, new Uint8Array(size).fill(5)); + track.append(undefined, new Uint8Array(size).fill(6)); + + const edited = await first.recv(); + if (!edited) throw new Error("missing datagram"); + edited.payload[0] = 9; + edited.sequence = 7n; + const read = await second.recv(); + expect(read?.sequence).toBe(0n); + expect(read?.payload[0]).toBe(5); + expect(read?.payload.byteLength).toBe(size); + + // Replacing a returned payload must not change what eviction subtracts. + edited.payload = new Uint8Array(0); + track.append(undefined, new Uint8Array(size).fill(7)); + expect((await second.recv())?.payload[0]).toBe(6); + expect((await second.recv())?.payload[0]).toBe(7); + first.close(); + second.close(); + track.finish(); +}); + +test("datagram priority is the maximum of live consumers and releases atomically", () => { + const track = new Datagram.Producer("test").accept(); + const low = track.consume(); + const high = low.clone(); + low.setPriority(-3); + high.setPriority(8); + expect(track.priority.peek()).toBe(8); + high.close(); + expect(track.priority.peek()).toBe(-3); + expect(hooks.closeUnusedDatagrams(track)).toBe(false); + low.close(); + expect(hooks.closeUnusedDatagrams(track)).toBe(true); +}); + +test("abandoned datagram requests leave neither a queued request nor a name claim", async () => { + const broadcast = new Broadcast.Producer(); + const consumer = broadcast.consume(); + const pending = wireOf(consumer).requestedDatagrams(); + const left = consumer.datagrams("test"); + left.close(); + await Promise.resolve(); + const next = consumer.datagrams("test"); + const request = await pending; + expect(request?.name).toBe("test"); + if (!request) throw new Error("missing request"); + const track = request.accept(); + track.append(undefined, new Uint8Array([1])); + expect((await next.recv())?.payload).toEqual(new Uint8Array([1])); + track.finish(); + next.close(); + consumer.close(); + broadcast.close(); +}); + +test("creating a datagram track in the turn its queued request closes serves the name fresh", async () => { + const broadcast = new Broadcast.Producer(); + const consumer = broadcast.consume(); + void wireOf(consumer).requestedDatagrams(); + const left = consumer.datagrams("test"); + left.close(); + const track = broadcast.createDatagrams("test"); + const next = consumer.datagrams("test"); + track.append(undefined, new Uint8Array([2])); + expect((await next.recv())?.payload).toEqual(new Uint8Array([2])); + track.finish(); + next.close(); + consumer.close(); + broadcast.close(); +}); + +test("closing one waiting datagram consumer releases its info wait while another still waits", async () => { + const track = new Datagram.Producer("waiting"); + const first = track.consume(); + const second = track.consume(); + const info = first.info(); + const cause = new StreamError(StreamCode.Cancel); + first.close(cause); + await expect(info).rejects.toBe(cause); + track.accept({ priority: 7 }); + expect((await second.info()).priority).toBe(7); + second.close(); + track.finish(); +}); diff --git a/js/net/src/datagram.ts b/js/net/src/datagram.ts index a73d40e2ea6..2e111d3847d 100644 --- a/js/net/src/datagram.ts +++ b/js/net/src/datagram.ts @@ -1,23 +1,305 @@ /** - * A datagram: a single unreliable payload on a track, parallel to groups. + * Datagram tracks: live, unreliable payloads with their own sequence counter. * * @module */ -import type { Timestamp } from "./time.ts"; +import { type GetPromise, type Getter, Once, Signal } from "@moq/signals"; +import { StreamCode, StreamError } from "./error.ts"; +import { hooks } from "./internal.ts"; +import { rescale } from "./rescale.ts"; +import { Timescale, type Timestamp } from "./time.ts"; +import { checkSequence, max, UNBOUNDED } from "./util/sequence.ts"; -/** - * A single unreliable payload on a track: a sequence number, a presentation timestamp on a timed track, and the bytes. - * - * Unlike a {@link Group} (an ordered stream of frames over a QUIC stream), a datagram is one - * self-contained payload carried in a single QUIC datagram: best-effort, unordered, never - * retransmitted. It shares the track's monotonic sequence-number namespace with groups but is - * otherwise independent. Mirrors the Rust `Datagram`. - */ +export { rescale } from "./rescale.ts"; + +/** A single unreliable payload, addressed within its datagram track. */ export interface Datagram { - /** Per-track sequence number, shared with the group namespace. */ + /** The datagram track's own sequence number. */ sequence: bigint; /** Presentation timestamp, present exactly when the track has a timescale. */ timestamp?: Timestamp; /** The datagram payload. */ payload: Uint8Array; } + +/** Publisher properties fixed for the lifetime of a datagram track. */ +export interface Info { + /** Timestamp units per second, or omitted for an untimed track. */ + timescale?: Timescale | null; + /** Publisher priority, higher first, breaking ties between subscriber priorities. */ + priority: number; +} + +// One shared buffer bounds payload retention independently of the number of readers. +const MAX_BUFFERED = 64; +const MAX_BYTES = 64 * 1024; +const MAX_PAYLOAD = 65535; + +function priority(value: number): number { + if (!Number.isInteger(value) || value < -128 || value > 127) throw new RangeError(`invalid priority: ${value}`); + return value; +} + +class State { + info = new Signal(undefined); + closed = new Once(); + changed = new Signal(0); + priority = new Signal(undefined); + consumers = new Map(); + buffer: Datagram[] = []; + bytes = 0; + offset = 0n; + next = 0n; + + release(): void { + this.offset += BigInt(this.buffer.length); + this.buffer = []; + this.bytes = 0; + } + + end(error?: Error): void { + if (this.closed.peek() !== undefined) return; + this.closed.set(error ?? null); + if (error || this.consumers.size === 0) this.release(); + } + + updatePriority(): void { + let highest: number | undefined; + for (const value of this.consumers.values()) highest = Math.max(highest ?? -128, value); + this.priority.set(highest); + if (highest === undefined) { + if (this.info.peek() === undefined) this.end(new StreamError(StreamCode.Cancel)); + if (this.closed.peek() !== undefined) this.release(); + } + } +} + +async function info(state: State, local?: GetPromise): Promise { + for (;;) { + const value = state.info.peek(); + if (value) return value; + const left = local?.peek(); + if (left !== undefined) throw left ?? new StreamError(StreamCode.Cancel); + const closed = state.closed.peek(); + if (closed !== undefined) throw closed ?? new Error("datagram track closed before its info"); + await (local !== undefined + ? Signal.race(state.info, state.closed, local) + : Signal.race(state.info, state.closed)); + } +} + +let consume: (name: string, state: State, cursor?: bigint, priority?: number) => Consumer; + +/** Writes a live datagram track, independently of any stream track of the same name. */ +export class Producer { + /** The track's name within its broadcast. */ + readonly name: string; + #state = new State(); + + constructor(name: string) { + this.name = name; + } + + static { + hooks.closeUnusedDatagrams = (producer) => { + if (producer.#state.consumers.size > 0) return false; + producer.abort(new StreamError(StreamCode.Cancel)); + return true; + }; + } + + /** Commit the immutable publisher properties, returning this producer. */ + accept(properties: Partial = {}): this { + try { + const state = this.#state; + if (state.closed.peek() !== undefined) throw state.closed.peek() ?? new Error("datagram track is closed"); + if (state.info.peek() !== undefined) throw new Error("datagram track is already accepted"); + const timescale = properties.timescale == null ? undefined : Timescale(properties.timescale); + state.info.set(Object.freeze({ timescale, priority: priority(properties.priority ?? 0) })); + return this; + } catch (error) { + this.abort(error instanceof Error ? error : new Error(String(error))); + throw error; + } + } + + /** The track's publisher properties, once accepted. */ + info(): Promise { + return info(this.#state); + } + + /** The highest subscriber priority, or undefined while nobody reads. */ + get priority(): Getter { + return this.#state.priority; + } + + /** The track's terminal result: null for a finish, or its abort error. */ + get closed(): GetPromise { + return this.#state.closed; + } + + /** A new consumer, starting with the next datagram written. */ + consume(): Consumer { + return consume(this.name, this.#state); + } + + /** Append a payload with the next sequence, returning that sequence. */ + append(timestamp: Timestamp | undefined, payload: Uint8Array): bigint { + const sequence = this.#state.next; + this.insert({ sequence, timestamp, payload }); + return sequence; + } + + /** Forward a datagram without renumbering; repeats and older sequences are delivered too. */ + insert(datagram: Datagram): void { + const state = this.#state; + const closed = state.closed.peek(); + if (closed !== undefined) throw closed ?? new Error("datagram track is closed"); + const properties = state.info.peek(); + if (!properties) throw new Error("datagram track has not been accepted"); + const sequence = checkSequence(datagram.sequence); + if (sequence >= UNBOUNDED) throw new RangeError("datagram sequence exhausted"); + if (datagram.payload.byteLength > MAX_PAYLOAD) throw new Error("datagram payload too large"); + const timestamp = rescale(properties, datagram.timestamp); + // A small view must not pin a peer's larger receive buffer. + const payload = new Uint8Array(datagram.payload); + state.next = max(state.next, sequence + 1n); + state.bytes += payload.byteLength; + while (state.buffer.length >= MAX_BUFFERED || (state.buffer.length > 0 && state.bytes > MAX_BYTES)) { + state.bytes -= state.buffer[0].payload.byteLength; + state.buffer.shift(); + state.offset++; + } + state.buffer.push({ sequence, timestamp, payload }); + state.changed.update((revision) => revision + 1); + } + + /** End cleanly; existing consumers first read their buffered datagrams. */ + finish(): void { + this.#state.end(); + } + + /** Abort immediately, discarding buffered datagrams. */ + abort(error: Error): void { + this.#state.end(error); + } +} + +/** A datagram track requested on demand, waiting for its publisher's answer. */ +export class Request { + #producer: Producer; + #pending: Set; + private constructor(producer: Producer, pending: Set) { + this.#producer = producer; + this.#pending = pending; + pending.add(this); + } + static { + hooks.datagramRequest = (producer, pending) => new Request(producer, pending); + } + /** The requested track's name. */ + get name(): string { + return this.#producer.name; + } + /** The highest priority among its waiting consumers. */ + get priority(): Getter { + return this.#producer.priority; + } + /** Settles when the request or its accepted producer ends. */ + get closed(): GetPromise { + return this.#producer.closed; + } + /** Accept the request and produce datagrams with these properties. */ + accept(properties: Partial = {}): Producer { + this.#pending.delete(this); + return this.#producer.accept(properties); + } + /** Refuse the request with the error its consumers receive. */ + reject(error: Error): void { + this.#pending.delete(this); + this.#producer.abort(error); + } +} + +/** Reads datagrams in arrival order, live from when this consumer was created. */ +export class Consumer { + /** The track's name within its broadcast. */ + readonly name: string; + #state: State; + #cursor: bigint; + #closed = new Once(); + #priority: Signal; + + private constructor(name: string, state: State, cursor?: bigint, value = 0) { + this.name = name; + this.#state = state; + this.#cursor = cursor ?? state.offset + BigInt(state.buffer.length); + this.#priority = new Signal(value); + if (state.closed.peek() === undefined || cursor !== undefined) { + state.consumers.set(this, value); + state.updatePriority(); + } + } + static { + consume = (name, state, cursor, priority) => new Consumer(name, state, cursor, priority); + hooks.datagramPriority = (consumer) => consumer.#priority; + hooks.datagramClosed = (consumer) => consumer.#closed; + } + + /** The track's publisher properties, once accepted. */ + info(): Promise { + return info(this.#state, this.#closed); + } + + /** This consumer's subscriber priority. */ + get priority(): number { + return this.#priority.peek(); + } + + /** Change this consumer's priority, higher first. */ + setPriority(value: number): void { + this.#priority.set(priority(value)); + if (!this.#state.consumers.has(this)) return; + this.#state.consumers.set(this, value); + this.#state.updatePriority(); + } + + /** Another consumer at the same position and priority. */ + clone(): Consumer { + if (this.#closed.peek() !== undefined) throw new Error("datagram consumer is closed"); + return consume(this.name, this.#state, this.#cursor, this.#priority.peek()); + } + + /** Read the next datagram as a copy this consumer owns, skipping any dropped by the bounded shared buffer. */ + async recv(): Promise { + for (;;) { + const local = this.#closed.peek(); + if (local !== undefined) { + if (local) throw local; + return; + } + const state = this.#state; + const closed = state.closed.peek(); + if (closed instanceof Error) throw closed; + this.#cursor = max(this.#cursor, state.offset); + const datagram = state.buffer[Number(this.#cursor - state.offset)]; + if (datagram) { + this.#cursor++; + // The buffered entry is shared by every reader and charged to the byte bound, so + // each reader gets its own envelope and payload to change as it likes. + const { sequence, timestamp, payload } = datagram; + return { sequence, timestamp, payload: payload.slice() }; + } + if (closed !== undefined) return; + await Signal.race(state.changed, state.closed, this.#closed); + } + } + + /** Release this consumer and wake its pending reads. */ + close(error?: Error): void { + if (this.#closed.peek() !== undefined) return; + this.#closed.set(error ?? null); + this.#state.consumers.delete(this); + this.#state.updatePriority(); + } +} diff --git a/js/net/src/datagram_stream.test.ts b/js/net/src/datagram_stream.test.ts index 325fa8456c2..8624994c338 100644 --- a/js/net/src/datagram_stream.test.ts +++ b/js/net/src/datagram_stream.test.ts @@ -1,7 +1,7 @@ import { expect, test } from "bun:test"; import * as DatagramStream from "./datagram_stream.ts"; -test("datagram writer disables send when createWritable throws", () => { +test("a declared createWritable failure is refused without trying the legacy writer", () => { const transport = { datagrams: { maxDatagramSize: 1200, @@ -11,5 +11,5 @@ test("datagram writer disables send when createWritable throws", () => { }, } as unknown as WebTransport; - expect(DatagramStream.datagramWriter(transport)).toBeUndefined(); + expect(() => DatagramStream.datagramWriter(transport)?.open(0)).toThrow("unsupported"); }); diff --git a/js/net/src/datagram_stream.ts b/js/net/src/datagram_stream.ts index d5ff2449772..0e8266fdcce 100644 --- a/js/net/src/datagram_stream.ts +++ b/js/net/src/datagram_stream.ts @@ -1,11 +1,12 @@ /** Compatibility helpers for browser WebTransport datagram stream shape changes. */ +import type { SendStream } from "./stream.ts"; /** The current standards-track datagram send stream, plus Chrome's legacy writable property. */ interface DatagramDuplex { readonly readable?: ReadableStream; readonly writable?: WritableStream; readonly maxDatagramSize?: number; - createWritable?: () => WritableStream; + createWritable?: (options: { sendOrder: number }) => SendStream; } function datagrams(quic: WebTransport): DatagramDuplex | undefined { @@ -18,6 +19,11 @@ export function maxDatagramSize(quic: WebTransport): number { return typeof size === "number" && size > 0 ? size : 0; } +/** Whether this transport exposes a receive channel for datagrams. */ +export function canReceiveDatagrams(quic: WebTransport): boolean { + return datagrams(quic)?.readable !== undefined; +} + /** Get a datagram reader, or undefined when this browser/transport does not expose one. */ export function datagramReader(quic: WebTransport): ReadableStreamDefaultReader | undefined { const readable = datagrams(quic)?.readable; @@ -34,21 +40,51 @@ export function datagramReader(quic: WebTransport): ReadableStreamDefaultReader< } } -/** Get a datagram writer, or undefined when this browser/transport does not expose one. */ -export function datagramWriter(quic: WebTransport): WritableStreamDefaultWriter | undefined { +/** One subscription's datagram send queue. */ +export interface Sender { + writer: WritableStreamDefaultWriter; + setPriority(order: number): void; + close(): Promise; +} + +/** Opens independent prioritized queues on current transports, sharing the legacy writer otherwise. */ +export function datagramWriter(quic: WebTransport): + | { + open(order: number): Sender; + close(): void; + } + | undefined { const stream = datagrams(quic); if (!stream || maxDatagramSize(quic) === 0) return undefined; - - try { - const writable = typeof stream.createWritable === "function" ? stream.createWritable() : stream.writable; - if (!writable) { - console.warn("datagram send disabled: WebTransport datagram writable stream is unavailable"); - return undefined; - } - - return writable.getWriter(); - } catch (err: unknown) { - console.warn("datagram send disabled: failed to open WebTransport datagram writer", err); - return undefined; + const createWritable = stream.createWritable; + if (typeof createWritable === "function") { + return { + open(order) { + const writable = createWritable.call(stream, { sendOrder: order }); + const writer = writable.getWriter(); + return { + writer, + setPriority(order) { + writable.sendOrder = order; + }, + async close() { + try { + await writer.close(); + } finally { + writer.releaseLock(); + } + }, + }; + }, + // Each request owns and closes its own current-API queue. + close() {}, + }; } + if (!stream.writable) return undefined; + const writer = stream.writable.getWriter(); + return { + // The legacy API has one session queue and no per-subscription send order. + open: () => ({ writer, setPriority() {}, async close() {} }), + close: () => writer.releaseLock(), + }; } diff --git a/js/net/src/fill.test.ts b/js/net/src/fill.test.ts index a69f70a546e..8376243bc62 100644 --- a/js/net/src/fill.test.ts +++ b/js/net/src/fill.test.ts @@ -647,33 +647,6 @@ test("fill: a reader parked on a miss asks once a handler opens", async () => { broadcast.close(); }); -test("fill: a sequence the track's datagram took is never asked, even once a reader read it", async () => { - const broadcast = new BroadcastProducer(); - const track = broadcast.createTrack("video", { maxAge: null }); - // Subscribed first: a datagram is not replayed to a later reader. - const reader = track.subscribe(history); - track.insertDatagram(2n, undefined, enc.encode("datagram 2")); - const group = track.createGroup(5n); - group.writeFrame({ payload: enc.encode("live 5") }); - group.finish(); - // The reader takes the datagram out of its own buffer; the track still holds it. - expect((await reader.recvDatagram())?.sequence).toBe(2n); - - const requests = track.requests(); - const ordered = reader.ordered(); - void ordered.nextGroup(); - const fills = [await requests.requestedFill(), await requests.requestedFill()]; - expect(fills.map((fill) => [fill?.start, fill?.end])).toEqual([ - [0n, 2n], - [3n, 5n], - ]); - - for (const fill of fills) fill?.close(); - reader.close(); - requests.close(); - broadcast.close(); -}); - test("fill: a reader joins a fill still served after its handler's handle closed", async () => { const { broadcast, track } = resumed(5n); const requests = track.requests(); diff --git a/js/net/src/flate/group.test.ts b/js/net/src/flate/group.test.ts index bf9987b9f15..09ac20a4525 100644 --- a/js/net/src/flate/group.test.ts +++ b/js/net/src/flate/group.test.ts @@ -5,7 +5,7 @@ import { hooks } from "../internal.ts"; import { Producer as Track } from "../track.ts"; import { Decoder } from "./decoder.ts"; import { Encoder } from "./encoder.ts"; -import { encodable, stored } from "./group.ts"; +import { encodable } from "./group.ts"; const payload = new TextEncoder().encode("one shared compression window ".repeat(100)); @@ -135,37 +135,6 @@ test("a shared plain read cannot change a later synchronous write's admission", track.close(); }); -test("datagrams buffered before accept share one stored form, each reader its own plain", async () => { - const track = new Track("late-info"); - const a = track.subscribe(); - const b = track.subscribe(); - track.appendDatagram(undefined, payload); - track.accept({ compress: true }); - const first = await a.recvDatagram(); - const second = await b.recvDatagram(); - if (!first || !second) throw new Error("missing datagram"); - expect(first).not.toBe(second); - expect(stored(first)).toBe(stored(second)); - expect(stored(first)?.inflated).toBe(payload.length); - expect(first.payload).toEqual(payload); - expect(second.payload).toEqual(payload); - track.finish(); - track.close(); -}); - -test("a malformed compressed datagram is refused before reaching readers", () => { - const track = new Track("compressed").accept({ compress: true }); - const reader = track.subscribe(); - const payload = new Uint8Array(100); - const slice = new Encoder().frame(payload); - expect(() => hooks.writeDatagram(track, { sequence: 0n, payload: slice }, payload.length + 1)).toThrow( - /declared size/, - ); - expect(hooks.tryRecvDatagram(reader).kind).toBe("idle"); - track.finish(); - track.close(); -}); - test("refresh measures stored slices without inflating a compressed group", () => { const track = new Track("compressed").accept({ compress: true }); const group = track.appendGroup(); diff --git a/js/net/src/flate/group.ts b/js/net/src/flate/group.ts index c2171bcd4e7..0fe9994950d 100644 --- a/js/net/src/flate/group.ts +++ b/js/net/src/flate/group.ts @@ -45,27 +45,6 @@ export function stored(frame: Frame): Stored | undefined { return slices.get(frame); } -/** - * A datagram's stored form: a local payload encoded, or a received slice checked to inflate - * to `inflated` bytes. The checked copy is dropped, since each reader inflates its own. - */ -export function datagramSlice(payload: Uint8Array, inflated?: number): Stored { - if (inflated === undefined) return { payload: new Encoder().frame(payload), inflated: payload.length }; - new Decoder().frame(payload, inflated); - return { payload, inflated }; -} - -/** One reader's lazy plain view of a slice, so no queue retains a copy another reader inflated. */ -export function sliceView(timestamp: Timestamp | undefined, slice: Stored): Frame { - let plain: Uint8Array | undefined; - const frame = new SliceFrame(timestamp, undefined, () => { - plain ??= new Decoder().frame(slice.payload, slice.inflated); - return plain; - }); - slices.set(frame, slice); - return frame; -} - interface Entry { index: number; slice: Stored; diff --git a/js/net/src/group-id.test.ts b/js/net/src/group-id.test.ts index 812c4aba7ef..d025cecad53 100644 --- a/js/net/src/group-id.test.ts +++ b/js/net/src/group-id.test.ts @@ -18,34 +18,49 @@ const url = new URL("https://localhost:4443/test"); const enc = new TextEncoder(); const dec = new TextDecoder(); -/** A negotiated wire: the protocol to offer, an explicit version where no ALPN names one, and whether it carries datagrams. */ +/** A negotiated wire: the protocol to offer, an explicit version where no ALPN names one. */ interface Wire { name: string; protocol: string; version?: number; - datagrams: boolean; + datagrams?: boolean; } const WIRES: Wire[] = [ - { name: "lite-01", protocol: "", version: Lite.Version.DRAFT_01, datagrams: false }, - { name: "lite-02", protocol: "", version: Lite.Version.DRAFT_02, datagrams: false }, - { name: "lite-03", protocol: Lite.ALPN_03, datagrams: false }, - { name: "lite-04", protocol: Lite.ALPN_04, datagrams: false }, + { name: "lite-01", protocol: "", version: Lite.Version.DRAFT_01 }, + { name: "lite-02", protocol: "", version: Lite.Version.DRAFT_02 }, + { name: "lite-03", protocol: Lite.ALPN_03 }, + { name: "lite-04", protocol: Lite.ALPN_04 }, { name: "lite-05", protocol: Lite.ALPN_05, datagrams: true }, { name: "lite-06", protocol: Lite.ALPN_06, datagrams: true }, { name: "lite-07-wip", protocol: Lite.ALPN_07_WIP, datagrams: true }, - { name: "ietf-18", protocol: Ietf.ALPN.DRAFT_18, datagrams: true }, - { name: "ietf-20", protocol: Ietf.ALPN.DRAFT_20, datagrams: true }, - { name: "ietf-22", protocol: Ietf.ALPN.DRAFT_22, datagrams: true }, + { name: "ietf-18", protocol: Ietf.ALPN.DRAFT_18 }, + { name: "ietf-20", protocol: Ietf.ALPN.DRAFT_20 }, + { name: "ietf-22", protocol: Ietf.ALPN.DRAFT_22 }, ]; /** - * Copy `input`'s video track into a broadcast this node produced, keeping every group's and - * datagram's ID, as a relay does. A JS origin refuses to serve a broadcast a session delivered. + * Copy `input`'s video track into a broadcast this node produced, keeping every group's + * ID, as a relay does. A JS origin refuses to serve a broadcast a session delivered. */ -async function forward(input: BroadcastConsumer): Promise { +async function forward(input: BroadcastConsumer, carriesDatagrams: boolean): Promise { const subscription = input.track("video").subscribe({ delay: { max: null } }); const copy = new BroadcastProducer(); + if (carriesDatagrams) { + const reader = input.datagrams("video"); + const writer = copy.createDatagrams("video", await reader.info()); + void (async () => { + try { + for (let datagram = await reader.recv(); datagram; datagram = await reader.recv()) + writer.insert(datagram); + writer.finish(); + } catch (err) { + writer.abort(err instanceof Error ? err : new Error(String(err))); + } finally { + reader.close(); + } + })(); + } const video = copy.createTrack("video", await subscription.info()); void (async () => { for (let group = await subscription.recvGroup(); group; group = await subscription.recvGroup()) { @@ -54,11 +69,6 @@ async function forward(input: BroadcastConsumer): Promise { out.finish(); } })().catch((err: unknown) => video.abort(err instanceof Error ? err : new Error(String(err)))); - void (async () => { - for (let datagram = await subscription.recvDatagram(); datagram; datagram = await subscription.recvDatagram()) { - video.insertDatagram(datagram.sequence, datagram.timestamp, datagram.payload); - } - })().catch(() => {}); return copy; } @@ -78,6 +88,7 @@ for (const wire of WIRES) { const path = Path.from("big"); const publication = source.createBroadcast(path); publication.announce(); + const datagrams = wire.datagrams ? publication.broadcast.createDatagrams("video") : undefined; const track = publication.broadcast.createTrack("video"); const dynamic = relay.requestsRoute(path); @@ -86,14 +97,14 @@ for (const wire of WIRES) { for await (const request of dynamic.requested()) { const input = wireOf(ingest).consume(request.path); inputs.push(input); - request.serve(await forward(input)); + request.serve(await forward(input, wire.datagrams ?? false)); } })(); const remote = wireOf(viewer).consume(path); const reader = remote.track("video").subscribe({ delay: { max: null } }); // Publish once the viewer's subscription reached the publisher through the relay, so - // every group and datagram rides it live rather than racing it. + // every group rides it live rather than racing it. const demand = track.demand(); while (demand.wanted.peek() === undefined) await demand.wanted.changed(); @@ -111,11 +122,15 @@ for (const wire of WIRES) { received?.close(); } - if (wire.datagrams) { - track.insertDatagram(PAST_SAFE + 1n, undefined, enc.encode("datagram")); - const datagram = await reader.recvDatagram(); - expect(datagram?.sequence).toBe(PAST_SAFE + 1n); - expect(dec.decode(datagram?.payload)).toBe("datagram"); + if (datagrams) { + const unreliable = remote.datagrams("video"); + await unreliable.info(); + datagrams.insert({ sequence: PAST_SAFE + 1n, payload: enc.encode("datagram") }); + const received = await unreliable.recv(); + expect(received?.sequence).toBe(PAST_SAFE + 1n); + expect(dec.decode(received?.payload)).toBe("datagram"); + unreliable.close(); + datagrams.finish(); } reader.close(); @@ -133,7 +148,6 @@ test("a group sequence that is not a non-negative bigint is refused where a call // A plain JS caller can still pass a number, which would otherwise mix with bigints later. expect(() => track.createGroup(1 as unknown as bigint)).toThrow(TypeError); expect(() => track.createGroup(-1n)).toThrow(RangeError); - expect(() => track.insertDatagram(1 as unknown as bigint, undefined, new Uint8Array(1))).toThrow(TypeError); expect(track.createGroup(MAX_QUIC).sequence).toBe(MAX_QUIC); const consumer = new BroadcastProducer().consume().track("video"); await expect(consumer.fetchGroup(1 as unknown as bigint)).rejects.toThrow(TypeError); diff --git a/js/net/src/ietf/connection.ts b/js/net/src/ietf/connection.ts index e5aa4d97e5e..c332e1c41b9 100644 --- a/js/net/src/ietf/connection.ts +++ b/js/net/src/ietf/connection.ts @@ -264,8 +264,8 @@ export class Connection implements Established { if (this.#closed) return; this.#closed = true; - this.#publisher.close(); + this.#publisher.close(); this.#auth.close(); // Before the session, whose own close would send a clean code first. diff --git a/js/net/src/ietf/fetch.test.ts b/js/net/src/ietf/fetch.test.ts index 462024d7ff5..d74b8f768c7 100644 --- a/js/net/src/ietf/fetch.test.ts +++ b/js/net/src/ietf/fetch.test.ts @@ -569,19 +569,6 @@ test("a cancelled or unauthorized FETCH withdraws its unanswered history fill", } }); -// Datagrams are never cached, so a FETCH reaching one finds no group there. -test("a FETCH of a datagram is refused", async () => { - for (const version of VERSIONS) { - using fx = serve(version); - publishPairs(fx.track, 3); - expect(fx.track.appendDatagram(new Timestamp(0, Timescale.MILLI), new Uint8Array([1]))).toBe(3n); - writeGroup(fx.track, 4n, ["4-0"]); - - const only = new Fetch({ requestId: FETCH_ID, type: range(version, at(3, 0), at(3, 0)) }); - expect(refused(await fetch(fx, only))).toBe(DOES_NOT_EXIST); - } -}); - // A filter other than a bounded absolute one resolves against Largest Object. test("draft-22: a FETCH filter resolves against Largest Object", async () => { const cases: [Filter.Filter, number[] | number][] = [ diff --git a/js/net/src/ietf/publisher.test.ts b/js/net/src/ietf/publisher.test.ts index 4caef78e3e3..5acaa91461a 100644 --- a/js/net/src/ietf/publisher.test.ts +++ b/js/net/src/ietf/publisher.test.ts @@ -25,7 +25,6 @@ import type { Producer as TrackProducer } from "../track.ts"; import { unwanted } from "../util/demand.ts"; import { wireOf } from "../wire.ts"; import type * as Cluster from "./cluster.ts"; -import { ObjectDatagram } from "./datagram.ts"; import { toRequestCode } from "./error.ts"; import { Fetch, FetchHeader } from "./fetch.ts"; import * as Filter from "./filter.ts"; @@ -2423,39 +2422,6 @@ test("draft-22: an absolute filter trims the range it serves", async () => { } }); -test.each([Version.DRAFT_22])("IETF %s: datagrams honor the absolute object range", async (version) => { - using _clock = manual(); - const fx = fixture(version); - const track = fx.broadcast.createTrack("video", { timescale: Timescale.MILLI }); - const { client } = await runSubscribe( - fx, - new Subscribe({ - requestId: 7n, - trackNamespace: Path.from("test"), - trackName: "video", - subscriberPriority: 0, - filter: { kind: "absolute", startGroup: 0n, startObject: 1n, endGroup: 1n, endObject: 0n }, - }), - ); - const reader = fx.pair.client.datagrams.readable.getReader(); - try { - // Datagrams go out in arrival order, so the in-range one arrives last: receiving it - // first proves both out-of-range ones ahead of it were dropped. - for (const sequence of [0, 2, 1]) - track.insertDatagram(BigInt(sequence), Timestamp.fromMillis(sequence), new Uint8Array([sequence])); - const next = await reader.read(); - if (next.done) throw new Error("the in-range datagram was never sent"); - const datagram = await ObjectDatagram.decode(next.value, version); - expect(datagram.trackAlias).toBe(7n); - expect(datagram.groupId).toBe(1n); - expect(datagram.objectId ?? 0).toBe(0); - } finally { - await reader.cancel(); - client.close(); - fx.close(); - } -}); - /** An absolute filter has a fixed start, so its SUBSCRIBE is no join even over the refresh ratio. */ test("draft-22: an absolute subscribe is no join", async () => { const fx = fixture(); diff --git a/js/net/src/ietf/publisher.ts b/js/net/src/ietf/publisher.ts index ce77b18880c..55034c0c5c3 100644 --- a/js/net/src/ietf/publisher.ts +++ b/js/net/src/ietf/publisher.ts @@ -6,7 +6,6 @@ import type { AuthSession } from "../auth_session.ts"; import { enforceGrant } from "../auth_session.ts"; import type * as broadcast from "../broadcast.ts"; import { Withdrawal } from "../connection/withdrawal.ts"; -import * as DatagramStream from "../datagram_stream.ts"; import { controlTimeout, error, @@ -39,7 +38,6 @@ import { TimeoutError, withTimeout } from "../util/timeout.ts"; import * as Varint from "../varint.ts"; import { type Advertised, type Advertisements, sameInstance, wireOf } from "../wire.ts"; import * as Cluster from "./cluster.ts"; -import { encodeDatagram } from "./datagram.ts"; import { requestReason } from "./error.ts"; import { type Fetch, FetchHeader, FetchOk, type FetchType } from "./fetch.ts"; import * as Filter from "./filter.ts"; @@ -325,10 +323,6 @@ export class Publisher { // Bumped on every change to #joins, which a waiting joining FETCH rechecks. #joinsChanged = new Signal(0); - // The one writer for the outbound datagram stream, shared by every subscription since a - // second getWriter would throw. Undefined when the transport carries no datagrams, which - // turns them off: there is no stream fallback. Released in close(). - #datagramWriter?: WritableStreamDefaultWriter; #hidden: boolean; // Whether a SUBSCRIBE_NAMESPACE's REQUEST_OK counts the NAMESPACEs that follow it (MoQ Active Count). #activeCount: boolean; @@ -414,19 +408,10 @@ export class Publisher { this.#hidden = hidden; this.#activeCount = activeCount; this.#advert = Cluster.advertise(cluster); - this.#datagramWriter = DatagramStream.datagramWriter(quic); } - /** - * Release the datagram writer's lock so the transport can be torn down. The broadcasts - * belong to the origin, which outlives this session. - * - * @internal - */ - close() { - this.#datagramWriter?.releaseLock(); - this.#datagramWriter = undefined; - } + /** The origin owns broadcasts; transport teardown ends this publisher's served requests. @internal */ + close(): void {} /** * Handles an incoming SUBSCRIBE request on a bidi stream. @@ -948,10 +933,6 @@ export class Publisher { } })(); - // Datagrams count toward no stream, so PUBLISH_DONE never waits on them; closing - // the track below ends the loop. - void this.#runDatagrams({ requestId: msg.requestId, track, timescale, publisherPriority, range }); - // Losing the grant ends the subscription, leaving the session alone. let revoke!: () => void; const revoked = new Promise<"revoked">((resolve) => { @@ -1272,51 +1253,6 @@ export class Publisher { } } - /** - * Send a track's datagrams as OBJECT_DATAGRAMs, best-effort, like moq-lite. - * - * One that does not fit the transport's limit is dropped; there is no stream fallback. A - * send failure stops the loop but never fails the subscription. - */ - async #runDatagrams(options: { - requestId: bigint; - track: TrackSubscriber; - /** The units datagrams are stamped in, if any; see {@link RunGroup.timescale}. */ - timescale: Timescale | undefined; - publisherPriority: number; - range: ServeRange; - }) { - const writer = this.#datagramWriter; - if (!writer) return; - const { requestId, track, timescale, publisherPriority, range } = options; - const version = this.#session.version; - - try { - for (;;) { - const datagram = await track.recvDatagram(); - if (!datagram) return; - // A model datagram is object 0 of its group, subject to the reliable slice. - if (groupSlice(range, datagram.sequence).skip > 0) continue; - - const body = await encodeDatagram( - datagram, - { trackAlias: requestId, publisherPriority, timescale }, - version, - ); - const max = DatagramStream.maxDatagramSize(this.#quic); - if (body.byteLength > max) { - console.debug(`dropping oversize datagram: alias=${requestId} size=${body.byteLength} max=${max}`); - continue; - } - - await writer.ready; - await writer.write(body); - } - } catch (err: unknown) { - console.debug(`datagram send stopped: alias=${requestId} error=${reason(error(err))}`); - } - } - /** Preserve the original terminal on a separate stream when its group already sent FIN. */ async #runEndOfTrack(options: { requestId: bigint; diff --git a/js/net/src/ietf/subscriber.test.ts b/js/net/src/ietf/subscriber.test.ts index 9e53c2ae68b..c7d99a205f9 100644 --- a/js/net/src/ietf/subscriber.test.ts +++ b/js/net/src/ietf/subscriber.test.ts @@ -9,13 +9,12 @@ import { createMockTransportPair } from "../mock.ts"; import * as Path from "../path.ts"; import { Reader, Stream } from "../stream.ts"; import { Tail } from "../tail.ts"; -import { Timescale, Timestamp } from "../time.ts"; +import { Timescale } from "../time.ts"; import * as track from "../track.ts"; import * as demandUtil from "../util/demand.ts"; import type * as Cluster from "./cluster.ts"; import { Connection } from "./connection.ts"; -import { ObjectDatagram } from "./datagram.ts"; -import { encodeObjectExtensions, type GroupFlags, Group as GroupMessage } from "./object.ts"; +import { type GroupFlags, Group as GroupMessage } from "./object.ts"; import { Parameters } from "./parameters.ts"; import type { Properties } from "./properties.ts"; import { PublishNamespace, PublishNamespaceUpdate } from "./publish_namespace.ts"; @@ -1456,63 +1455,6 @@ test("object extension limit accepts 64 KiB and stops one byte over before readi } }); -/** - * An OBJECT_DATAGRAM at object 0 is a datagram at its Group ID; anything the model cannot - * carry is dropped like a lost datagram, and a malformed one ends the session. - */ -test("an object datagram is a datagram group", async () => { - const pair = createMockTransportPair(ALPN.DRAFT_18); - const subscriber = new Subscriber({ session: new NativeSession(pair.server, VERSION, true), quic: pair.server }); - const track = subscriber.consume(Path.from("room")).track("video").subscribe(); - - const peer = await nextStream(pair.client); - if (!peer) throw new Error("the subscriber never opened a subscribe stream"); - expect(await peer.reader.u53()).toBe(Subscribe.id); - await Subscribe.decode(peer.reader, VERSION); - await peer.writer.u53(SubscribeOk.id); - await new SubscribeOk({ - trackAlias: ALIAS, - properties: { timescale: Timescale.MILLI }, - }).encode(peer.writer, VERSION); - await track.info(); - - const receiving = subscriber.runDatagrams(); - const writer = pair.client.datagrams.writable.getWriter(); - const send = (fields: Partial[0]>) => - writer.write( - new ObjectDatagram({ - trackAlias: ALIAS, - groupId: 4n, - endOfGroup: true, - body: { payload: new TextEncoder().encode("no") }, - ...fields, - }).encode(VERSION), - ); - - // Past object 0, an unbound alias, a status other than Normal, and no Timestamp on a timed track. - await send({ objectId: 1n }); - await send({ trackAlias: ALIAS + 1n }); - await send({ endOfGroup: false, body: { status: 3 } }); - await send({ groupId: 5n, objectId: 0n }); - const properties = encodeObjectExtensions(Timestamp.fromMillis(1234), Timescale.MILLI, VERSION); - await send({ groupId: 9n, objectId: 0n, publisherPriority: 7, properties, body: { payload: Uint8Array.of(1) } }); - - const datagram = await track.recvDatagram(); - expect(datagram?.sequence).toBe(9n); - expect(datagram?.payload).toEqual(Uint8Array.of(1)); - expect(datagram?.timestamp?.as(Timescale.MILLI)).toBe(1234); - - // A status cannot end the group. - await writer.write(Uint8Array.of(0x22, Number(ALIAS), 4, 0, 0)); - expect( - await receiving.then( - () => undefined, - (err: unknown) => err, - ), - ).toBeInstanceOf(ProtocolViolation); - track.close(); -}); - test("a more specific unnamed announcement replaces the consumed source", async () => { const pair = createMockTransportPair(ALPN.DRAFT_18); const subscriber = new Subscriber({ session: new NativeSession(pair.server, VERSION, true) }); diff --git a/js/net/src/ietf/subscriber.ts b/js/net/src/ietf/subscriber.ts index f2fde65d173..a97fbc8d720 100644 --- a/js/net/src/ietf/subscriber.ts +++ b/js/net/src/ietf/subscriber.ts @@ -27,9 +27,9 @@ import { finalSequence, hiddenBelow, hooks, maxDelayOf, scopeCaptures, scopeHead import type { Lister } from "../listing.ts"; import * as Path from "../path.ts"; import { receiveFrame, skipPayload } from "../payload.ts"; -import { type Cursor, Reader, type Stream } from "../stream.ts"; +import type { Cursor, Reader, Stream } from "../stream.ts"; import { Tail } from "../tail.ts"; -import type { Timescale, Timestamp } from "../time.ts"; +import type { Timescale } from "../time.ts"; import type * as track from "../track.ts"; import { untilAborted } from "../util/abort.ts"; import { closeUnwanted } from "../util/announce.ts"; @@ -48,14 +48,7 @@ import { isDraft20 } from "./filter.ts"; import { Join } from "./join.ts"; import * as Message from "./message.ts"; import * as Namespace from "./namespace.ts"; -import { - decodeFetchObject, - decodeObjectTime, - EndOfTrackProperties, - Frame, - type Group as GroupMessage, - ObjectIdGap, -} from "./object.ts"; +import { decodeFetchObject, EndOfTrackProperties, Frame, type Group as GroupMessage, ObjectIdGap } from "./object.ts"; import { Parameters } from "./parameters.ts"; import { DEFAULT_WIRE, fromWire, toWire } from "./priority.ts"; import { type Publish, PublishDone, PublishDoneStatus, publishDoneClean } from "./publish.ts"; @@ -703,6 +696,13 @@ export class Subscriber { void this.#runSubscribe(path, request, instance.validity?.signal); } })(); + void (async () => { + for (;;) { + const request = await wireOf(consumer).requestedDatagrams(); + if (!request) return; + request.reject(new NotFound("datagram tracks over moq-transport")); + } + })(); return this.#consumes.insert(path, instance, consumer); } @@ -2154,67 +2154,10 @@ export class Subscriber { } } - /** - * Deliver one OBJECT_DATAGRAM as a datagram on its subscription's track: a single-frame - * group at the Group ID. - * - * One the model cannot carry is dropped like any lost datagram: an Object past ID 0 (the - * group would need a second object), a status other than Normal, an alias that is not - * bound yet (the draft lets us drop rather than buffer), or no Timestamp on a track that - * declared a timescale. - */ + // Stream subscriptions never admit datagrams; the separate IETF datagram-track adapter + // owns mapping DATAGRAMS demand onto SUBSCRIBE/OBJECT_DATAGRAM. async #recvDatagram(data: Uint8Array): Promise { - const version = this.#session.version; - const datagram = await ObjectDatagram.decode(data, version); - const { trackAlias: alias, groupId: sequence } = datagram; - - if ((datagram.objectId ?? 0n) !== 0n) { - console.debug(`dropping a datagram past object 0: alias=${alias} group=${sequence}`); - return; - } - let payload: Uint8Array; - if ("status" in datagram.body) { - if (datagram.body.status !== 0) { - console.debug( - `dropping a datagram status: alias=${alias} group=${sequence} status=${datagram.body.status}`, - ); - return; - } - payload = new Uint8Array(); - } else { - payload = datagram.body.payload; - } - - const subscription = this.#aliases.peek(alias); - if (!subscription) { - console.debug(`dropping a datagram for an unbound alias: alias=${alias} group=${sequence}`); - return; - } - - // Like a subgroup object: a track that declared no timescale is untimed. - const timescale = this.#timescales.get(alias); - let timestamp: Timestamp | undefined; - if (timescale !== undefined && datagram.properties !== undefined) { - try { - timestamp = await new Reader(undefined, datagram.properties, version).decode((c) => - decodeObjectTime(c, timescale), - ); - } catch (err: unknown) { - throw new ProtocolViolation(`malformed OBJECT_DATAGRAM properties: ${reason(error(err))}`, { - cause: err, - }); - } - } - if (timescale !== undefined && timestamp === undefined) { - console.debug(`dropping an unstamped datagram: alias=${alias} group=${sequence}`); - return; - } - - try { - subscription.track.insertDatagram(sequence, timestamp, payload); - } catch (err: unknown) { - console.debug(`dropping datagram: alias=${alias} group=${sequence} error=${reason(error(err))}`); - } + await ObjectDatagram.decode(data, this.#session.version); } } diff --git a/js/net/src/index.ts b/js/net/src/index.ts index 4a2de7943aa..f22234d3edc 100644 --- a/js/net/src/index.ts +++ b/js/net/src/index.ts @@ -17,6 +17,8 @@ export * as Bandwidth from "./bandwidth_api.ts"; export * as Broadcast from "./broadcast.ts"; /** A reconnecting, shareable handle on a MoQ session. */ export { Connection } from "./connection/index.ts"; +/** Live datagram track producers, consumers, and payloads. */ +export * as Datagram from "./datagram.ts"; /** The codes a session closes with and a stream resets with. */ export { SessionCode, StreamCode } from "./error.ts"; /** Session, stream, and local request errors. */ diff --git a/js/net/src/integration.test.ts b/js/net/src/integration.test.ts index d36509d4ed5..68469b293d7 100644 --- a/js/net/src/integration.test.ts +++ b/js/net/src/integration.test.ts @@ -766,239 +766,6 @@ for (const [protocol, carriesOptIn] of [ }); } -test("integration: lite draft-05 datagram delivery", async () => { - const enc = new TextEncoder(); - const dec = new TextDecoder(); - const pair = createMockTransportPair(Lite.ALPN_05); - const origin = new OriginProducer(); - - const [client, server] = await Promise.all([ - connect(url, { transport: pair.client }), - accept(pair.server, url, { publish: origin.consume() }), - ]); - - // A static track fans datagrams out to whoever subscribes. - const broadcast = publish(origin, Path.from("test")); - const producer = broadcast.createTrack("video", { timescale: Timescale.MILLI }); - - const remote = wireOf(client).consume(Path.from("test")); - const track = remote.track("video").subscribe(); - - // Datagrams aren't cached, so the first few may race the subscription setup. Pump until - // the subscriber receives one, then stop. - const received = track.recvDatagram(); - let stop = false; - const pump = (async () => { - for (let i = 0; !stop; i++) { - producer.appendDatagram(Timestamp.fromMillis(i), enc.encode("dgram")); - await sleepMs(2); - } - })(); - - const got = await received; - stop = true; - await pump; - - expect(got).toBeDefined(); - expect(dec.decode(got?.payload)).toBe("dgram"); - - broadcast.close(); - remote.close(); - client.abort(); - server.abort(); -}); - -test("integration: lite draft-05 datagrams not sent on a non-datagram transport", async () => { - const enc = new TextEncoder(); - // maxDatagramSize 0 simulates a qmux/WebSocket session: the publisher must fall back to - // not sending datagrams (there is no group fallback), while groups still flow. - const pair = createMockTransportPair(Lite.ALPN_05, { datagrams: false }); - const origin = new OriginProducer(); - - const [client, server] = await Promise.all([ - connect(url, { transport: pair.client }), - accept(pair.server, url, { publish: origin.consume() }), - ]); - - const broadcast = publish(origin, Path.from("test")); - const producer = broadcast.createTrack("video", { timescale: Timescale.MILLI }); - - const remote = wireOf(client).consume(Path.from("test")); - const track = remote.track("video").subscribe().ordered(); - const datagrams = remote.track("video").subscribe(); - - // Keep pushing a group (to prove the connection is live) and a datagram (which must be dropped). - let stop = false; - const pump = (async () => { - for (let i = 0; !stop; i++) { - producer.appendDatagram(Timestamp.fromMillis(i), enc.encode("dgram")); - producer.writeFrame(textFrame("group")); - await sleepMs(2); - } - })(); - - // A group arrives, confirming the subscription works over this transport. - const grp = await track.readString(); - expect(grp).toBe("group"); - - // No datagram is ever delivered: recvDatagram stays pending until the timeout wins. - // Its own handle, since the group reads above took the ordered cursor. - const datagram = datagrams.recvDatagram(); - datagram.catch(() => {}); // The track close below settles it; swallow to avoid a stray rejection. - const outcome = await Promise.race([datagram, sleepMs(50).then(() => "timeout" as const)]); - expect(outcome).toBe("timeout"); - - stop = true; - await pump; - - broadcast.close(); - remote.close(); - client.abort(); - server.abort(); -}); - -test("integration: lite draft-05 datagrams sent with standards-track createWritable", async () => { - const enc = new TextEncoder(); - const dec = new TextDecoder(); - const pair = createMockTransportPair(Lite.ALPN_05, { datagramWritable: "createWritable" }); - const origin = new OriginProducer(); - - const [client, server] = await Promise.all([ - connect(url, { transport: pair.client }), - accept(pair.server, url, { publish: origin.consume() }), - ]); - - const broadcast = publish(origin, Path.from("test")); - const producer = broadcast.createTrack("video", { timescale: Timescale.MILLI }); - - const remote = wireOf(client).consume(Path.from("test")); - const track = remote.track("video").subscribe(); - - const received = track.recvDatagram(); - let stop = false; - const pump = (async () => { - for (let i = 0; !stop; i++) { - producer.appendDatagram(Timestamp.fromMillis(i), enc.encode("dgram")); - await sleepMs(2); - } - })(); - - const got = await received; - stop = true; - await pump; - - expect(got).toBeDefined(); - expect(dec.decode(got?.payload)).toBe("dgram"); - - broadcast.close(); - remote.close(); - client.abort(); - server.abort(); -}); - -test("integration: lite draft-05 missing datagram writer does not close streams", async () => { - const enc = new TextEncoder(); - const pair = createMockTransportPair(Lite.ALPN_05, { datagramWritable: "none" }); - const origin = new OriginProducer(); - - const [client, server] = await Promise.all([ - connect(url, { transport: pair.client }), - accept(pair.server, url, { publish: origin.consume() }), - ]); - - const broadcast = publish(origin, Path.from("test")); - const producer = broadcast.createTrack("video", { timescale: Timescale.MILLI }); - - const remote = wireOf(client).consume(Path.from("test")); - const track = remote.track("video").subscribe().ordered(); - - producer.appendDatagram(Timestamp.fromMillis(0), enc.encode("dgram")); - producer.writeFrame(textFrame("group")); - - expect(await track.readString()).toBe("group"); - - broadcast.close(); - remote.close(); - client.abort(); - server.abort(); -}); - -for (const [name, protocol, version] of [ - ["draft-18", Ietf.ALPN.DRAFT_18, undefined], - ["draft-22", Ietf.ALPN.DRAFT_22, undefined], -] as const) { - for (const timed of [true, false]) { - test(`integration: ietf ${name} delivers ${timed ? "timed" : "untimed"} datagrams`, async () => { - const enc = new TextEncoder(); - const dec = new TextDecoder(); - const pair = createMockTransportPair(protocol); - const origin = new OriginProducer(); - - const [client, server] = await Promise.all([ - connect(url, { transport: pair.client }), - accept(pair.server, url, { version, publish: origin.consume() }), - ]); - - const broadcast = publish(origin, Path.from("test")); - const producer = broadcast.createTrack("video", timed ? { timescale: Timescale.MILLI } : {}); - - const remote = wireOf(client).consume(Path.from("test")); - const track = remote.track("video").subscribe().ordered(); - const datagrams = remote.track("video").subscribe(); - - // A group first, so the subscription is serving and its alias is bound on both ends. - producer.writeFrame({ payload: enc.encode("group"), timestamp: timed ? Timestamp.now() : undefined }); - expect(await track.readString()).toBe("group"); - - producer.insertDatagram(7n, timed ? Timestamp.fromMillis(1234) : undefined, enc.encode("dgram")); - const datagram = await withTimeout( - datagrams.recvDatagram(), - Nano.fromMilli(Milli(1000)), - "no datagram arrived", - ); - expect(datagram?.sequence).toBe(7n); - expect(dec.decode(datagram?.payload)).toBe("dgram"); - if (timed) { - expect(datagram?.timestamp?.as(Timescale.MILLI)).toBe(1234); - } else { - expect(datagram?.timestamp).toBeUndefined(); - } - - broadcast.close(); - remote.close(); - client.abort(); - server.abort(); - }); - } -} - -test("integration: lite draft-05 missing datagram reader does not close streams", async () => { - const enc = new TextEncoder(); - const pair = createMockTransportPair(Lite.ALPN_05, { datagramReadable: false }); - const origin = new OriginProducer(); - - const [client, server] = await Promise.all([ - connect(url, { transport: pair.client }), - accept(pair.server, url, { publish: origin.consume() }), - ]); - - const broadcast = publish(origin, Path.from("test")); - const producer = broadcast.createTrack("video", { timescale: Timescale.MILLI }); - - const remote = wireOf(client).consume(Path.from("test")); - const track = remote.track("video").subscribe().ordered(); - - producer.appendDatagram(Timestamp.fromMillis(0), enc.encode("dgram")); - producer.writeFrame(textFrame("group")); - - expect(await track.readString()).toBe("group"); - - broadcast.close(); - remote.close(); - client.abort(); - server.abort(); -}); - /** A stream reset as a transport delivers one: the peer's code, and nothing else useful. */ class Reset extends Error { readonly source = "stream" as const; diff --git a/js/net/src/internal.ts b/js/net/src/internal.ts index cd477c7a208..b038b8445c0 100644 --- a/js/net/src/internal.ts +++ b/js/net/src/internal.ts @@ -8,7 +8,7 @@ import { type Instant, Milli } from "@moq/clock"; import type { Dispose, GetPromise, Getter } from "@moq/signals"; import type { Consumer as BroadcastConsumer, Producer as BroadcastProducer } from "./broadcast.ts"; -import type { Datagram } from "./datagram.ts"; +import type * as datagram from "./datagram.ts"; import type { Delivered } from "./delivered.ts"; import { toStreamCode } from "./error.ts"; import type { Frame, FrameConsumer, Consumer as GroupConsumer, Producer as GroupProducer } from "./group.ts"; @@ -313,13 +313,6 @@ export interface Replacement { discard(): void; } -/** A datagram read that does not wait. */ -export type RecvDatagram = - | { kind: "datagram"; datagram: Datagram } - | { kind: "idle" } - | { kind: "done" } - | { kind: "error"; error: Error }; - /** * Which groups of a track's live feed to ask whether one may still land: one the feed has not * accounted for, the publisher did not drop, and the feed reaches, in a gap a later arrival @@ -352,11 +345,6 @@ export interface ResumeSource { tryGroup(ordered: boolean, delivered: Delivered, dispositions: boolean): Recv; /** The next run no copy may still deliver, reported once as a `drop`, without taking a group. */ tryDrop(delivered: Delivered): (Recv & { kind: "drop" }) | undefined; - /** - * The next datagram in arrival order, without waiting. `delivered` is the group record, so a - * copy that still holds a group for a later group read is kept. - */ - tryDatagram(delivered: Delivered): RecvDatagram; /** * Whether the reader's bounded `window` has nothing left to deliver, judged by the serving * copy against what it and the logical reader (`delivered`) handed out. A copy that may still @@ -446,9 +434,6 @@ export interface FillStarts { fillsFromStart?: boolean; } -/** Maximum buffered datagrams per subscriber; mirrors Rust's bounded send buffer. */ -export const MAX_DATAGRAMS = 64; - /** The first group an exclusive track end excludes whole: one past the last group that may exist. */ export function finalSequence(end: Position): bigint { return end.group + (end.frame === 0 ? 0n : 1n); @@ -456,6 +441,14 @@ export function finalSequence(end: Position): bigint { /** Hooks assigned in static blocks by the owning class. */ export const hooks: { + /** Make a request for an unanswered datagram track. */ + datagramRequest: (producer: datagram.Producer, pending: Set) => datagram.Request; + /** End a received datagram track if its last consumer left. */ + closeUnusedDatagrams: (producer: datagram.Producer) => boolean; + /** Follow this consumer's subscriber priority for transport scheduling. */ + datagramPriority: (consumer: datagram.Consumer) => Getter; + /** Settles once this consumer is closed, the end of whatever request it serves. */ + datagramClosed: (consumer: datagram.Consumer) => GetPromise; /** * Declare the end a wire subscription received: repeating it is a no-op, and a mid-group * end may refine the group boundary it falls just before. Throws `ProtocolViolation` on @@ -597,15 +590,10 @@ export const hooks: { peekGroup: (subscriber: Subscriber, ordered: boolean, floor?: bigint) => bigint | undefined; /** Take the next group in sequence order without waiting, as {@link tryRecvGroup} does in arrival order. */ tryNextGroup: (subscriber: Subscriber, floor?: bigint) => Recv; - /** Wake once a subscriber's datagram cursor may read differently; fires at most once. */ - datagramChanged: (subscriber: Subscriber, fn: () => void) => Dispose; - /** Take the next datagram in arrival order without waiting. */ - tryRecvDatagram: (subscriber: Subscriber) => RecvDatagram; /** Whether a buffered group in `[start, end)` that `delivered` lacks still waits to be read. */ hasUnreadGroup: (subscriber: Subscriber, start: bigint, end: bigint | undefined, delivered: Delivered) => boolean; /** - * Whether a subscriber's track ended and it holds nothing left to read: no datagram within its - * cursor, and no group in `[start, end)` that `delivered` lacks. + * Whether a subscriber's track ended and it holds nothing left to read: no group in `[start, end)` that `delivered` lacks. */ spent: (subscriber: Subscriber, start: bigint, end: bigint | undefined, delivered: Delivered) => boolean; /** @@ -671,8 +659,6 @@ export const hooks: { * is refused. Throws `TimestampMismatch` if a buffered frame already disagrees. */ bindGroupTimed: (group: GroupProducer, track: Info) => void; - /** Admit a datagram in its stored wire form. */ - writeDatagram: (producer: Producer, datagram: Datagram, inflated?: number) => void; /** Bind the group to its immutable track codec. */ bindGroupCompression: (group: GroupProducer, compressed: boolean) => void; /** Admit a received stored slice without recompressing it. */ @@ -754,6 +740,18 @@ export const hooks: { /** Whether a broadcast handle was {@link stampUnlisted marked unlisted}. */ unlisted: (target: BroadcastConsumer) => boolean; } = { + datagramRequest: () => { + throw new Error("datagram module not loaded"); + }, + closeUnusedDatagrams: () => { + throw new Error("datagram module not loaded"); + }, + datagramPriority: () => { + throw new Error("datagram module not loaded"); + }, + datagramClosed: () => { + throw new Error("datagram module not loaded"); + }, endTrack: () => { throw new Error("track.ts not loaded"); }, @@ -859,18 +857,12 @@ export const hooks: { unreport: () => { throw new Error("track.ts not loaded"); }, - tryRecvDatagram: () => { - throw new Error("track.ts not loaded"); - }, deliveredRuns: () => { throw new Error("track.ts not loaded"); }, handedOut: () => { throw new Error("track.ts not loaded"); }, - datagramChanged: () => { - throw new Error("track.ts not loaded"); - }, hasUnreadGroup: () => { throw new Error("track.ts not loaded"); }, @@ -925,9 +917,6 @@ export const hooks: { askFetch: () => { throw new Error("track.ts not loaded"); }, - writeDatagram: () => { - throw new Error("track.ts not loaded"); - }, bindGroupCompression: () => { throw new Error("group.ts not loaded"); }, diff --git a/js/net/src/lite/compression.test.ts b/js/net/src/lite/compression.test.ts index 13f556f0ce1..43d7d1c9a41 100644 --- a/js/net/src/lite/compression.test.ts +++ b/js/net/src/lite/compression.test.ts @@ -15,6 +15,7 @@ for (const downstream of [ALPN_06, ALPN_07_WIP]) { const destination = new Producer(); const broadcastPublication = source.createBroadcast(Path.from("compressed")); const broadcast = broadcastPublication.broadcast; + const datagrams = broadcast.createDatagrams("data"); const track = broadcast.createTrack("data", { compress: true }); broadcastPublication.announce(); const peers: { abort(): void }[] = []; @@ -55,11 +56,15 @@ for (const downstream of [ALPN_06, ALPN_07_WIP]) { const fetched = await remote.fetchGroup(0n); for (const payload of payloads) expect((await fetched.readFrame())?.payload).toEqual(payload); expect(await fetched.readFrame()).toBeUndefined(); - // Each datagram has a cold window; no lost or reordered predecessor is needed. + // Datagram payloads stay plain beside the compressed stream track. + const unreliable = active.datagrams("data"); + await unreliable.info(); + while (datagrams.priority.peek() === undefined) await datagrams.priority.changed(); for (const payload of [new Uint8Array(100).fill(4), new Uint8Array(100).fill(8)]) { - track.appendDatagram(undefined, payload); - expect((await reader.recvDatagram())?.payload).toEqual(payload); + datagrams.append(undefined, payload); + expect((await unreliable.recv())?.payload).toEqual(payload); } + unreliable.close(); reader.close(); fetched.close(); received?.close(); diff --git a/js/net/src/lite/connection.ts b/js/net/src/lite/connection.ts index 46ac27753d6..d5aa87581aa 100644 --- a/js/net/src/lite/connection.ts +++ b/js/net/src/lite/connection.ts @@ -18,6 +18,7 @@ import { withTimeout } from "../util/timeout.ts"; import { registerWire } from "../wire.ts"; import { AnnounceRequest } from "./announce.ts"; import { LiteAuthWire } from "./auth.ts"; +import { Datagrams } from "./datagram.ts"; import { closeViolation } from "./error.ts"; import { Fetch } from "./fetch.ts"; import { Goaway } from "./goaway.ts"; @@ -33,6 +34,7 @@ import { Tracks as TracksMessage } from "./tracks.ts"; import { hasAuth, hasDatagrams, + hasDatagramTracks, hasGoaway, hasPadding, hasProbeRtt, @@ -430,6 +432,14 @@ export class Connection implements Established { await this.#publisher.runSubscribe(msg, stream); })(), ); + } else if (typ === StreamId.Datagrams) { + if (!hasDatagramTracks(this.#version)) throw new Error("DATAGRAMS is not supported on this version"); + await this.#publisher.owe( + (async () => { + const msg = await Datagrams.decode(stream.reader, this.#version); + await this.#publisher.runDatagrams(msg, stream); + })(), + ); } else if (typ === StreamId.Fetch) { if (!hasTrackStream(this.#version)) throw new Error("FETCH is not supported on this version"); await this.#publisher.owe( diff --git a/js/net/src/lite/datagram-tracks.test.ts b/js/net/src/lite/datagram-tracks.test.ts new file mode 100644 index 00000000000..c5138d7c0f3 --- /dev/null +++ b/js/net/src/lite/datagram-tracks.test.ts @@ -0,0 +1,283 @@ +import { expect, test } from "bun:test"; +import { accept, connect } from "../connection/index.ts"; +import { createMockTransportPair } from "../mock.ts"; +import { Producer as Origin } from "../origin.ts"; +import * as Path from "../path.ts"; +import { sendOrder } from "../priority.ts"; +import { Timescale, Timestamp } from "../time.ts"; +import { wireOf } from "../wire.ts"; +import { ALPN_04, ALPN_05, ALPN_06, ALPN_07_WIP } from "./version.ts"; + +const url = new URL("https://localhost/datagrams"); +const path = Path.from("live"); + +for (const alpn of [ALPN_05, ALPN_06, ALPN_07_WIP]) { + for (const streams of [false, true]) { + test(`${alpn}: ${streams ? "same-name kinds" : "datagram-only track"} delivery`, async () => { + const source = new Origin(); + const publication = source.createBroadcast(path); + const datagrams = publication.broadcast.createDatagrams("audio", { + timescale: Timescale.MICRO, + priority: -7, + }); + const track = streams + ? publication.broadcast.createTrack("audio", { timescale: Timescale.MILLI }) + : undefined; + publication.announce(); + const pair = createMockTransportPair(alpn); + const peers = await Promise.all([ + connect({ url, transport: pair.client }), + accept({ url, transport: pair.server, publish: source.consume() }), + ]); + const remote = wireOf(peers[0]).consume(path); + const reader = remote.datagrams("audio"); + reader.setPriority(17); + try { + const info = await reader.info(); + while (datagrams.priority.peek() !== 17) await datagrams.priority.changed(); + expect(info.priority).toBe(streams && alpn !== ALPN_07_WIP ? 0 : -7); + const stamp = Timestamp.fromMicros(1234567); + const sequence = (1n << 53n) + 1n; + datagrams.insert({ sequence, timestamp: stamp, payload: new Uint8Array([7, 8]) }); + const received = await reader.recv(); + expect(received?.sequence).toBe(sequence); + expect(received?.timestamp?.asMicros()).toBe(streams && alpn !== ALPN_07_WIP ? 1234000 : 1234567); + expect(received?.payload).toEqual(new Uint8Array([7, 8])); + if (track) { + const grouped = remote.track("audio").subscribe(); + track.writeFrame({ timestamp: Timestamp.fromMillis(1), payload: new Uint8Array([9]) }); + expect((await grouped.recvGroup())?.sequence).toBe(0n); + grouped.close(); + } + reader.setPriority(-9); + while (datagrams.priority.peek() !== -9) await datagrams.priority.changed(); + datagrams.finish(); + track?.finish(); + expect(await reader.recv()).toBeUndefined(); + } finally { + reader.close(); + remote.close(); + for (const peer of peers) peer.abort(); + source.close(); + } + }); + } +} + +for (const options of [{ datagrams: false }, { datagramReadable: false }]) { + test(`unsupported datagram receive setup is refused (${JSON.stringify(options)})`, async () => { + const pair = createMockTransportPair(ALPN_07_WIP, options); + const [client, server] = await Promise.all([ + connect({ url, transport: pair.client }), + accept({ url, transport: pair.server }), + ]); + const remote = wireOf(client).consume(path); + const reader = remote.datagrams("missing"); + try { + await expect(reader.info()).rejects.toThrow("does not support datagram"); + } finally { + reader.close(); + remote.close(); + client.abort(); + server.abort(); + } + }); +} + +test("lite-04 refuses datagram tracks at setup", async () => { + const pair = createMockTransportPair(ALPN_04); + const [client, server] = await Promise.all([ + connect({ url, transport: pair.client }), + accept({ url, transport: pair.server }), + ]); + const remote = wireOf(client).consume(path); + const reader = remote.datagrams("missing"); + try { + await expect(reader.info()).rejects.toThrow("does not support datagram"); + } finally { + reader.close(); + remote.close(); + client.abort(); + server.abort(); + } +}); + +test("lite-07 bounds and prioritizes independent createWritable datagram queues", async () => { + const source = new Origin(); + const publication = source.createBroadcast(path); + const datagrams = publication.broadcast.createDatagrams("audio"); + publication.broadcast.createTrack("audio", { compress: true }); + publication.announce(); + const other = publication.broadcast.createDatagrams("other", { priority: 7 }); + const pair = createMockTransportPair(ALPN_07_WIP, { datagramWritable: "createWritable" }); + Object.defineProperty(pair.server.datagrams, "maxDatagramSize", { value: 900 }); + const [client, server] = await Promise.all([ + connect({ url, transport: pair.client }), + accept({ url, transport: pair.server, publish: source.consume() }), + ]); + const remote = wireOf(client).consume(path); + const reader = remote.datagrams("audio"); + const second = remote.datagrams("other"); + reader.setPriority(19); + second.setPriority(31); + try { + await reader.info(); + while (datagrams.priority.peek() !== 19) await datagrams.priority.changed(); + await second.info(); + other.append(undefined, new Uint8Array([1])); + await second.recv(); + expect(pair.server.sendStreams.datagrams).toHaveLength(2); + const queue = pair.server.sendStreams.datagrams.find( + (queue) => queue.sendOrder === sendOrder({ priority: 19 }), + ); + expect(queue).toBeDefined(); + reader.setPriority(-11); + while (datagrams.priority.peek() !== -11) await datagrams.priority.changed(); + expect(queue?.sendOrder).toBe(sendOrder({ priority: -11 })); + expect(pair.server.sendStreams.datagrams.find((other) => other !== queue)?.sendOrder).toBe( + sendOrder({ priority: 31, publisher: 7 }), + ); + // Type, Subscribe ID, and Sequence each occupy one byte on this untimed track. + datagrams.append(undefined, new Uint8Array(898)); + datagrams.append(undefined, new Uint8Array(897)); + expect((await reader.recv())?.sequence).toBe(1n); + } finally { + second.close(); + reader.close(); + remote.close(); + client.abort(); + server.abort(); + source.close(); + } +}); + +for (const alpn of [ALPN_05, ALPN_06]) + for (const first of ["stream", "datagram"]) { + test(`${alpn}: datagram readers rejoin and outlive streams (${first} first)`, async () => { + const source = new Origin(); + const publication = source.createBroadcast(path); + const datagrams = publication.broadcast.createDatagrams("audio"); + const track = publication.broadcast.createTrack("audio"); + publication.announce(); + const pair = createMockTransportPair(alpn); + const [client, server] = await Promise.all([ + connect({ url, transport: pair.client }), + accept({ url, transport: pair.server, publish: source.consume() }), + ]); + const remote = wireOf(client).consume(path); + let reader = first === "datagram" ? remote.datagrams("audio") : undefined; + const grouped = remote.track("audio").subscribe(); + try { + await grouped.info(); + track.writeFrame({ payload: new Uint8Array([1]) }); + (await grouped.recvGroup())?.close(); + reader ??= remote.datagrams("audio"); + await reader.info(); + reader.setPriority(37); + while (datagrams.priority.peek() !== 37) await datagrams.priority.changed(); + reader.close(); + while (datagrams.priority.peek() !== 0) await datagrams.priority.changed(); + reader = remote.datagrams("audio"); + await reader.info(); + reader.setPriority(37); + while (datagrams.priority.peek() !== 37) await datagrams.priority.changed(); + grouped.close(); + datagrams.append(undefined, new Uint8Array([2])); + expect((await reader.recv())?.payload).toEqual(new Uint8Array([2])); + datagrams.finish(); + track.finish(); + expect(await reader.recv()).toBeUndefined(); + } finally { + reader?.close(); + grouped.close(); + remote.close(); + client.abort(); + server.abort(); + source.close(); + } + }); + } + +test("lite-07 forwards a priority change made while the previous DATAGRAMS_UPDATE writes", async () => { + const source = new Origin(); + const publication = source.createBroadcast(path); + const datagrams = publication.broadcast.createDatagrams("audio"); + publication.announce(); + const pair = createMockTransportPair(ALPN_07_WIP); + const [client, server] = await Promise.all([ + connect({ url, transport: pair.client }), + accept({ url, transport: pair.server, publish: source.consume() }), + ]); + const remote = wireOf(client).consume(path); + const reader = remote.datagrams("audio"); + try { + await reader.info(); + // Land the second change at each point of the first update's write. + for (let yields = 0; yields < 40; yields++) { + reader.setPriority(yields + 1); + for (let i = 0; i < yields; i++) await Promise.resolve(); + reader.setPriority(-(yields + 1)); + // Without a recheck after the write, the publisher stays at the first value and this hangs. + while (datagrams.priority.peek() !== -(yields + 1)) await datagrams.priority.changed(); + } + } finally { + reader.close(); + remote.close(); + client.abort(); + server.abort(); + source.close(); + } +}); + +test("lite-06: a request cancelled behind the shared datagram writer sends nothing", async () => { + const source = new Origin(); + const publication = source.createBroadcast(path); + const first = publication.broadcast.createDatagrams("a"); + const second = publication.broadcast.createDatagrams("b"); + publication.announce(); + const pair = createMockTransportPair(ALPN_06); + // The legacy API's one session queue, stalled on its first write. + const sent: Uint8Array[] = []; + const stall = Promise.withResolvers(); + Object.defineProperty(pair.server.datagrams, "writable", { + value: new WritableStream({ + write(chunk) { + sent.push(chunk); + return sent.length === 1 ? stall.promise : undefined; + }, + }), + }); + const [client, server] = await Promise.all([ + connect({ url, transport: pair.client }), + accept({ url, transport: pair.server, publish: source.consume() }), + ]); + const remote = wireOf(client).consume(path); + const a = remote.datagrams("a"); + const b = remote.datagrams("b"); + const settle = async () => { + for (let i = 0; i < 20; i++) await new Promise((resolve) => setImmediate(resolve)); + }; + try { + await Promise.all([a.info(), b.info()]); + while (first.priority.peek() === undefined || second.priority.peek() === undefined) + await Promise.race([first.priority.changed(), second.priority.changed()]); + first.append(undefined, Uint8Array.of(0xaa)); + while (sent.length === 0) await settle(); + // The second request now waits on the writer the first one holds. + second.append(undefined, Uint8Array.of(0xbb)); + await settle(); + b.close(); + while (second.priority.peek() !== undefined) await second.priority.changed(); + stall.resolve(); + await settle(); + expect(sent.map((chunk) => chunk.at(-1))).toEqual([0xaa]); + } finally { + stall.resolve(); + a.close(); + b.close(); + remote.close(); + client.abort(); + server.abort(); + source.close(); + } +}); diff --git a/js/net/src/lite/datagram.test.ts b/js/net/src/lite/datagram.test.ts index 9c6d0ba66f9..7a65aa65f5f 100644 --- a/js/net/src/lite/datagram.test.ts +++ b/js/net/src/lite/datagram.test.ts @@ -87,3 +87,31 @@ test("an untimed datagram body is refused before lite-07", () => { expect(() => dg.encode(version)).toThrow("has no untimed datagram"); } }); + +test("lite-07 datagram requests match Rust's size-prefixed fields", async () => { + const { Datagrams, DatagramsInfo, DatagramsUpdate } = await import("./datagram.ts"); + const { Reader, Writer } = await import("../stream.ts"); + const Path = await import("../path.ts"); + const { Timescale } = await import("../time.ts"); + const request = new Datagrams({ id: 7n, broadcast: Path.from("room/alice"), track: "audio", priority: -3 }); + const body = await Writer.collect(Version.DRAFT_07, (writer) => request.encode(writer, Version.DRAFT_07)); + expect([...body]).toEqual([20, 7, 10, ...enc.encode("room/alice"), 0, 5, ...enc.encode("audio"), 125]); + const decoded = await Datagrams.decode(new Reader(undefined, body, Version.DRAFT_07), Version.DRAFT_07); + expect(decoded).toEqual(request); + for (const timescale of [undefined, Timescale.MILLI]) { + const info = new DatagramsInfo({ priority: 2, timescale }); + const body = await Writer.collect(Version.DRAFT_07, (writer) => info.encode(writer, Version.DRAFT_07)); + expect(await DatagramsInfo.decode(new Reader(undefined, body, Version.DRAFT_07), Version.DRAFT_07)).toEqual( + info, + ); + } + const update = new DatagramsUpdate(9); + const updated = await Writer.collect(Version.DRAFT_07, (writer) => update.encode(writer, Version.DRAFT_07)); + expect([...updated]).toEqual([1, 137]); + expect( + await DatagramsUpdate.decodeMaybe(new Reader(undefined, updated, Version.DRAFT_07), Version.DRAFT_07), + ).toEqual(update); + await expect( + Writer.collect(Version.DRAFT_06, (writer) => request.encode(writer, Version.DRAFT_06)), + ).rejects.toThrow("require lite-07"); +}); diff --git a/js/net/src/lite/datagram.ts b/js/net/src/lite/datagram.ts index c067585ad00..4bc7068f18c 100644 --- a/js/net/src/lite/datagram.ts +++ b/js/net/src/lite/datagram.ts @@ -3,14 +3,20 @@ * * @module */ -import { encodeVarint, Reader } from "../stream.ts"; -import { hasCompression, hasPadding, type Version, wireTimescale } from "./version.ts"; + +import type * as Instance from "../instance.ts"; +import * as Path from "../path.ts"; +import { encodeVarint, Reader, type Writer } from "../stream.ts"; +import { Timescale } from "../time.ts"; +import { decodePin, encodePin } from "./instance.ts"; +import * as Message from "./message.ts"; +import { fromWire, toWire } from "./priority.ts"; +import { hasDatagrams, hasDatagramTracks, hasPadding, type Version, wireTimescale } from "./version.ts"; /** - * A QUIC datagram body: `subscribe (i) | sequence (i) | [timestamp (i)] | [inflated (i)] | payload (b)`, with the - * timestamp present only when the subscription's track carries one (see `wireTimescale`), - * and the inflated size only on a compressed lite-07 track. - * Lite-07 prefixes group datagrams with type 0x0; type 0x1 is discarded padding. + * A QUIC datagram body: `subscribe (i) | sequence (i) | [timestamp (i)] | payload (b)`, with the + * timestamp present only when the subscription's track carries one (see `wireTimescale`). + * Lite-07 prefixes datagram-track bodies with type 0x0; type 0x1 is discarded padding. * * The payload runs to the datagram boundary, so unlike a size-prefixed lite message there is no * inner length prefix. Mirrors the Rust `lite::Datagram`; the model counterpart is @@ -22,14 +28,12 @@ export class Datagram { /** Subscribe ID this datagram is delivered on. */ subscribe: bigint; - /** Group sequence number (shared with the track's group namespace). */ + /** The datagram track's sequence, mapped to a group sequence on lite-05/06. */ sequence: bigint; /** Absolute presentation timestamp in the track's wire timescale, or undefined on a track that carries none. */ timestamp: number | undefined; /** The frame payload, delimited by the datagram boundary. */ payload: Uint8Array; - /** Plain payload length on a compressed lite-07 track. */ - inflated?: number; constructor(subscribe: bigint, sequence: bigint, timestamp: number | undefined, payload: Uint8Array) { this.subscribe = subscribe; @@ -40,14 +44,12 @@ export class Datagram { /** Encode the body to a single `Uint8Array` (no length prefix; the datagram boundary delimits it). */ encode(version: Version): Uint8Array { + if (!hasDatagrams(version)) throw new Error("datagrams require lite-05 or newer"); // A version that always carries a timescale has no untimed body. if (this.timestamp === undefined && wireTimescale(version, undefined) !== undefined) { throw new Error(`version ${version} has no untimed datagram`); } - if (this.inflated !== undefined && !hasCompression(version)) - throw new Error("compressed datagram needs lite-07"); - const inflated = this.inflated === undefined ? new Uint8Array() : encodeVarint(this.inflated, version); const subscribe = encodeVarint(this.subscribe, version); const sequence = encodeVarint(this.sequence, version); const timestamp = this.timestamp === undefined ? new Uint8Array() : encodeVarint(this.timestamp, version); @@ -58,7 +60,6 @@ export class Datagram { subscribe.byteLength + sequence.byteLength + timestamp.byteLength + - inflated.length + this.payload.byteLength, ); let offset = 0; @@ -70,8 +71,6 @@ export class Datagram { offset += sequence.byteLength; out.set(timestamp, offset); offset += timestamp.byteLength; - out.set(inflated, offset); - offset += inflated.length; out.set(this.payload, offset); return out; } @@ -83,8 +82,9 @@ export class Datagram { static async decode( data: Uint8Array, version: Version, - framing: (subscribe: bigint) => { timed: boolean; compressed?: boolean }, + framing: (subscribe: bigint) => { timed: boolean }, ): Promise { + if (!hasDatagrams(version)) throw new Error("datagrams require lite-05 or newer"); const r = new Reader(undefined, data, version); if (hasPadding(version)) { const type = await r.u62(); @@ -93,12 +93,120 @@ export class Datagram { } const subscribe = await r.u62(); const sequence = await r.u62(); - const { timed, compressed } = framing(subscribe); + const { timed } = framing(subscribe); const timestamp = timed ? await r.u53() : undefined; - const inflated = hasCompression(version) && compressed ? await r.u53() : undefined; const payload = await r.readAll(); const result = new Datagram(subscribe, sequence, timestamp, payload); - result.inflated = inflated; return result; } } + +function guard(version: Version): void { + if (!hasDatagramTracks(version)) throw new Error("datagram tracks require lite-07 or newer"); +} + +/** The subscriber's first message on a Datagrams Stream (0x9), live from now. */ +export class Datagrams { + /** The shared SUBSCRIBE/DATAGRAMS identifier space. */ + id: bigint; + /** The requested broadcast path. */ + broadcast: Path.Valid; + /** The publisher instance the subscriber expects. */ + instance?: Instance.Valid; + /** The datagram track's name. */ + track: string; + /** The subscriber priority, higher first. */ + priority: number; + + constructor(properties: { + id: bigint; + broadcast: Path.Valid; + instance?: Instance.Valid; + track: string; + priority: number; + }) { + this.instance = properties.instance; + this.id = properties.id; + this.broadcast = properties.broadcast; + this.track = properties.track; + this.priority = properties.priority; + } + + /** Encode the size-prefixed request. */ + async encode(writer: Writer, version: Version): Promise { + guard(version); + await Message.encode(writer, async (w) => { + await w.u62(this.id); + await w.string(Path.encode(this.broadcast)); + await encodePin(w, version, this.instance); + await w.string(this.track); + await w.u8(toWire(this.priority)); + }); + } + + /** Decode the size-prefixed request. */ + static async decode(reader: Reader, version: Version): Promise { + guard(version); + return Message.decode( + reader, + async (r) => + new Datagrams({ + id: await r.u62(), + broadcast: Path.decode(await r.string()), + instance: await decodePin(r, version), + track: await r.string(), + priority: fromWire(await r.u8()), + }), + ); + } +} + +/** A subscriber priority update on a Datagrams Stream. */ +export class DatagramsUpdate { + /** The subscriber's replacement priority. */ + priority: number; + constructor(priority: number) { + this.priority = priority; + } + /** Encode the size-prefixed update. */ + async encode(writer: Writer, version: Version): Promise { + guard(version); + await Message.encode(writer, async (w) => { + await w.u8(toWire(this.priority)); + }); + } + /** Decode an update, or undefined at the subscriber's FIN. */ + static async decodeMaybe(reader: Reader, version: Version): Promise { + guard(version); + return Message.decodeMaybe(reader, async (r) => new DatagramsUpdate(fromWire(await r.u8()))); + } +} + +/** The publisher's sole answer on a Datagrams Stream, fixing timestamp framing. */ +export class DatagramsInfo { + /** Publisher priority, breaking ties at one subscriber priority. */ + priority: number; + /** Timestamp units per second, or undefined for an untimed track. */ + timescale?: Timescale; + constructor(properties: { priority: number; timescale?: Timescale | null }) { + this.priority = properties.priority; + this.timescale = properties.timescale ?? undefined; + } + /** Encode the size-prefixed answer. */ + async encode(writer: Writer, version: Version): Promise { + guard(version); + await Message.encode(writer, async (w) => { + await w.u8(toWire(this.priority)); + await w.u53(this.timescale ?? 0); + }); + } + /** Decode the publisher properties. */ + static async decode(reader: Reader, version: Version): Promise { + guard(version); + return Message.decode(reader, async (r) => { + const priority = fromWire(await r.u8()); + const scale = await r.u53(); + return new DatagramsInfo({ priority, timescale: scale === 0 ? undefined : Timescale(scale) }); + }); + } +} diff --git a/js/net/src/lite/malformed.test.ts b/js/net/src/lite/malformed.test.ts index f81e1e460ef..8a32babc43f 100644 --- a/js/net/src/lite/malformed.test.ts +++ b/js/net/src/lite/malformed.test.ts @@ -132,7 +132,7 @@ for (const [alpn, version] of versions) { for (const [alpn, version] of [[ALPN_04, Version.DRAFT_04], ...versions] as const) { // An unassigned type, and the Tracks and Auth Streams on versions that predate them. - for (const typ of version === Version.DRAFT_07 ? [9] : [9, StreamId.Tracks, StreamId.Auth]) { + for (const typ of version === Version.DRAFT_07 ? [10] : [9, StreamId.Tracks, StreamId.Auth]) { test(`${alpn}: unknown stream type ${typ} resets only its stream`, async () => { const f = fixture(alpn, version); try { diff --git a/js/net/src/lite/publisher.test.ts b/js/net/src/lite/publisher.test.ts index 1202b6d16d0..80cdfe958f0 100644 --- a/js/net/src/lite/publisher.test.ts +++ b/js/net/src/lite/publisher.test.ts @@ -17,7 +17,6 @@ import { Timescale, Timestamp } from "../time.ts"; import { Subscriber as TrackSubscriber } from "../track.ts"; import { wireOf } from "../wire.ts"; import { type AnnounceBroadcast, AnnounceOk, AnnounceRequest, decodeAnnounceBroadcast } from "./announce.ts"; -import { Datagram as DatagramMessage } from "./datagram.ts"; import { Fetch } from "./fetch.ts"; import { Group as GroupMessage } from "./group.ts"; import { Probe as ProbeMessage } from "./probe.ts"; @@ -1200,104 +1199,6 @@ for (const version of [Version.DRAFT_06, Version.DRAFT_07]) { }); } -for (const version of [Version.DRAFT_06, Version.DRAFT_07]) { - for (const transport of [900, 1400]) { - test(`lite ${version}: datagram bodies fit the wire and ${transport}-byte transport`, async () => { - const installed = new Manual().install(); - const sub = await servedSubscription({ version, startGroup: 0n, maxDatagramSize: transport }); - const reader = sub.datagrams.getReader(); - const max = Math.min(1200, transport); - try { - // Lite-07's type adds one byte to the three one-byte group fields. - const header = version === Version.DRAFT_07 ? 4 : 3; - sub.track.insertDatagram(0n, Timestamp.fromMillis(0), new Uint8Array(max - header + 1)); - sub.track.insertDatagram(1n, Timestamp.fromMillis(1), new Uint8Array(max - header)); - const next = await reader.read(); - if (next.done) throw new Error("datagrams ended early"); - expect(next.value.byteLength).toBe(max); - expect((await DatagramMessage.decode(next.value, version, () => ({ timed: true })))?.sequence).toBe(1n); - } finally { - await reader.cancel(); - await sub.close(); - await sub.serving; - installed.close(); - } - }); - } -} - -test.each([Version.DRAFT_06, Version.DRAFT_07])("lite %s: datagrams honor group and frame bounds", async (version) => { - const clock = new Manual(); - const installed = clock.install(); - const sub = await servedSubscription({ version, startGroup: 0n, startFrame: 1, endGroup: 1n, endFrame: 0 }); - const reader = sub.datagrams.getReader(); - // Datagrams go out in arrival order, so each batch inserts its out-of-range ones ahead of - // the last in-range one: reading exactly the in-range ones proves the rest were dropped. - const send = (sequences: number[]) => { - for (const sequence of sequences) - sub.track.insertDatagram(BigInt(sequence), Timestamp.fromMillis(sequence), new Uint8Array([sequence])); - }; - const recv = async () => { - const next = await reader.read(); - if (next.done) throw new Error("datagrams ended early"); - return (await DatagramMessage.decode(next.value, version, () => ({ timed: true })))?.sequence; - }; - // The publisher applies an update's datagram bounds before forwarding its group range, and - // each update here moves the range's end. - const applied = async (end: bigint) => { - let ranges = sub.track.subscription.peek()?.ranges; - while (ranges?.[0]?.end?.absolute !== end) ranges = (await sub.track.subscription.changed())?.ranges; - }; - try { - send([0, 2, 1]); - expect(await recv()).toBe(1n); - // Widen both boundaries, then narrow to one datagram. - await replayUpdate({ - priority: 0, - startGroup: 0n, - startFrame: 0, - endGroup: 4n, - endFrame: 0, - ...restate(version), - }).encode(sub.client.writer, version); - await applied(5n); - send([2, 3, 5, 4]); - expect([await recv(), await recv(), await recv()]).toEqual([2n, 3n, 4n]); - await replayUpdate({ - priority: 0, - startGroup: 6n, - startFrame: 1, - endGroup: 7n, - endFrame: 0, - ...restate(version, 1n), - }).encode(sub.client.writer, version); - await applied(8n); - send([6, 8, 7]); - expect(await recv()).toBe(7n); - } finally { - await reader.cancel(); - await sub.close(); - await sub.serving; - installed.close(); - } -}); - -// A sent datagram is accounted for, so the end sweep does not drop it, as the Rust publisher does. -test("lite draft-06: the end sweep does not drop a sent datagram", async () => { - const sub = await servedSubscription({ version: Version.DRAFT_06, startGroup: 0n }); - try { - sub.serve(0n); - expect(await sub.servedSequence()).toBe(0n); - sub.track.insertDatagram(1n, Timestamp.fromMillis(1), new Uint8Array([1])); - await flush(); - sub.track.finish(); - sub.track.close(); - expect(await drops(sub.client.reader, Version.DRAFT_06)).toEqual([]); - } finally { - await sub.close(); - } -}); - // A drop describes what was missing then; a late copy of the same group is still served. test("lite draft-06: a reported cut still allows a late group", async () => { const sub = await servedSubscription({ version: Version.DRAFT_06, startGroup: 0n }); @@ -3389,3 +3290,57 @@ test("lite-07 SYNCED survives an initial removal and continuous earlier-prefix c pair.server.close(); } }); + +test("lite-06: a datagram request failing behind the shared writer resets rather than ends", async () => { + const version = Version.DRAFT_06; + const pair = createMockTransportPair(ALPN_06); + // The legacy API's one session queue, stalled on its first write. + let writes = 0; + const stall = Promise.withResolvers(); + Object.defineProperty(pair.server.datagrams, "writable", { + value: new WritableStream({ + write() { + writes++; + return writes === 1 ? stall.promise : undefined; + }, + }), + }); + const origin = new OriginProducer(); + const publisher = new Publisher(pair.server, version, randomHop(), origin.consume()); + const broadcast = publish(origin, Path.from("test")); + const first = broadcast.createDatagrams("a"); + const second = broadcast.createDatagrams("b"); + + const subscribe = async (id: bigint, track: string) => { + const client = await Stream.open(pair.client, { version }); + const server = await Stream.accept(pair.server, version); + if (!server) throw new Error("publisher never accepted the subscribe stream"); + void publisher.runSubscribe(new Subscribe({ id, broadcast: Path.from("test"), track, priority: 0 }), server); + expect("start" in (await decodeSubscribeResponse(client.reader, version))).toBe(true); + return client; + }; + const settle = async () => { + for (let i = 0; i < 20; i++) await new Promise((resolve) => setImmediate(resolve)); + }; + const a = await subscribe(0n, "a"); + const b = await subscribe(1n, "b"); + try { + first.append(undefined, Uint8Array.of(0xaa)); + while (writes === 0) await settle(); + // The second request now waits on the writer the first one holds. + second.append(undefined, Uint8Array.of(0xbb)); + await settle(); + // A SUBSCRIBE_UPDATE holding only its priority fails decoding, while the stream stays open. + await b.writer.u53(1); + await b.writer.u8(0); + await expect(decodeSubscribeResponse(b.reader, version)).rejects.toThrow(); + } finally { + stall.resolve(); + a.close(); + b.close(); + publisher.close(); + origin.close(); + pair.client.close(); + pair.server.close(); + } +}); diff --git a/js/net/src/lite/publisher.ts b/js/net/src/lite/publisher.ts index 322e1dffdc2..8cefca22724 100644 --- a/js/net/src/lite/publisher.ts +++ b/js/net/src/lite/publisher.ts @@ -6,6 +6,7 @@ import type { AuthSession } from "../auth_session.ts"; import { enforceGrant } from "../auth_session.ts"; import type * as broadcast from "../broadcast.ts"; import { Withdrawal } from "../connection/withdrawal.ts"; +import type * as datagram from "../datagram.ts"; import * as DatagramStream from "../datagram_stream.ts"; import { error, @@ -17,7 +18,6 @@ import { toStreamCode, unauthorized, } from "../error.ts"; -import { stored } from "../flate/group.ts"; import type * as group from "../group.ts"; import { Cost, type Hop, type Route, routesEqual } from "../hop.ts"; import { finalSequence, hiddenBelow, hooks, insertRun, presented, type Run, runGaps } from "../internal.ts"; @@ -33,10 +33,11 @@ import { untilAborted } from "../util/abort.ts"; import { nextChange } from "../util/change.ts"; import { GivenUp } from "../util/given_up.ts"; import { max, min } from "../util/sequence.ts"; +import { Budget } from "../util/yield.ts"; import type { Resolved } from "../window.ts"; import { type Advertised, type Advertisements, sameInstance, wireOf } from "../wire.ts"; import { AnnounceInit, AnnounceOk, type AnnounceRequest, encodeAnnounceBroadcast } from "./announce.ts"; -import { Datagram as DatagramMessage } from "./datagram.ts"; +import { Datagram as DatagramMessage, type Datagrams, DatagramsInfo, DatagramsUpdate } from "./datagram.ts"; import { closeViolation } from "./error.ts"; import type { Fetch } from "./fetch.ts"; import { Group as GroupMessage } from "./group.ts"; @@ -65,6 +66,7 @@ import { hasAnnounceRestart, hasCompression, hasDatagrams, + hasDatagramTracks, hasEndError, hasInstance, hasLargest, @@ -84,6 +86,7 @@ const PROBE_INTERVAL = 100; // ms const PROBE_MAX_AGE = 10_000; // ms const PROBE_MAX_DELTA = 0.25; const PROBE_RTT_DELTA = 0.25; +const datagramBudget = new Budget(); /** Map a signed delta to an unsigned zigzag varint value (mirrors Rust `varint::zigzag`). */ function zigzag(delta: bigint): bigint { @@ -512,11 +515,8 @@ export class Publisher { #quic: WebTransport; - // The one writer for the outbound datagram stream (getWriter locks it), acquired once at - // construction when this version + transport carry datagrams, released in close(). Its - // presence is the gate: undefined means datagrams aren't served on this connection. All - // subscriptions share it, since a second getWriter on the same stream would throw. - #datagramWriter?: WritableStreamDefaultWriter; + // Select the transport's current or legacy datagram send API once at setup. + #datagramWriter?: ReturnType; // Originated advertisements this session forwards. #advertised: Getter; @@ -1001,9 +1001,7 @@ export class Publisher { // The ranges cap what is served; the floor only pins where an older wire starts. track.withRange({ start: { included: floor } }); - // The best-effort datagram loop, started once serving begins. It parks when the - // track finishes (recvDatagram returns undefined), so #runTrack alone ends the - // subscription; awaited during teardown so it doesn't outlive the subscription. + // An old subscription carries both kinds until its stream track ends. let datagrams = Promise.resolve(); const bounds: FrameBounds = { startGroup: msg.startGroup, @@ -1011,13 +1009,11 @@ export class Publisher { endGroup: msg.endGroup, endFrame: msg.endFrame, }; - // Datagram reads continue while a reliable response is blocked, so they use - // the latest decoded bounds rather than the reliable loop's next-pop snapshot. - const datagramBounds = { ...bounds }; + let legacy: datagram.Consumer | undefined; let controls: SubscriptionControls | undefined; // Lite-05 and later name every sequence below the end that this subscription never // got with SUBSCRIBE_DROP, so the subscriber settles without waiting out its grace - // for a group that will not come. The sequences served (as a group or a datagram) or + // for a group that will not come. The group sequences served or // dropped. A gap that never arrived stays until the end sweep names it: this side // cannot tell when the subscriber stopped waiting for it. Undefined on other versions. const served: Run[] | undefined = hasTrackStream(this.version) ? [] : undefined; @@ -1027,7 +1023,21 @@ export class Publisher { // (accept never called, track closed) as an error here, which resets the stream, and // its priority breaks ties between this session's subscriptions at one subscriber // priority. - const info = await untilAborted(track.info(), left.signal); + let info: track.Info; + try { + info = await untilAborted(track.info(), left.signal); + } catch (err) { + if ( + hasDatagrams(this.version) && + !hasDatagramTracks(this.version) && + toStreamCode(error(err)) === StreamCode.NotFound + ) { + track.close(); + await this.#serveLegacyDatagrams(front, msg, stream, left.signal); + return; + } + throw err; + } const publisher = info.priority; // A SUBSCRIBE can arrive before its track resolves, such as one a relay serves from a // copy still fetching upstream, or on lite-05 one sent alongside its TRACK, when no @@ -1067,8 +1077,14 @@ export class Publisher { // Serve datagrams concurrently with groups whenever the transport carries them // (the writer exists iff so). No group fallback: otherwise they simply aren't sent. - if (this.#datagramWriter) { - datagrams = this.#runDatagrams(track, { sub: msg.id, stamp, served, bounds: datagramBounds }); + if (this.#datagramWriter && !hasDatagramTracks(this.version)) { + legacy = wireOf(front).liveDatagrams(msg.track); + if (legacy) { + legacy.setPriority(msg.priority); + datagrams = this.#runDatagrams(legacy, { sub: msg.id, stamp }) + .then(() => {}) + .catch((err) => console.debug(`datagram send stopped: ${reason(err)}`)); + } } // The SUBSCRIBE's ranges, and the Range ID of each range the model holds, in its order. @@ -1082,6 +1098,7 @@ export class Publisher { version: this.version, live: new LiveRanges(this.version, asked), apply: (update, live) => { + legacy?.setPriority(update.priority); if (hasRanges(this.version)) { const resolved = hooks.servingResolved(track); const entries = live.map(({ id, entry }) => { @@ -1091,12 +1108,6 @@ export class Publisher { Object.assign(update, spanOf(this.version, entries)); positions = new Map(live.map(({ id }, index) => [id, index])); } - Object.assign(datagramBounds, { - startGroup: update.startGroup, - startFrame: update.startFrame, - endGroup: update.endGroup, - endFrame: update.endFrame, - }); // Every update restates the wanted group. Past our largest it is a join, which // repeats the pending ask; at or below it the group already exists. const largest = hooks.peekLargest(track).largest?.group; @@ -1131,8 +1142,9 @@ export class Publisher { // here, which ends the decoder. if (waitsForSubscriberFin(this.version)) stream.writer.close(); else stream.close(); - // Ends the datagram loop. + // Ends both kinds carried by this old subscription. track.close(); + legacy?.close(); // A subscriber that left is owed nothing more, and its stream may already be reset. await Promise.all([datagrams, controls.decoding, finished && acknowledged(stream.writer)]); } catch (err: unknown) { @@ -1141,6 +1153,7 @@ export class Publisher { if (e === cancel) console.debug(`publish cancelled: broadcast=${msg.broadcast} track=${track.name}`); else console.warn(`publish error: broadcast=${msg.broadcast} track=${track.name} error=${reason(e)}`); track.close(e); + legacy?.close(e); stream.abort(e); await Promise.all([datagrams, controls?.decoding]); } @@ -1793,7 +1806,23 @@ export class Publisher { (await wireOf(this.#publish).demand(msg.broadcast, msg.instance, hold.signal))); if (!front) throw new NotFound(`broadcast ${msg.broadcast}`); - const info = await this.#resolveTrackInfo(front, msg.track, hold.signal); + let info: TrackInfoMessage; + try { + info = await this.#resolveTrackInfo(front, msg.track, hold.signal); + } catch (err) { + if ( + !hasDatagrams(this.version) || + hasDatagramTracks(this.version) || + toStreamCode(error(err)) !== StreamCode.NotFound + ) + throw err; + const datagrams = front.datagrams(msg.track); + try { + info = trackInfoMessage(await untilAborted(datagrams.info(), hold.signal), this.version); + } finally { + datagrams.close(); + } + } if (watch.revoked()) throw unauthorized(msg.broadcast); await info.encode(stream.writer, this.version); console.debug(`track info: broadcast=${msg.broadcast} track=${msg.track}`); @@ -1931,56 +1960,146 @@ export class Publisher { return untilAborted(held.info, hold); } - /** - * Forwards a track's datagrams best-effort over QUIC datagrams (lite-05 §6.4), parallel to - * its groups. Each datagram is dropped (there is no group fallback) if the encoded body - * doesn't fit the wire and transport datagram limits or the send fails. Returns once the track - * finishes; a failure never tears down the subscription. - * - * @internal - */ - async #runDatagrams( - track: track.Subscriber, - options: { - sub: bigint; - stamp: Stamp | undefined; - served: Run[] | undefined; - bounds: FrameBounds; - }, - ) { - const { sub, stamp, served, bounds } = options; - const writer = this.#datagramWriter; - if (!writer) return; // Only reached with a writer (see the #datagramWriter gate). - const maxSize = Math.min(DatagramMessage.MAX_SIZE, DatagramStream.maxDatagramSize(this.#quic)); + /** Serve a lite-07 datagram-track request until its publisher or subscriber ends it. */ + async runDatagrams(msg: Datagrams, stream: Stream): Promise { + const cancel = new AbortController(); + let track: datagram.Consumer | undefined; + const watch = this.#gate.watch(msg.broadcast, () => cancel.abort(unauthorized(msg.broadcast))); + let updates: Promise | undefined; + try { + if (watch.revoked()) throw unauthorized(msg.broadcast); + if (!this.#datagramWriter) throw new Error("transport does not support datagrams"); + // Read updates from dispatch so a requester leaving releases even an unanswered lookup. + updates = (async () => { + let next = msg.priority; + for (;;) { + const yielded = datagramBudget.poll(); + if (yielded !== undefined) await yielded; + const update = await DatagramsUpdate.decodeMaybe(stream.reader, this.version); + if (!update) return; + next = update.priority; + msg.priority = next; + track?.setPriority(next); + } + })(); + void updates.then( + () => cancel.abort(new StreamError(StreamCode.Cancel)), + (err) => cancel.abort(error(err)), + ); + const origin = this.#publish && wireOf(this.#publish); + const front = + origin && + (origin.local(msg.broadcast, msg.instance) ?? + (await origin.demand(msg.broadcast, msg.instance, cancel.signal))); + if (!front) throw new NotFound(`broadcast ${msg.broadcast}`); + track = front.datagrams(msg.track); + track.setPriority(msg.priority); + const info = await untilAborted(track.info(), cancel.signal); + await untilAborted(new DatagramsInfo(info).encode(stream.writer, this.version), cancel.signal); + await untilAborted( + this.#runDatagrams(track, { sub: msg.id, stamp: stamping(this.version, info) }), + cancel.signal, + ); + stream.writer.close(); + await updates; + } catch (err) { + closeViolation(this.#quic, err); + stream.abort(error(err)); + } finally { + track?.close(); + watch.dispose(); + stream.reader.stop(StreamCode.Cancel); + await updates?.catch(() => {}); + } + } + + // Lite-05/06: a name with no stream track serves its datagram track alone. + async #serveLegacyDatagrams( + front: broadcast.Consumer, + msg: Subscribe, + stream: Stream, + signal: AbortSignal, + ): Promise { + const track = front.datagrams(msg.track); + track.setPriority(msg.priority); + const updates = (async () => { + for (;;) { + const yielded = datagramBudget.poll(); + if (yielded !== undefined) await yielded; + const update = await SubscribeUpdate.decodeMaybe(stream.reader, this.version); + if (!update) return; + track.setPriority(update.priority); + } + })(); + void updates.then( + () => track.close(), + (err) => track.close(error(err)), + ); + try { + const info = await untilAborted(track.info(), signal); + await encodeSubscribeResponse(stream.writer, { start: new SubscribeStart([0n]) }, this.version); + const end = await untilAborted( + this.#runDatagrams(track, { sub: msg.id, stamp: stamping(this.version, info) }), + signal, + ); + await encodeSubscribeResponse(stream.writer, { finish: new SubscribeFinish(end) }, this.version); + stream.writer.close(); + await updates; + } finally { + track.close(); + stream.reader.stop(StreamCode.Cancel); + await updates.catch(() => {}); + } + } + // Shared by dedicated lite-07 requests and the old SUBSCRIBE carry-both mapping. + async #runDatagrams(track: datagram.Consumer, options: { sub: bigint; stamp?: Stamp }): Promise { + if (!this.#datagramWriter) throw new Error("transport does not support datagrams"); + const info = await track.info(); + const rank = () => sendOrder({ priority: track.priority, publisher: info.priority }); + const sender = this.#datagramWriter.open(rank()); + const dispose = hooks.datagramPriority(track).subscribe(() => sender.setPriority(rank())); + const left = hooks.datagramClosed(track); + const { writer } = sender; + const maxSize = Math.min(DatagramMessage.MAX_SIZE, DatagramStream.maxDatagramSize(this.#quic)); + let end = 0n; try { for (;;) { - const datagram = await track.recvDatagram(); - if (!datagram) return; // Track finished; #runTrack tears the subscription down. - // A datagram is frame 0 of its group, subject to the reliable frame range. - if (frameRange(bounds, datagram.sequence).start > 0) continue; - - // Convert the timestamp to the track's advertised timescale, matching #serveGroup. - const ts = stamp === undefined ? undefined : wireTime(datagram.timestamp, stamp); - const slice = hasCompression(this.version) ? stored(datagram) : undefined; - const message = new DatagramMessage(sub, datagram.sequence, ts, slice?.payload ?? datagram.payload); - message.inflated = slice?.inflated; - const body = message.encode(this.version); - - // No group fallback: drop anything that doesn't fit a single datagram. + const datagram = await track.recv(); + if (!datagram) return end; + end = max(end, datagram.sequence + 1n); + const stamp = options.stamp; + // An old subscription uses its stream track's timescale for both kinds. + const timestamp = + stamp === undefined + ? undefined + : datagram.timestamp === undefined + ? floorAt(Timestamp.now(), stamp.timescale) + : floorAt(datagram.timestamp, stamp.timescale); + const body = new DatagramMessage(options.sub, datagram.sequence, timestamp, datagram.payload).encode( + this.version, + ); if (body.byteLength > maxSize) { - console.debug(`dropping oversize datagram: sub=${sub} size=${body.byteLength} max=${maxSize}`); + console.debug( + `dropping oversize datagram: sub=${options.sub} size=${body.byteLength} max=${maxSize}`, + ); continue; } - - // A sent datagram counts as delivered even if it is lost, so the end sweep skips it. - if (served) insertRun(served, datagram.sequence, datagram.sequence + 1n); - await writer.ready; + // The legacy writer is shared, so another request's write can hold back this one: + // stop waiting once this request ends, and send nothing after it does. + // Raced directly, so a ready writer drops its close listener rather than keep one per send. + await race([writer.ready, left]); + // As recv() does: a clean close ends the request, an error close fails it. + const reason = left.peek(); + if (reason !== undefined) { + if (reason) throw reason; + return end; + } await writer.write(body); } - } catch (err: unknown) { - // Best-effort: a datagram send failure stops sending but never fails the subscription. - console.debug(`datagram send stopped: sub=${sub} error=${reason(err)}`); + } finally { + dispose(); + await sender.close(); } } @@ -2307,7 +2426,7 @@ export class Publisher { // only drops the borrow. The peer sees the unannounce when the streams die. // Release the datagram writer's lock so the stream can be torn down. - this.#datagramWriter?.releaseLock(); + this.#datagramWriter?.close(); this.#datagramWriter = undefined; } } diff --git a/js/net/src/lite/stream.ts b/js/net/src/lite/stream.ts index 7c95e5ca558..486487ede3b 100644 --- a/js/net/src/lite/stream.ts +++ b/js/net/src/lite/stream.ts @@ -8,6 +8,7 @@ export const StreamId = { Track: 6, Auth: 7, Tracks: 8, + Datagrams: 9, ClientCompat: 0x20, ServerCompat: 0x21, } as const; diff --git a/js/net/src/lite/subscriber.test.ts b/js/net/src/lite/subscriber.test.ts index cde4620246b..a9004ee48f2 100644 --- a/js/net/src/lite/subscriber.test.ts +++ b/js/net/src/lite/subscriber.test.ts @@ -29,7 +29,6 @@ import { SEGMENT_BYTES } from "../util/limits.ts"; import { Budget } from "../util/yield.ts"; import * as Varint from "../varint.ts"; import { type AnnounceBroadcast, AnnounceInit, AnnounceOk, encodeAnnounceBroadcast } from "./announce.ts"; -import { Datagram as DatagramMessage } from "./datagram.ts"; import { Group } from "./group.ts"; import { Probe } from "./probe.ts"; import { ProbeLevel, Setup } from "./setup.ts"; @@ -1126,41 +1125,6 @@ async function answerTrackInfo( stream.inbound.close(); } -test.each([Version.DRAFT_06, Version.DRAFT_07])( - "lite %s: oversized incoming datagrams are dropped", - async (version) => { - const installed = new Manual().install(); - const { quic, opened } = fakeSession(); - const incoming = new TransformStream(); - Object.defineProperty(quic, "datagrams", { value: { readable: incoming.readable, maxDatagramSize: 1400 } }); - const subscriber = new Subscriber(quic, version, HopSchema.parse(1n)); - const track = subscriber.consume(Path.from("room")).track("video").subscribe(); - const writer = incoming.writable.getWriter(); - const receiving = subscriber.runDatagrams(); - try { - await answerTrackInfo(await opened(0), version, 1000); - const request = await opened(1); - await request.reading; - await respond(request, version, [{ start: new SubscribeStart([0n]) }]); - const received = track.recvDatagram(); - if (version === Version.DRAFT_07) { - await writer.write(new Uint8Array([1])); - await writer.write(new Uint8Array([1, ...new Uint8Array(1199).fill(0xff)])); - } - const size = version === Version.DRAFT_07 ? 1196 : 1197; - await writer.write(new DatagramMessage(0n, 0n, 0, new Uint8Array(size + 1)).encode(version)); - await writer.write(new DatagramMessage(0n, 1n, 1, new Uint8Array(size)).encode(version)); - expect((await received)?.sequence).toBe(1n); - } finally { - await writer.close(); - await receiving; - track.close(); - subscriber.close(); - installed.close(); - } - }, -); - function expectCut(err: unknown, cause: Error | undefined) { if (cause) { expect(err).toBe(cause); @@ -3138,3 +3102,61 @@ test("a FETCH stops a stalled frame only after its last coalesced reader leaves" subscriber.close(); } }); + +test("a cancelled legacy datagram lookup cannot refuse a stream answer arriving before auth settles", async () => { + const { quic, streams, opened } = fakeSession(); + Object.defineProperty(quic, "datagrams", { + value: { maxDatagramSize: 1200, readable: new ReadableStream() }, + }); + const waiting = Promise.withResolvers(); + const release = Promise.withResolvers(); + let calls = 0; + const auth = { + async ready() { + if (++calls === 2) { + waiting.resolve(); + await release.promise; + } + return true; + }, + } as unknown as AuthSession; + const version = Version.DRAFT_06; + const subscriber = new Subscriber( + quic, + version, + HopSchema.parse(1n), + undefined, + undefined, + undefined, + undefined, + undefined, + auth, + ); + const remote = subscriber.consume(Path.from("room")); + const grouped = remote.track("video").subscribe(); + let datagrams: ReturnType | undefined; + try { + await opened(1); + datagrams = remote.datagrams("video"); + await waiting.promise; + const info = datagrams.info(); + datagrams.close(); + await expect(info).rejects.toBeInstanceOf(StreamError); + await answerTrackInfo(streams[0], version); + await grouped.info(); + const writer = new Writer( + new WritableStream({ write: (chunk) => streams[1].inbound.enqueue(new Uint8Array(chunk)) }), + version, + ); + await encodeSubscribeResponse(writer, { start: new SubscribeStart([0n]) }, version); + await encodeSubscribeResponse(writer, { finish: new SubscribeFinish(0n) }, version); + streams[1].inbound.close(); + expect(await grouped.recvGroup()).toBeUndefined(); + } finally { + release.resolve(); + datagrams?.close(); + grouped.close(); + remote.close(); + subscriber.close(); + } +}); diff --git a/js/net/src/lite/subscriber.ts b/js/net/src/lite/subscriber.ts index 87515b03144..7c6fb03d36c 100644 --- a/js/net/src/lite/subscriber.ts +++ b/js/net/src/lite/subscriber.ts @@ -7,6 +7,7 @@ import * as broadcast from "../broadcast.ts"; import type { Drain } from "../connection/goaway.ts"; import type { Probe as ProbeStats } from "../connection/stats.ts"; import { BroadcastCache } from "../consume.ts"; +import type * as datagram from "../datagram.ts"; import * as DatagramStream from "../datagram_stream.ts"; import { error, @@ -46,7 +47,7 @@ import { AnnounceRequest, decodeAnnounceBroadcastMaybe, } from "./announce.ts"; -import { Datagram as DatagramMessage } from "./datagram.ts"; +import { Datagram as DatagramMessage, Datagrams, DatagramsInfo, DatagramsUpdate } from "./datagram.ts"; import { closeViolation } from "./error.ts"; import { Fetch as FetchMessage } from "./fetch.ts"; import { type Group as GroupMessage, readFrames } from "./group.ts"; @@ -72,6 +73,7 @@ import { hasAnnounceId, hasAnnounceOk, hasDatagrams, + hasDatagramTracks, hasFetchMaxDelay, hasMediaTime, hasMinDelay, @@ -93,6 +95,7 @@ import { // A peer's buffered TRACKS events decode without reaching a task boundary, and a flood of adds // and removals of one name never reaches the announces cap: yield so it cannot starve the host. const listingBudget = new Budget(); +const datagramBudget = new Budget(); // Request streams refuse exhausted peer credit, with one deadline covering create and answer. export const SUBSCRIBE_SETUP_TIMEOUT_MS = REQUEST_TIMEOUT_MS; @@ -105,6 +108,8 @@ const DRAIN_COST: Cost = 2n ** 62n - 1n; // What a subscription has set up so far, so a revocation (or a timeout) can reach it at // any stage. interface SubscribeSetup { + consumer?: broadcast.Consumer; + legacyKey?: string; stream?: Stream; track?: Stream; producer?: track.Producer; @@ -128,7 +133,16 @@ interface Consumed { readonly answering?: AbortSignal; } +interface DatagramEntry { + producer: datagram.Producer; + timescale?: Time.Timescale; +} + interface SubscribeEntry { + datagrams?: DatagramEntry; + /** Release the stream demand held by readers of the legacy datagram track. */ + datagramDemand?: Dispose; + legacyKey?: string; // The write side: incoming GROUP streams are routed here. The application reads // the matching track.Subscriber it got from broadcast.Consumer.subscribe. track: track.Producer; @@ -230,6 +244,9 @@ export class Subscriber { // Registered only after immutable TRACK_INFO or the older drafts' defaults resolve, // so every incoming group and datagram already knows its timestamp framing. #subscribes = new Map(); + #datagrams = new Map(); + #legacyDatagrams = new Map(); + #legacyRequests = new Map(); // A SUBSCRIBE sent alongside its TRACK, until its TRACK_INFO lands: settles with the // entry, or `undefined` if it never does. #pending = new Map>(); @@ -804,13 +821,160 @@ export class Subscriber { for (;;) { const request = await wireOf(consumer).requested(); if (!request) break; - void this.#runSubscribe(path, consumed, request); + void this.#runSubscribe(path, consumed, request, consumer); } })(); + void (async () => { + for (;;) { + const request = await wireOf(consumer).requestedDatagrams(); + if (!request) return; + void this.#runDatagramRequest(path, consumed, consumer, request); + } + })(); return this.#consumes.insert(path, instance, consumer); } + // Each request owns its setup and subscription until it ends or loses its last reader. + async #runDatagramRequest( + path: Path.Valid, + consumed: Consumed, + consumer: broadcast.Consumer, + request: datagram.Request, + ): Promise { + const cancel = new AbortController(); + const signal = AbortSignal.any([ + cancel.signal, + this.#closed.signal, + ...(consumed.valid ? [consumed.valid] : []), + ]); + let producer: datagram.Producer | undefined; + let stream: Stream | undefined; + const id = this.#subscribeNext++; + const legacyKey = + hasDatagrams(this.version) && !hasDatagramTracks(this.version) + ? JSON.stringify([consumed.id, request.name]) + : undefined; + // A stream answer can arrive while this request waits for authorization. + if (legacyKey !== undefined) this.#legacyRequests.set(legacyKey, request); + const unused = async () => { + while (request.priority.peek() !== undefined && request.closed.peek() === undefined) + await Signal.race(request.priority, request.closed); + // An accepted legacy producer belongs to the stream subscription, including gaps + // between datagram readers. Its priority watcher releases only their extra demand. + if (legacyKey !== undefined && this.#legacyDatagrams.has(legacyKey)) return; + if (request.priority.peek() === undefined && (!producer || hooks.closeUnusedDatagrams(producer))) + cancel.abort(new StreamError(StreamCode.Cancel)); + }; + void unused(); + const grant = this.#grant?.subscribe(() => { + if (this.#denied(path)) cancel.abort(unauthorized(path)); + }); + try { + if ( + !hasDatagrams(this.version) || + DatagramStream.maxDatagramSize(this.#quic) === 0 || + !DatagramStream.canReceiveDatagrams(this.#quic) + ) + throw new Error("transport does not support datagram tracks"); + const ready = await this.#auth?.ready(path, "subscribe", { + signal, + until: () => Promise.resolve(request.closed), + }); + if (ready === false) throw new StreamError(StreamCode.Cancel); + if (this.#denied(path)) throw unauthorized(path); + signal.throwIfAborted(); + if (!hasDatagramTracks(this.version)) { + await this.#legacyDatagramRequest(consumed, consumer, request, signal); + return; + } + const deadline = new Deadline(); + let priority = request.priority.peek() ?? 0; + const answered = await deadline.wait( + this.#exchange( + { version: this.version }, + async (opened) => { + stream = opened; + await opened.writer.u53(StreamId.Datagrams); + await new Datagrams({ + id, + broadcast: path, + instance: consumed.instance, + track: request.name, + priority, + }).encode(opened.writer, this.version); + const info = await DatagramsInfo.decode(opened.reader, this.version); + return info; + }, + signal, + deadline, + ), + ); + signal.throwIfAborted(); + if (!stream) throw new Error("datagram stream was not opened"); + producer = request.accept(answered); + this.#datagrams.set(id, { producer, timescale: answered.timescale }); + const opened = stream; + const receiving = producer; + // The only thing following DATAGRAMS_INFO is the publisher's FIN. + const ended = opened.reader.done().then((done) => { + if (!done) throw new ProtocolViolation("data after DATAGRAMS_INFO"); + receiving.finish(); + }); + void ended.catch(() => {}); + while (producer.closed.peek() === undefined) { + const next = producer.priority.peek(); + if (next === undefined && hooks.closeUnusedDatagrams(producer)) break; + if (next !== undefined && next !== priority) { + await untilAborted(new DatagramsUpdate(next).encode(opened.writer, this.version), signal); + priority = next; + // The priority may have changed again during the write, before the wait below. + continue; + } + await untilAborted(race([ended, Signal.race(producer.priority, producer.closed)]), signal); + } + const closed = producer.closed.peek(); + if (closed instanceof Error) throw closed; + stream.writer.close(); + } catch (err) { + closeViolation(this.#quic, err); + const cause = error(err); + cancel.abort(cause); + request.reject(cause); + stream?.abort(cause); + } finally { + if (legacyKey !== undefined && this.#legacyRequests.get(legacyKey) === request) + this.#legacyRequests.delete(legacyKey); + grant?.(); + this.#datagrams.delete(id); + cancel.abort(new StreamError(StreamCode.Cancel)); + stream?.reader.stop(StreamCode.Cancel); + } + } + + // Lite-05/06: the shared producer holds stream demand while any datagram reader exists. + async #legacyDatagramRequest( + consumed: Consumed, + consumer: broadcast.Consumer, + request: datagram.Request, + signal: AbortSignal, + ): Promise { + const key = JSON.stringify([consumed.id, request.name]); + const subscriber = wireOf(consumer).subscribe(request.name, { priority: request.priority.peek() ?? 0 }); + try { + await untilAborted(subscriber.info(), signal); + const shared = this.#legacyDatagrams.get(key); + if (!shared) throw new Error("stream subscription did not bind its datagram track"); + const producer = shared.producer; + // Once registered, the shared producer owns demand regardless of which kind joined first. + subscriber.close(); + const closed = await untilAborted(Promise.resolve(producer.closed), signal); + if (closed instanceof Error) throw closed; + } finally { + subscriber.close(); + } + } + // Answer listings of a consumed broadcast from the peer's TRACKS while anyone lists it, until // the broadcast closes. Every listing of the broadcast shares one TRACKS stream. async #runListing(broadcast: Path.Valid, consumed: Consumed, lister: Lister) { @@ -953,7 +1117,12 @@ export class Subscriber { } } - async #runSubscribe(broadcast: Path.Valid, consumed: Consumed, request: track.Request) { + async #runSubscribe( + broadcast: Path.Valid, + consumed: Consumed, + request: track.Request, + consumer: broadcast.Consumer, + ) { const id = this.#subscribeNext++; if (wireRanges(request.subscription.ranges ?? [{}]).length === 0) { request.reject(new Error(EMPTY_RANGE)); @@ -978,7 +1147,11 @@ export class Subscriber { } // Armed before the first check and held to the end, so a shrink while the // subscription sets up is never missed: it resets whatever reached the wire. - const state: SubscribeSetup = { cancel: new AbortController() }; + const state: SubscribeSetup = { + cancel: new AbortController(), + consumer, + legacyKey: JSON.stringify([consumed.id, request.name]), + }; const disposeGrant = this.#grant?.subscribe(() => { if (state.revoked || !this.#denied(broadcast)) return; state.revoked = refused; @@ -1198,6 +1371,17 @@ export class Subscriber { } finally { answering?.removeEventListener("abort", replaced); entry.held?.close(); + entry.datagramDemand?.(); + if (entry.datagrams) { + const closed = hooks.trackReleased(producer).peek(); + if (closed instanceof Error) entry.datagrams.producer.abort(closed); + else if (entry.end !== undefined) entry.datagrams.producer.finish(); + else entry.datagrams.producer.abort(new StreamError(StreamCode.Cancel)); + this.#datagrams.delete(id); + if (entry.legacyKey && this.#legacyDatagrams.get(entry.legacyKey) === entry.datagrams) + this.#legacyDatagrams.delete(entry.legacyKey); + } + this.#subscribes.delete(id); } } @@ -1339,6 +1523,38 @@ export class Subscriber { live: new LiveRanges(this.version, msg.ranges ?? []), starts: new Map(), }; + if (hasDatagrams(this.version) && !hasDatagramTracks(this.version) && state.consumer && state.legacyKey) { + const info = { priority: producer.priority, timescale: framing.scale }; + const pending = this.#legacyRequests.get(state.legacyKey); + const request = pending?.closed.peek() === undefined ? pending : undefined; + this.#legacyRequests.delete(state.legacyKey); + const datagrams = request + ? request.accept(info) + : wireOf(state.consumer).createDatagrams(producer.name, info); + entry.datagrams = { producer: datagrams, timescale: framing.scale }; + entry.legacyKey = state.legacyKey; + this.#datagrams.set(id, entry.datagrams); + this.#legacyDatagrams.set(state.legacyKey, entry.datagrams); + let held: track.Subscriber | undefined; + const sync = () => { + const priority = datagrams.priority.peek(); + if (priority === undefined) { + held?.close(); + held = undefined; + } else if (held) { + held.update({ priority }); + } else { + held = producer.subscribe({ priority }); + } + }; + const dispose = datagrams.priority.subscribe(sync); + sync(); + entry.datagramDemand = () => { + dispose(); + held?.close(); + }; + } + this.#subscribes.set(id, entry); state.producer = producer; return entry; @@ -2169,6 +2385,8 @@ export class Subscriber { try { try { for (;;) { + const yielded = datagramBudget.poll(); + if (yielded !== undefined) await yielded; const { value, done } = await reader.read(); if (done) break; if (!value) continue; @@ -2199,22 +2417,17 @@ export class Subscriber { // Read with the entry it was decoded with: one registered while the decode awaited would // read a datagram parsed with the defaults. An unknown subscription's datagram, or one // whose TRACK_INFO had not landed, which a datagram cannot wait for, is dropped below. - let entry: SubscribeEntry | undefined; + let entry: DatagramEntry | undefined; const dg = await DatagramMessage.decode(payload, this.version, (id) => { - entry = this.#subscribes.get(id); - return { timed: !!entry?.timescale, compressed: entry?.compress ?? false }; + entry = this.#datagrams.get(id); + return { timed: entry?.timescale !== undefined }; }); - if (!dg) return; - // Gone since, too: the subscription ended while the decode awaited. - if (!entry || this.#subscribes.get(dg.subscribe) !== entry) return; - - const scale = entry.timescale; - + if (!dg || !entry || this.#datagrams.get(dg.subscribe) !== entry) return; const timestamp = - dg.timestamp === undefined ? undefined : new Time.Timestamp(dg.timestamp, Time.Timescale(scale)); - // A datagram's sequence is never owed a stream, so it never holds the tail open. - entry.tail.account(dg.sequence, dg.sequence + 1n); - hooks.writeDatagram(entry.track, { sequence: dg.sequence, timestamp, payload: dg.payload }, dg.inflated); + dg.timestamp === undefined || entry.timescale === undefined + ? undefined + : new Time.Timestamp(dg.timestamp, entry.timescale); + entry.producer.insert({ sequence: dg.sequence, timestamp, payload: dg.payload }); } /** @@ -2355,6 +2568,9 @@ export class Subscriber { } this.#subscribes.clear(); + for (const { producer } of this.#datagrams.values()) producer.abort(err); + this.#datagrams.clear(); + this.#legacyDatagrams.clear(); // This also releases callers still awaiting acceptance. for (const { group } of this.#fetches.values()) { diff --git a/js/net/src/lite/version.ts b/js/net/src/lite/version.ts index 88e849ba25f..91c3310693d 100644 --- a/js/net/src/lite/version.ts +++ b/js/net/src/lite/version.ts @@ -70,9 +70,7 @@ export function hasTrackStream(version: Version): boolean { } } -/// Whether the session may deliver groups over unreliable QUIC datagrams (lite-05 §6.4). -/// A datagram carries one single-frame group's `subscribe | sequence | timestamp | payload` and is -/// routed over the existing subscription. Added in lite-05; older versions never send/accept them. +/** Whether the session carries unreliable datagrams, from lite-05 onward. */ export function hasDatagrams(version: Version): boolean { // Explicitly list older versions so future versions default to having datagrams. switch (version) { @@ -86,6 +84,11 @@ export function hasDatagrams(version: Version): boolean { } } +/** Whether datagram tracks have their own request and sequence space. */ +export function hasDatagramTracks(version: Version): boolean { + return hasPadding(version); +} + /** Whether the session supports the GOAWAY control stream (0x5) for graceful shutdown and migration. Added in lite-04. */ export function hasGoaway(version: Version): boolean { // Explicitly list older versions so future versions default to having GOAWAY. diff --git a/js/net/src/mock.ts b/js/net/src/mock.ts index 40c27246419..726a9341e8f 100644 --- a/js/net/src/mock.ts +++ b/js/net/src/mock.ts @@ -132,7 +132,11 @@ export class MockTransport implements WebTransport { * re-rank), so a test can assert how the wire layer prioritized it. The mock always * delivers in creation order, so nothing else observes it. */ - readonly sendStreams: { uni: SendStream[]; bidi: SendStream[] } = { uni: [], bidi: [] }; + readonly sendStreams: { uni: SendStream[]; bidi: SendStream[]; datagrams: SendStream[] } = { + uni: [], + bidi: [], + datagrams: [], + }; #closeResolve!: (info: WebTransportCloseInfo) => void; #bidiController!: ReadableStreamDefaultController; @@ -185,20 +189,21 @@ export class MockTransport implements WebTransport { }, }) : undefined; - const writable = new WritableStream({ - write: (chunk) => { - const peer = this.#peer; - if (!peer) return; - const controller = peer.#datagramController; - if (!controller) return; - try { - // Copy so the caller's scratch buffer reuse can't corrupt queued data. - controller.enqueue(new Uint8Array(chunk)); - } catch { - // Peer closed; drop (datagrams are best-effort). - } - }, - }); + const writable = () => + new WritableStream({ + write: (chunk) => { + const peer = this.#peer; + if (!peer) return; + const controller = peer.#datagramController; + if (!controller) return; + try { + // Copy so the caller's scratch buffer reuse can't corrupt queued data. + controller.enqueue(new Uint8Array(chunk)); + } catch { + // Peer closed; drop (datagrams are best-effort). + } + }, + }); const datagrams: { readable?: ReadableStream; writable?: WritableStream; @@ -207,7 +212,7 @@ export class MockTransport implements WebTransport { outgoingHighWaterMark: number; incomingMaxAge: null; outgoingMaxAge: null; - createWritable?: () => WritableStream; + createWritable?: (options?: WebTransportSendStreamOptions) => SendStream; } = { readable, incomingHighWaterMark: 0, @@ -217,9 +222,10 @@ export class MockTransport implements WebTransport { maxDatagramSize: 1200, }; if (datagramWritable === "writable") { - datagrams.writable = writable; + datagrams.writable = writable(); } else if (datagramWritable === "createWritable") { - datagrams.createWritable = () => writable; + datagrams.createWritable = (options) => + this.#sendStream(writable(), this.sendStreams.datagrams, options); } this.datagrams = datagrams as WebTransportDatagramDuplexStream; } else { diff --git a/js/net/src/rescale.ts b/js/net/src/rescale.ts index 216eb487733..93190f95a38 100644 --- a/js/net/src/rescale.ts +++ b/js/net/src/rescale.ts @@ -1,17 +1,19 @@ -// Kept apart from track.ts so group.ts can import it without a cycle; track.ts re-exports it. +// Shared by stream and datagram tracks, without a cycle through their model modules. import { TimestampMismatch } from "./error.ts"; -import { floorAt, Timestamp } from "./time.ts"; -import type { Info } from "./track.ts"; +import { floorAt, type Timescale, Timestamp } from "./time.ts"; /** - * `timestamp` at `info`'s {@link Info.timescale}, rounded down, as Rust's `Timestamp::convert` does. + * `timestamp` at `info`'s timescale, rounded down, as Rust's `Timestamp::convert` does. * * A track is all timed or all untimed, so a frame must match it: a timestamp on an untimed track, * none on a timed one, or one the scale can't hold throws {@link TimestampMismatch}. A writer that * changes state before its frame reaches the track, such as signing or encrypting it, rescales - * first so a refused timestamp changes nothing. Mirrors the Rust `track::Info::rescale`. + * first so a refused timestamp changes nothing. Mirrors Rust's track and datagram `Info::rescale`. */ -export function rescale(info: Pick, timestamp: Timestamp | undefined): Timestamp | undefined { +export function rescale( + info: { timescale?: Timescale | null }, + timestamp: Timestamp | undefined, +): Timestamp | undefined { const timescale = info.timescale ?? undefined; if ((timescale !== undefined) !== (timestamp !== undefined)) throw new TimestampMismatch(); if (timestamp === undefined || timescale === undefined) return undefined; diff --git a/js/net/src/resume.test.ts b/js/net/src/resume.test.ts index 5481f530ab5..34dbb1c308a 100644 --- a/js/net/src/resume.test.ts +++ b/js/net/src/resume.test.ts @@ -8,7 +8,7 @@ import type { Session as IetfSession } from "./ietf/session.ts"; import { Subscriber as IetfSubscriber } from "./ietf/subscriber.ts"; import { Version as IetfVersion } from "./ietf/version.ts"; import * as Instance from "./instance.ts"; -import { hooks, MAX_DATAGRAMS } from "./internal.ts"; +import { hooks } from "./internal.ts"; import type { Lister } from "./listing.ts"; import { Producer } from "./origin.ts"; import * as Path from "./path.ts"; @@ -869,103 +869,6 @@ test("a fetched group failing as it is handed out releases its route once", asyn } }); -test("datagrams a new copy replays are handed out once", async () => { - const { a, b, swap, active, close } = await setup(); - const ta = copy(a); - const tb = copy(b); - const sub = active.track("seq").subscribe(); - ta.producer.appendDatagram(undefined, enc.encode("a0")); - expect(dec.decode((await sub.recvDatagram())?.payload)).toBe("a0"); - - tb.producer.appendDatagram(undefined, enc.encode("b0")); - await swap(); - tb.producer.appendDatagram(undefined, enc.encode("b1")); - expect(dec.decode((await sub.recvDatagram())?.payload)).toBe("b1"); - - sub.close(); - close(); -}); - -test("a replacement delivers unseen datagrams below the newest sequence", async () => { - const clock = new Manual(); - const installed = clock.install(); - try { - const { a, b, swap, active, close } = await setup(); - const ta = copy(a); - const tb = copy(b); - const sub = active.track("seq").subscribe({ delay: { max: Milli(100) } }); - ta.producer.insertDatagram(10n, undefined, enc.encode("10")); - expect((await sub.recvDatagram())?.sequence).toBe(10n); - await swap(); - tb.producer.insertDatagram(10n, undefined, enc.encode("10")); - tb.producer.insertDatagram(5n, undefined, enc.encode("5")); - const next = hooks.tryRecvDatagram(sub); - expect(next.kind).toBe("datagram"); - if (next.kind === "datagram") expect(next.datagram.sequence).toBe(5n); - expect(hooks.tryRecvDatagram(sub).kind).toBe("idle"); - sub.close(); - close(); - } finally { - installed.close(); - } -}); - -test.each([0, 100])("datagram identities stay within the reader's %p ms budget", async (budget) => { - const clock = new Manual(); - const installed = clock.install(); - try { - const front = new Front(); - const a = new BroadcastProducer(); - const ta = copy(a); - front.serve(a.consume()); - const sub = front.consumer.track("seq").subscribe({ delay: { max: Milli(budget) } }); - const send = (sequence: number) => ta.producer.insertDatagram(BigInt(sequence), undefined, enc.encode("d")); - const recv = () => hooks.tryRecvDatagram(sub); - send(10); - expect(recv().kind).toBe("datagram"); - await clock.advance(Nano(60_000_000_000)); - send(10); - expect(recv().kind).toBe("idle"); - send(5); - expect(recv().kind).toBe("datagram"); - if (budget > 0) { - await clock.advance(Nano(budget * 500_000)); - send(10); - expect(recv().kind).toBe("idle"); - await clock.advance(Nano(budget * 500_000)); - } - send(10); - expect(recv().kind).toBe("datagram"); - sub.close(); - front.consumer.close(); - a.close(); - } finally { - installed.close(); - } -}); - -test("datagram identities are capped by the replay buffer", () => { - const front = new Front(); - const a = new BroadcastProducer(); - const ta = copy(a); - front.serve(a.consume()); - // Superseded identities never expire within an hour, so only the cap bounds them. - const sub = front.consumer.track("seq").subscribe({ delay: { max: Milli(3_600_000) } }); - const send = (sequence: number) => ta.producer.insertDatagram(BigInt(sequence), undefined, enc.encode("d")); - const recv = () => hooks.tryRecvDatagram(sub).kind; - for (let sequence = 0; sequence <= MAX_DATAGRAMS + 1; sequence++) { - send(sequence); - expect(recv()).toBe("datagram"); - } - send(1); - expect(recv()).toBe("idle"); - send(0); - expect(recv()).toBe("datagram"); - sub.close(); - front.consumer.close(); - a.close(); -}); - test("a route that goes for good ends the track as its last copy does", async () => { const { a, active, disposeA, request, close } = await setup(); const ta = copy(a); @@ -1031,27 +934,6 @@ test("a copy that loses its route mid-serve on a route still serving ends the re close(); }); -test("a datagram reader whose copy fails mid-serve on a route still serving is subscribed afresh", async () => { - const { a, active, close } = await setup(); - const ta = copy(a); - const sub = active.track("seq").subscribe(); - ta.producer.appendDatagram(undefined, enc.encode("a0")); - expect(dec.decode((await sub.recvDatagram())?.payload)).toBe("a0"); - - ta.producer.abort(new StreamError(StreamCode.Internal)); - const again = copy(a); - // Datagrams are not replayed, so send them once the fresh copy is subscribed. - const next = sub.recvDatagram(); - await settle(); - await settle(); - again.producer.appendDatagram(undefined, enc.encode("a0")); - again.producer.appendDatagram(undefined, enc.encode("a1")); - expect(dec.decode((await next)?.payload)).toBe("a1"); - - sub.close(); - close(); -}); - test("between routes a reader keeps reading the copy it has", async () => { const { origin, a, b, active, request, disposeA, close } = await setup(); const ta = copy(a); @@ -1560,28 +1442,6 @@ for (const [field, first, second] of [ }); } -test("a datagram reader lets go of the copies a swap replaced", async () => { - const front = new Front(); - const a = new BroadcastProducer(); - const b = new BroadcastProducer(); - const ta = copy(a); - copy(b); - front.serve(a.consume()); - const sub = front.consumer.track("seq").subscribe(); - expect(ta.producer.demand().wanted.peek()).toBeDefined(); - front.serve(b.consume()); - await settle(); - // A read with nothing to deliver is the pass that lets go of a's copy. - void sub.recvDatagram(); - await settle(); - expect(ta.producer.demand().wanted.peek()).toBeUndefined(); - - sub.close(); - front.consumer.close(); - a.close(); - b.close(); -}); - test("a failed group on a route whose replacement was refused ends with the refusal", async () => { const front = new Front(); const a = new BroadcastProducer(); @@ -1660,48 +1520,6 @@ test("a failed group whose replacement cannot fetch ends with the failure, or co } }); -test("draining groups to their end leaves buffered datagrams readable", async () => { - const front = new Front(); - const a = new BroadcastProducer(); - const ta = copy(a); - front.serve(a.consume()); - const sub = front.consumer.track("seq").subscribe(); - ta.write(0n); - ta.producer.appendDatagram(undefined, enc.encode("d")); - ta.producer.finish(); - ta.producer.close(); - expect(drain(sub)).toEqual([0n]); - expect(await sub.recvGroup()).toBeUndefined(); - expect(dec.decode((await sub.recvDatagram())?.payload)).toBe("d"); - - sub.close(); - front.consumer.close(); - a.close(); -}); - -test("a datagram read keeps a replaced copy a group reader still needs", async () => { - const front = new Front(); - const a = new BroadcastProducer(); - const b = new BroadcastProducer(); - const ta = copy(a); - copy(b); - front.serve(a.consume()); - const sub = front.consumer.track("seq").subscribe({ delay: { max: 10_000 as track.Delay["max"] } }); - ta.write(0n); - expect(drain(sub)).toEqual([0n]); - ta.write(1n); - - front.serve(b.consume()); - await settle(); - void sub.recvDatagram(); - expect(drain(sub)).toEqual([1n]); - - sub.close(); - front.consumer.close(); - a.close(); - b.close(); -}); - test("a fetch never reads a replacement refused for its track properties", async () => { const front = new Front(); const a = new BroadcastProducer(); @@ -2589,62 +2407,6 @@ test("a clean end waits for a group the replaced copy still carries", async () = } }); -test.each([false, true])( - "a reader that asked for groups keeps a replaced copy's unread group across a datagram read (datagrams first: %p)", - async (datagramsFirst) => { - const front = new Front(); - const a = new BroadcastProducer(); - const b = new BroadcastProducer(); - const ta = copy(a); - copy(b); - front.serve(a.consume()); - const sub = front.consumer.track("seq").subscribe({ delay: { max: Milli(10_000) } }); - expect(hooks.tryRecvGroup(sub, false).kind).toBe("idle"); - expect(hooks.tryRecvDatagram(sub).kind).toBe("idle"); - ta.write(0n); - front.serve(b.consume()); - await settle(); - if (datagramsFirst) expect(hooks.tryRecvDatagram(sub).kind).toBe("idle"); - expect(drain(sub)).toEqual([0n]); - - sub.close(); - front.consumer.close(); - a.close(); - b.close(); - }, -); - -test.each([Milli(100), Milli(3_600_000)])( - "a datagram reader keeps no replaced copy, whatever its budget (%p ms)", - async (maxDelay) => { - const front = new Front(); - const sources: BroadcastProducer[] = []; - const copies: ReturnType[] = []; - const sub = front.consumer.track("seq").subscribe({ delay: { max: maxDelay } }); - try { - for (let sequence = 0n; sequence < 20n; sequence++) { - const source = new BroadcastProducer(); - const held = copy(source); - sources.push(source); - copies.push(held); - front.serve(source.consume()); - await settle(); - held.write(sequence); - expect(hooks.tryRecvDatagram(sub).kind).toBe("idle"); - await settle(); - const subscribed = copies.filter((held) => held.producer.demand().wanted.peek() !== undefined); - expect(subscribed).toEqual([held]); - } - // Only the serving copy's group is left. - expect(drain(sub)).toEqual([19n]); - } finally { - sub.close(); - front.consumer.close(); - for (const source of sources) source.close(); - } - }, -); - test("a group-only reader of a finite track settles closed and lets go of its copy", async () => { const front = new Front(); const a = new BroadcastProducer(); @@ -2707,28 +2469,6 @@ test("a group-only fork of a finite track settles closed and lets go of its copy a.close(); }); -test("the serving copy closing cleanly settles closed and leaves its datagrams readable", async () => { - const front = new Front(); - const a = new BroadcastProducer(); - const ta = copy(a); - front.serve(a.consume()); - const sub = front.consumer.track("seq").subscribe(); - ta.write(0n); - ta.producer.appendDatagram(undefined, enc.encode("d")); - ta.producer.finish(); - ta.producer.close(); - await settle(); - expect(sub.closed.peek()).toBeNull(); - expect(drain(sub)).toEqual([0n]); - expect(dec.decode((await sub.recvDatagram())?.payload)).toBe("d"); - expect(await sub.recvDatagram()).toBeUndefined(); - expect(await sub.recvGroup()).toBeUndefined(); - - sub.close(); - front.consumer.close(); - a.close(); -}); - test("an end a replaced copy declared holds across a swap", async () => { const front = new Front(); const a = new BroadcastProducer(); @@ -2762,32 +2502,6 @@ test("an end a replaced copy declared holds across a swap", async () => { b.close(); }); -test("an end a replaced copy declared holds once the copy is let go in the same turn", async () => { - const front = new Front(); - const a = new BroadcastProducer(); - const b = new BroadcastProducer(); - const ta = copy(a); - const tb = copy(b); - front.serve(a.consume()); - const sub = front.consumer.track("seq").subscribe(); - front.serve(b.consume()); - await settle(); - - // A datagram read lets go of the replaced copy before its end is reported. - ta.producer.finishAt({ group: 2n, frame: 0 }); - expect(hooks.tryRecvDatagram(sub).kind).toBe("idle"); - expect(hooks.subscriberEnd(sub)).toEqual({ group: 2n, frame: 0 }); - - tb.producer.finishAt({ group: 3n, frame: 0 }); - await settle(); - expect(sub.closed.peek()).toBeInstanceOf(Error); - - sub.close(); - front.consumer.close(); - a.close(); - b.close(); -}); - test("a replaced copy whose session closed binds no end", async () => { const front = new Front(); const a = new BroadcastProducer(); @@ -2992,29 +2706,6 @@ test.each([ }, ); -test("a datagram below the group floor does not keep a finite track's copy", async () => { - const front = new Front(); - const a = new BroadcastProducer(); - const ta = copy(a); - front.serve(a.consume()); - const sub = front.consumer.track("seq").subscribe(); - ta.producer.appendDatagram(undefined, enc.encode("d")); - await settle(); - // The floor passes the datagram: a datagram read would only discard it. - sub.withRange({ start: { included: 1n } }); - ta.write(1n); - ta.producer.finishAt({ group: 2n, frame: 0 }); - expect(drain(sub)).toEqual([1n]); - expect(await sub.recvGroup()).toBeUndefined(); - await settle(); - expect(sub.closed.peek()).toBeNull(); - expect(ta.producer.demand().wanted.peek()).toBeUndefined(); - - sub.close(); - front.consumer.close(); - a.close(); -}); - test("a clean end keeps the serving copy while a group read from a replaced one may continue from it", async () => { const front = new Front(); const a = new BroadcastProducer(); @@ -3156,34 +2847,6 @@ test.each(["handed out", "buffered"])("an end below a group another copy %s is r } }); -test.each(["handed out", "buffered"])("an end below a datagram another copy %s is refused", async (seen) => { - const front = new Front(); - const a = new BroadcastProducer(); - const b = new BroadcastProducer(); - const ta = copy(a); - const tb = copy(b); - front.serve(a.consume()); - const sub = front.consumer.track("seq").subscribe(); - try { - ta.producer.insertDatagram(5n, undefined, enc.encode("5")); - if (seen === "handed out") expect(hooks.tryRecvDatagram(sub).kind).toBe("datagram"); - front.serve(b.consume()); - await settle(); - // The new route delivers an older datagram last, then ends the track before the old route's. - tb.producer.insertDatagram(2n, undefined, enc.encode("2")); - if (seen === "handed out") expect(hooks.tryRecvDatagram(sub).kind).toBe("datagram"); - tb.producer.finishAt({ group: 3n, frame: 0 }); - await settle(); - expect(hooks.tryRecvDatagram(sub).kind).toBe("error"); - expect(sub.closed.peek()).toBeInstanceOf(Error); - } finally { - sub.close(); - front.consumer.close(); - a.close(); - b.close(); - } -}); - test("a group past the declared end on another copy is refused", async () => { const front = new Front(); const a = new BroadcastProducer(); diff --git a/js/net/src/resume.ts b/js/net/src/resume.ts index 721a39c8efc..c4868c61f1e 100644 --- a/js/net/src/resume.ts +++ b/js/net/src/resume.ts @@ -14,8 +14,7 @@ * route got after the switch is not lost. It is kept while it may still deliver a group under its * bound the reader lacks (unread, or still landing within the grace a passed group is waited for), * and while a group still read is read from it or may continue from it (it holds that group, or may - * yet get it). Nothing under the bound is waited for past the reader's budget. Datagrams are - * ephemeral, so no copy is kept for them: a reader that never asked for a group keeps none. + * yet get it). Nothing under the bound is waited for past the reader's budget. * * A new copy is asked from where the reader got to, so it never has the groups below the reader's * local floor that the copies it replaced cached. Lowering that floor asks the newest route for @@ -41,7 +40,7 @@ * still land below that end on a copy it replaced. Its `closed` settles and the reader leaves the * route, keeping each copy only while it holds something left to read, as a direct reader keeps * its buffer. Its end is the first one any copy declared, which every later copy has to match, and - * no copy may hold a group or datagram at or past it. + * no copy may hold a group at or past it. * * @module */ @@ -58,10 +57,8 @@ import { hooks, insertDrop, insertRun, - MAX_DATAGRAMS, maxDelayOf, type Recv, - type RecvDatagram, type Replacement, type ResumeShell, type ResumeSource, @@ -276,7 +273,7 @@ interface Copy { sub: track.Subscriber; /** Where the subscription was asked to start, so demand updates never ask for less. */ floor?: bigint; - /** Replaced by a newer route: its datagrams are the newer copy's to deliver. */ + /** Replaced by a newer route. */ replaced: boolean; /** * Bounded once the newer copy delivered its first group: nothing past this group is taken @@ -286,8 +283,6 @@ interface Copy { until?: bigint | null; /** The group channel ran out: cleanly (`null`) or failed. */ done?: Error | null; - /** The datagram channel ran out. */ - datagramsDone?: Error | null; /** * When a newer group superseded every group under the bound: the newer copy's first group, or * the track's end. What the reader still lacks from it is waited for from here, for no longer @@ -296,7 +291,7 @@ interface Copy { since?: Instant; /** Groups still read that came from this copy, which keep a replaced copy subscribed. */ lease: number; - /** It handed out a group or datagram, so a failure is its route failing mid-serve rather than refusing. */ + /** It handed out a group, so a failure is its route failing mid-serve rather than refusing. */ delivered?: boolean; } @@ -348,16 +343,6 @@ class Reader implements ResumeSource { * reader lacks. Kept until the gaps are backfilled; see {@link #backfillGaps}. */ #resumed = false; - /** The last datagram handed out, which never ages until another supersedes it. */ - #datagram?: bigint; - /** - * Previously handed-out identities, in the order they were superseded. A replacement copy - * replays at most its buffered datagrams, so no more than that many are kept, whatever the - * budget: a publisher cannot grow it. - */ - #datagrams = new Map(); - /** The newest datagram handed out, which a declared end must stay past. */ - #newestDatagram?: bigint; /** Bumped when this reader subscribes a new copy or a copy's last lease ends, waking its parked reads. */ #changed = new Signal(0); #dispose: Dispose[] = []; @@ -455,7 +440,6 @@ class Reader implements ResumeSource { copy.until = undefined; copy.since = undefined; copy.done = undefined; - copy.datagramsDone = undefined; // Resume where a fresh copy would: a group it buffered while replaced, below the newest // another route handed out, is not handed out. if (this.#newest !== undefined) copy.floor = max(copy.floor ?? 0n, this.#newest); @@ -849,50 +833,6 @@ class Reader implements ResumeSource { return lowest?.copy; } - tryDatagram(delivered: Delivered): RecvDatagram { - this.#delivered = delivered; - this.sync(); - this.#learnEnd(); - if (this.#closed) return this.#refused ? { kind: "error", error: this.#refused } : { kind: "done" }; - // Like groups, the last delivery stays fresh; earlier identities cost memory only - // while the reader's budget still covers their successor's arrival. - const at = now(); - const budget = this.#budget() * 1_000_000; - for (const [sequence, since] of this.#datagrams) { - if (at - since < budget) break; - this.#datagrams.delete(sequence); - } - for (let index = this.#copies.length - 1; index >= 0; index--) { - const copy = this.#copies[index]; - // A replaced copy's datagrams are the serving one's to deliver. - if (copy.datagramsDone !== undefined || copy.replaced) continue; - for (;;) { - const recv = hooks.tryRecvDatagram(copy.sub); - if (recv.kind === "datagram") { - const sequence = recv.datagram.sequence; - if (sequence === this.#datagram || this.#datagrams.has(sequence)) continue; - if (this.#datagram !== undefined) { - if (this.#datagrams.size === MAX_DATAGRAMS) { - this.#datagrams.delete(this.#datagrams.keys().next().value as bigint); - } - this.#datagrams.set(this.#datagram, at); - } - this.#datagram = sequence; - this.#newestDatagram = max(this.#newestDatagram ?? 0n, sequence); - copy.delivered = true; - return recv; - } - if (recv.kind === "done") copy.datagramsDone = copy.sub.closed.peek() ?? null; - else if (recv.kind === "error") copy.datagramsDone = recv.error; - break; - } - } - this.#letGo(delivered); - const end = this.#end((copy) => copy.datagramsDone); - if (end.kind === "error" || end.kind === "done") return end; - return { kind: "idle" }; - } - /** * Whether the track ended, once the serving copy's channel (as `done` reads it) ran out: * cleanly, or with a verdict a route failure does not excuse. @@ -1002,8 +942,7 @@ class Reader implements ResumeSource { * newest group the serving copy has. */ #retains(copy: Copy, delivered: Delivered, edge: bigint | undefined): "wait" | "keep" | undefined { - // Datagrams are ephemeral, so only groups keep a copy: a reader that only reads datagrams - // keeps none, whatever its budget. + // Until the first group read, no cached copy needs to be retained. if (!this.#groupsRead) return undefined; // Not yet bounded, it may still feed a group the new copy never has. if (copy.until === undefined && copy.done === undefined) return "wait"; @@ -1146,8 +1085,7 @@ class Reader implements ResumeSource { fn(); }; disposes.push(this.route.changed.changed(once), this.#changed.changed(once)); - for (const copy of this.#copies) - disposes.push(hooks.groupChanged(copy.sub, once), hooks.datagramChanged(copy.sub, once)); + for (const copy of this.#copies) disposes.push(hooks.groupChanged(copy.sub, once)); return dispose; } @@ -1170,7 +1108,7 @@ class Reader implements ResumeSource { return last?.generation === this.#generation ? last : undefined; } - // The logical track's end, learning it from any copy that declared one. A group or datagram + // The logical track's end, learning it from any copy that declared one. A group // at or past it, on any copy or already handed out, claims other content: a direct track // refuses an end below what it received. #learnEnd(): track.Position | undefined { @@ -1186,8 +1124,6 @@ class Reader implements ResumeSource { const seen = (sequence: bigint | undefined) => sequence !== undefined && sequence >= final; if ( seen(this.#newest) || - seen(this.#newestDatagram) || - // The largest counts datagrams too, which the latest group does not. this.#copies.some( (copy) => seen(hooks.latestSequence(copy.sub)) || seen(hooks.peekLargest(copy.sub).largest?.group), ) @@ -1970,6 +1906,17 @@ export class Front { ); } + /** Join the serving source's live datagram track; it holds no cached route copies. */ + datagrams(name: string): import("./datagram.ts").Consumer { + if (!this.#source) throw new NotFound(`datagram track ${name}`); + return this.#source.wire.datagrams(name); + } + + /** Join an already present datagram track without opening a request. */ + liveDatagrams(name: string): import("./datagram.ts").Consumer | undefined { + return this.#source?.wire.liveDatagrams(name); + } + // Wait for the source to change, or `signal` to abort. #next(signal?: AbortSignal): Promise { return until(Signal.race(this.#changed), signal); @@ -2131,7 +2078,9 @@ class FrontConsumer extends broadcast.Consumer { this.#front = front; const view = wireOf(this); registerWire(this, { - requested: view.requested, + ...view, + datagrams: (name) => front.datagrams(name), + liveDatagrams: (name) => front.liveDatagrams(name), tracks: view.tracks, lister: view.lister, subscribe: (name, options) => front.subscribe(name, options), diff --git a/js/net/src/track.test.ts b/js/net/src/track.test.ts index 591856855c8..27d7f808883 100644 --- a/js/net/src/track.test.ts +++ b/js/net/src/track.test.ts @@ -2,8 +2,6 @@ import { expect, spyOn, test } from "bun:test"; import { Manual, Milli, Nano, sleep, Wall, yieldNow } from "@moq/clock"; import { race } from "@moq/signals"; import { ProtocolViolation, StreamCode, StreamError, TimestampMismatch, TooFarBehind } from "./error.ts"; -import { Decoder } from "./flate/decoder.ts"; -import { Encoder } from "./flate/encoder.ts"; import type { Consumer as GroupConsumer, Producer as GroupProducer } from "./group.ts"; import { Version as IetfVersion } from "./ietf/version.ts"; import { hooks } from "./internal.ts"; @@ -225,36 +223,6 @@ test("an ordered owed hole wakes when its ledger gap folds", async () => { } }); -test("ordered reads pass accounted sequences and honor the feed and reader floors", () => { - const clock = manualTime(0); - const producer = receivedTrack(); - const reader = producer.subscribe({ delay: { max: null } }).ordered(); - try { - putGroup(producer, 0n); - producer.appendDatagram(undefined, enc.encode("d")); - putGroup(producer, 2n); - expect(drainOrdered(reader)).toEqual([0n, 2n]); - hooks.dropTrackGroups(producer, 3n, 4n, new StreamError(StreamCode.Old)); - putGroup(producer, 4n); - expect(drainOrdered(reader)).toEqual([4n]); - producer.createGroup(5n).abort(new StreamError(StreamCode.Cancel)); - putGroup(producer, 6n); - expect(drainOrdered(reader)).toEqual([5n, 6n]); - hooks.startTrack(producer, { start: 8n }); - putGroup(producer, 8n); - expect(drainOrdered(reader)).toEqual([8n]); - putGroup(producer, 10n); - expect(drainOrdered(reader)).toEqual([]); - reader.withRange({ start: { included: 10n } }); - expect(drainOrdered(reader)).toEqual([10n]); - } finally { - reader.close(); - producer.finish(); - producer.close(); - clock.installed.close(); - } -}); - test("ordered owed holes follow zero, narrowed, unlimited, and publisher budgets", async () => { const clock = manualTime(0); try { @@ -295,56 +263,6 @@ test("ordered owed holes follow zero, narrowed, unlimited, and publisher budgets } }); -test("ordered owed waits wake on a datagram, a drop, a declared start, or track close", async () => { - const clock = manualTime(0); - try { - for (const resolve of ["datagram", "drop", "start", "close"] as const) { - const producer = receivedTrack(); - const reader = producer.subscribe({ delay: { max: null } }).ordered(); - putGroup(producer, 1n); - // With no declared start the first arrival is the floor. - expect(drainOrdered(reader)).toEqual([1n]); - hooks.startTrack(producer, { start: 0n }); - putGroup(producer, 3n); - let ready = false; - const held = reader.nextGroup().then((group) => { - ready = true; - return group; - }); - await settle(); - expect(ready).toBe(false); - // A datagram shares the sequence namespace, so it accounts for the hole too. - if (resolve === "datagram") producer.insertDatagram(2n, undefined, new Uint8Array([2])); - if (resolve === "drop") hooks.dropTrackGroups(producer, 2n, 3n, new StreamError(StreamCode.Old)); - if (resolve === "start") hooks.startTrack(producer, { start: 3n }); - if (resolve === "close") { - producer.finish(); - producer.close(); - } - await settle(); - expect(ready).toBe(true); - expect((await held)?.sequence).toBe(3n); - reader.close(); - if (resolve !== "close") { - producer.finish(); - producer.close(); - } - } - const producer = receivedTrack(); - hooks.startTrack(producer, { start: 0n }); - const reader = producer.subscribe({ delay: { max: null } }).ordered(); - putGroup(producer, 1n); - expect(drainOrdered(reader)).toEqual([]); - putGroup(producer, 0n); - expect(drainOrdered(reader)).toEqual([0n, 1n]); - reader.close(); - producer.finish(); - producer.close(); - } finally { - clock.installed.close(); - } -}); - test("wanted reflects subscriber demand and clears when the last one leaves", async () => { const producer = new TrackProducer("test"); @@ -440,31 +358,6 @@ test("a join over the refresh ratio wants the group after the newest", async () } }); -test("a join wants past a datagram after the newest group", () => { - const producer = new TrackProducer("test").accept({ refresh: 1 }); - refreshGroup(producer, 4); - expect(producer.appendDatagram(undefined, enc.encode("dgram"))).toBe(1n); - const subscriber = producer.subscribe(); - expect(producer.demand().wanted.peek()).toBe(2n); - subscriber.close(); -}); - -test("a datagram never overtakes a pending refresh", () => { - const producer = new TrackProducer("test").accept({ refresh: 1 }); - refreshGroup(producer, 4); - const subscriber = producer.subscribe(); - expect(producer.demand().wanted.peek()).toBe(1n); - - expect(producer.appendDatagram(undefined, enc.encode("dgram"))).toBe(1n); - expect(producer.demand().wanted.peek()).toBe(2n); - - // Group 2 answers it, and later datagrams ask for nothing. - refreshGroup(producer, 0); - producer.appendDatagram(undefined, enc.encode("dgram")); - expect(producer.demand().wanted.peek()).toBe(2n); - subscriber.close(); -}); - test("joins coalesce until the wanted group lands", () => { const producer = new TrackProducer("test").accept({ refresh: 1 }); refreshGroup(producer, 4); @@ -533,200 +426,11 @@ test("rescale matches the track's timeline, rounded down as Rust's convert does" expect(() => rescale({}, Timestamp.fromMillis(1))).toThrow(TimestampMismatch); }); -test("appendDatagram shares the group sequence counter", () => { - const producer = new TrackProducer("test"); - const ts = Timestamp.fromMillis(10); - - // Interleave groups and datagrams: they draw from one monotonic counter. - expect(producer.appendGroup().sequence).toBe(0n); - expect(producer.appendDatagram(ts, enc.encode("a"))).toBe(1n); - expect(producer.appendGroup().sequence).toBe(2n); - expect(producer.appendDatagram(ts, enc.encode("b"))).toBe(3n); -}); - -test("appendDatagram delivers to a subscriber", async () => { - const producer = new TrackProducer("test"); - const track = producer.subscribe(); - const ts = Timestamp.fromMillis(42); - - const seq = producer.appendDatagram(ts, enc.encode("hello")); - const got = await track.recvDatagram(); - expect(got?.sequence).toBe(seq); - expect(got?.timestamp).toBe(ts); - expect(got && dec.decode(got.payload)).toBe("hello"); -}); - // The in-range datagrams, sent between the dropped ones, show the reader is live, so the -// range is what dropped the others. -test("the group range bounds datagrams", async () => { - const producer = new TrackProducer("test"); - const track = producer.subscribe().withRange({ start: { included: 5n }, end: { excluded: 7n } }); - for (const sequence of [4n, 5n, 7n, 6n]) { - producer.insertDatagram(sequence, Timestamp.fromMillis(0), enc.encode("x")); - } - producer.finish(); - producer.close(); - - expect((await track.recvDatagram())?.sequence).toBe(5n); - expect((await track.recvDatagram())?.sequence).toBe(6n); - expect(await track.recvDatagram()).toBeUndefined(); -}); - -for (const timed of [true, false]) { - test(`late ${timed ? "timed" : "untimed"} subscribers do not replay datagrams`, async () => { - const producer = new TrackProducer("test").accept(timed ? { timescale: Timescale.MICRO } : {}); - const timestamp = timed ? Timestamp.fromMillis(180_000) : undefined; - producer.appendDatagram(timestamp, enc.encode("before")); - const subscriber = producer.subscribe({ delay: { max: Milli(50) } }); - const sequence = producer.appendDatagram(timestamp, enc.encode("after")); - producer.finish(); - producer.close(); - const datagram = await subscriber.recvDatagram(); - expect(datagram?.sequence).toBe(sequence); - expect(datagram && dec.decode(datagram.payload)).toBe("after"); - expect(await subscriber.recvDatagram()).toBeUndefined(); - }); -} - -test("datagram buffers drop oldest at capacity without delaying active subscribers", async () => { - const producer = new TrackProducer("test"); - const slow = producer.subscribe(); - const fast = producer.subscribe(); - const count = 192; - for (let sequence = 0n; sequence < BigInt(count); sequence++) { - producer.appendDatagram(Timestamp.fromMillis(0), enc.encode("x")); - expect((await fast.recvDatagram())?.sequence).toBe(sequence); - } - producer.finish(); - producer.close(); - for (let sequence = BigInt(count - 64); sequence < BigInt(count); sequence++) { - expect((await slow.recvDatagram())?.sequence).toBe(sequence); - } - expect(await slow.recvDatagram()).toBeUndefined(); -}); // A received compressed datagram inflates far past its wire size, so each reader inflates -// its own copy: a slow reader's queue holds only the slices, never what a fast one realized. -test("a datagram queue holds no plain copy another reader realized", async () => { - const producer = new TrackProducer("test").accept({ compress: true }); - const slow = producer.subscribe(); - const fast = producer.subscribe(); - const plain = new Uint8Array(65535); - const slice = new Encoder().frame(plain); - expect(slice.length).toBeLessThan(128); - const decode = spyOn(Decoder.prototype, "frame"); - try { - const count = 64; - for (let sequence = 0n; sequence < BigInt(count); sequence++) { - hooks.writeDatagram(producer, { sequence, payload: slice }, plain.length); - } - const arrived = decode.mock.calls.length; - const read = []; - for (let sequence = 0; sequence < count; sequence++) { - const datagram = await fast.recvDatagram(); - expect(datagram?.payload).toEqual(plain); - read.push(datagram); - } - expect(decode.mock.calls.length).toBe(arrived + count); - for (let sequence = 0; sequence < count; sequence++) { - const datagram = await slow.recvDatagram(); - expect(datagram).not.toBe(read[sequence]); - expect(datagram?.payload).toEqual(plain); - } - expect(decode.mock.calls.length).toBe(arrived + 2 * count); - } finally { - decode.mockRestore(); - } -}); - -test("insertDatagram preserves an explicit sequence", async () => { - const producer = new TrackProducer("test"); - const track = producer.subscribe(); - - // A relay forwarding an upstream datagram keeps its sequence number. - producer.insertDatagram(100n, Timestamp.fromMillis(5), enc.encode("x")); - expect((await track.recvDatagram())?.sequence).toBe(100n); - - // The shared counter advanced past it, so the next appended group continues from 101. - expect(producer.appendGroup().sequence).toBe(101n); -}); // Datagrams and groups are separate channels that share only a sequence namespace, so -// consuming one must not move the other's cursor. -test("recvDatagram leaves the ordered group cursor alone", async () => { - const producer = new TrackProducer("test"); - const datagrams = producer.subscribe(); - const track = producer.subscribe({ delay: { max: Milli(5000) } }).ordered(); - - producer.insertDatagram(5n, Timestamp.fromMillis(5), enc.encode("x")); - expect((await datagrams.recvDatagram())?.sequence).toBe(5n); - - producer.createGroup(3n); - producer.createGroup(6n); - expect((await track.nextGroup())?.sequence).toBe(3n); - expect((await track.nextGroup())?.sequence).toBe(6n); -}); - -test("insertDatagram leaves a gap and does not rewind out of order", async () => { - const producer = new TrackProducer("test"); - const track = producer.subscribe(); - const ts = Timestamp.fromMillis(0); - - producer.insertDatagram(10n, ts, enc.encode("high")); - producer.insertDatagram(5n, ts, enc.encode("low")); - expect((await track.recvDatagram())?.sequence).toBe(10n); - expect((await track.recvDatagram())?.sequence).toBe(5n); - expect(producer.appendDatagram(ts, enc.encode("next"))).toBe(11n); - expect(producer.appendGroup().sequence).toBe(12n); -}); - -test("insertDatagram duplicate is best-effort", async () => { - const producer = new TrackProducer("test"); - const track = producer.subscribe(); - const ts = Timestamp.fromMillis(0); - - producer.insertDatagram(3n, ts, enc.encode("first")); - producer.insertDatagram(3n, ts, enc.encode("again")); - expect(dec.decode((await track.recvDatagram())?.payload ?? new Uint8Array())).toBe("first"); - expect(dec.decode((await track.recvDatagram())?.payload ?? new Uint8Array())).toBe("again"); - expect(producer.appendDatagram(ts, enc.encode("next"))).toBe(4n); -}); - -test("insertDatagram stale does not rewind after append", async () => { - const producer = new TrackProducer("test"); - const track = producer.subscribe(); - const ts = Timestamp.fromMillis(0); - - expect(producer.appendDatagram(ts, enc.encode("0"))).toBe(0n); - expect(producer.appendDatagram(ts, enc.encode("1"))).toBe(1n); - producer.insertDatagram(0n, ts, enc.encode("stale")); - expect((await track.recvDatagram())?.sequence).toBe(0n); - expect((await track.recvDatagram())?.sequence).toBe(1n); - expect((await track.recvDatagram())?.sequence).toBe(0n); - expect(producer.appendDatagram(ts, enc.encode("2"))).toBe(2n); - expect(producer.appendGroup().sequence).toBe(3n); -}); - -test("insertDatagram after close is refused", () => { - const producer = new TrackProducer("test"); - const ts = Timestamp.fromMillis(0); - producer.finish(); - producer.close(); - expect(() => producer.insertDatagram(0n, ts, enc.encode("x"))).toThrow("track is closed"); - expect(() => producer.appendDatagram(ts, enc.encode("x"))).toThrow("track is closed"); -}); - -test("insertDatagram after abort is refused", () => { - const producer = new TrackProducer("test"); - producer.abort(new Error("cancel")); - expect(() => producer.insertDatagram(0n, Timestamp.fromMillis(0), enc.encode("x"))).toThrow("track is closed"); -}); - -test("appendDatagram rejects a payload over the QUIC datagram frame ceiling", () => { - const producer = new TrackProducer("test"); - expect(() => producer.appendDatagram(Timestamp.fromMillis(0), new Uint8Array(65536))).toThrow(); - expect(() => producer.insertDatagram(0n, Timestamp.fromMillis(0), new Uint8Array(65536))).toThrow(); -}); test("subscriber options and updates are forwarded to the producer's aggregate", async () => { const producer = new TrackProducer("test"); @@ -1122,19 +826,6 @@ test("a time range reads the groups containing its times, settling its end as gr expect(hooks.servingEnd(reader)).toBe(4n); }); -test("a datagram is judged against a time range by its own timestamp", () => { - const producer = new TrackProducer("test"); - producer.accept({ timescale: Timescale.MILLI }); - const reader = producer.subscribe({ - ranges: [{ start: { absolute: 0n }, end: { time: Timestamp.fromMillis(3500) } }], - }); - producer.insertDatagram(0n, Timestamp.fromMillis(3600), enc.encode("late")); - producer.insertDatagram(1n, Timestamp.fromMillis(3400), enc.encode("in")); - const recv = hooks.tryRecvDatagram(reader); - expect(recv.kind === "datagram" && dec.decode(recv.datagram.payload)).toBe("in"); - expect(hooks.tryRecvDatagram(reader)).toEqual({ kind: "idle" }); -}); - // A reached end leaves a held group that may still start, so a time start still waits on it. test("a time start waits on groups below a reached end that have yet to start", () => { for (const order of ["ascending", "descending"] as const) { @@ -1193,18 +884,6 @@ test("an untimed track or a moq-transport copy refuses a time anchor", () => { expect(() => copy.subscribe({ ranges }).tryRecvGroup()).toThrow("moq-transport"); }); -// Waiting reads see the refusal once the track is accepted, with no group or datagram arriving. -test("accepting an untimed track refuses a waiting read's time anchor", async () => { - const producer = new TrackProducer("test"); - const reader = producer.subscribe({ ranges: [{ start: { time: Timestamp.fromMillis(0) } }] }); - const group = reader.recvGroup(); - const datagram = reader.recvDatagram(); - producer.accept({}); - const [grouped, received] = await Promise.allSettled([group, datagram]); - expect(grouped).toMatchObject({ status: "rejected", reason: expect.any(TimestampMismatch) }); - expect(received).toMatchObject({ status: "rejected", reason: expect.any(TimestampMismatch) }); -}); - test("only unstated minimums aggregate to an unstated one", () => { const producer = new TrackProducer("test"); producer.subscribe({ delay: { min: null } }); @@ -1338,18 +1017,6 @@ test("a long group is stale once its successor falls behind", async () => { }); // Datagrams are unordered by construction, so the sequence cursor carries them too: a -// track using both channels needs one subscription, not two. -test("the ordered handle carries datagrams", async () => { - const producer = new TrackProducer("test"); - const track = producer.subscribe({ delay: { max: Milli(5000) } }).ordered(); - - producer.insertDatagram(5n, Timestamp.fromMillis(5), enc.encode("x")); - producer.createGroup(3n); - - expect((await track.recvDatagram())?.sequence).toBe(5n); - // The datagram did not consume the group cursor. - expect((await track.nextGroup())?.sequence).toBe(3n); -}); // Both cursors write a backlog off: a buffered group is not free to deliver, since a // consumer reading one that is already too old plays it at 1x and never catches up. @@ -2437,30 +2104,6 @@ test("a subscription cap chooses its own live edge", async () => { } }); -test("subscription updates bound each reader's datagrams", async () => { - const producer = new TrackProducer("test"); - const everything = producer.subscribe({ delay: { max: Milli(5000) } }); - const track = producer.subscribe({ ranges: [{ start: { absolute: 1n }, end: { absolute: 2n } }] }); - try { - for (let sequence = 0n; sequence < 4n; sequence++) - producer.insertDatagram(sequence, undefined, new Uint8Array([Number(sequence)])); - expect((await track.recvDatagram())?.sequence).toBe(1n); - track.update({ ranges: [{ start: { absolute: 3n }, end: { absolute: 4n } }] }); - for (let sequence = 2n; sequence < 5n; sequence++) - producer.insertDatagram(sequence, undefined, new Uint8Array([Number(sequence)])); - expect((await track.recvDatagram())?.sequence).toBe(3n); - const pending = track.recvDatagram(); - expect(await Promise.race([pending, Promise.resolve("pending")])).toBe("pending"); - track.close(); - await pending; - } finally { - track.close(); - everything.close(); - producer.finish(); - producer.close(); - } -}); - test("withRange with an excluded end of 0 is the empty range and does not ride another subscriber's demand", async () => { const producer = new TrackProducer("test"); const everything = producer.subscribe({ delay: { max: Milli(5000) } }); @@ -2752,19 +2395,6 @@ test("the first group read commits the cursor and refuses the other order", asyn }); // Datagrams are a separate cursor on either handle, so reading them must not commit -// the subscription to a group order. -test("recvDatagram does not commit the group cursor", async () => { - const producer = new TrackProducer("test"); - const track = producer.subscribe({ delay: { max: Milli(5000) } }); - - producer.insertDatagram(0n, Timestamp.fromMillis(0), enc.encode("x")); - expect((await track.recvDatagram())?.sequence).toBe(0n); - - // Still uncommitted: the subscription can go sequence-ordered. - const ordered = track.ordered(); - producer.createGroup(1n); - expect((await ordered.nextGroup())?.sequence).toBe(1n); -}); test("nextGroup returns undefined when track closes", async () => { const producer = new TrackProducer("test"); @@ -2805,21 +2435,6 @@ test("a closed track still delivers a group parked above the cap once the cap is expect(await track.nextGroup()).toBeUndefined(); }); -// The final boundary counts datagrams: they share the group sequence namespace, and a -// Rust peer feeds SUBSCRIBE_FINISH into finish_at, so a boundary that ignored a trailing -// datagram would finalize the track before it arrives. -test("end reports the head after the highest produced sequence, datagrams included", async () => { - const producer = new TrackProducer("test"); - const track = producer.subscribe(); - expect(hooks.subscriberEnd(track)).toBeUndefined(); - - producer.createGroup(0n); - producer.appendDatagram(Timestamp.fromMillis(1), enc.encode("x")); - producer.finish(); - producer.close(); - expect(hooks.subscriberEnd(track)).toEqual({ group: 2n, frame: 0 }); -}); - // 0 is the only encoding for "nothing produced"; an abort declares no boundary at all. test("end is group 0 for an empty track and undefined after an abort", async () => { const empty = new TrackProducer("test"); @@ -2977,16 +2592,6 @@ test("a recreated largest group is reported before the end", async () => { producer.close(); }); -test("a datagram raises the largest", async () => { - const producer = new TrackProducer("test").accept({ timescale: Timescale.MILLI }); - const subscriber = producer.subscribe(); - producer.appendGroup().finish(); - expect(producer.appendDatagram(Timestamp.now(), enc.encode("d"))).toBe(1n); - expect(await subscriber.largest()).toEqual({ group: 1n, frame: 0 }); - producer.finish(); - producer.close(); -}); - /** Write untimed group `sequence` with `frames` frames, left open. */ function framedGroup(producer: TrackProducer, sequence: bigint, frames: number): GroupProducer { const group = producer.createGroup(sequence); @@ -3210,24 +2815,6 @@ test("malformed group bounds do not partially advance the cursor", () => { track.close(); }); -test("finishAt refuses an end at or below a produced sequence", () => { - const producer = new TrackProducer("test"); - producer.createGroup(5n); - - // Exclusive, so it must be above the highest produced sequence. - expect(() => producer.finishAt({ group: 5n, frame: 0 })).toThrow(); - expect(() => producer.finishAt({ group: 4n, frame: 0 })).toThrow(); - producer.finishAt({ group: 6n, frame: 0 }); - - // A second end is refused, as is any group at or past the first. - expect(() => producer.finishAt({ group: 7n, frame: 0 })).toThrow(); - expect(() => producer.createGroup(6n)).toThrow(); - expect(() => producer.appendGroup()).toThrow(); - expect(() => producer.insertDatagram(6n, Timestamp.now(), new Uint8Array(1))).toThrow(); - producer.finish(); - producer.close(); -}); - test("finishAt declares an end ahead of the live edge without ending the track", async () => { const producer = new TrackProducer("test"); const track = producer.subscribe({ delay: { max: Milli(60_000) } }); @@ -3297,33 +2884,6 @@ for (const ordered of [false, true]) { }); } -for (const channel of ["group", "datagram"]) { - test(`a ${channel} reaching the end wakes both reader channels`, async () => { - const producer = new TrackProducer("test"); - const track = producer.subscribe(); - try { - producer.finishAt({ group: 3n, frame: 0 }); - const groupRead = track.recvGroup(); - const datagramRead = track.recvDatagram(); - if (channel === "group") { - const group = producer.createGroup(2n); - group.finish(); - } else { - producer.insertDatagram(2n, Timestamp.fromMillis(0), enc.encode("tail")); - } - expect((await groupRead)?.sequence).toBe(channel === "group" ? 2n : undefined); - expect((await datagramRead)?.sequence).toBe(channel === "datagram" ? 2n : undefined); - expect(await track.recvGroup()).toBeUndefined(); - expect(await track.recvDatagram()).toBeUndefined(); - expect(track.closed.peek()).toBeUndefined(); - } finally { - producer.finish(); - producer.close(); - track.close(); - } - }); -} - test("finished rejects when the track aborts or closes without an end", async () => { const aborted = new TrackProducer("test"); const pending = aborted.subscribe().finished(); @@ -3502,14 +3062,6 @@ test("an abort after the declared end settles ends clean", async () => { } }); -test("an abort after the declared end reached by a datagram ends clean", () => { - const producer = new TrackProducer("test").accept({ timescale: Timescale.MILLI }); - producer.appendDatagram(Timestamp.fromMillis(0), enc.encode("d")); - producer.finishAt({ group: 1n, frame: 0 }); - producer.abort(new Error("boom")); - expect(producer.closed.peek()).toBeNull(); -}); - test("a settled end survives a late group and an abort", async () => { const producer = new TrackProducer("test").accept({ timescale: Timescale.MILLI }); const subscriber = producer.subscribe(); @@ -3648,19 +3200,6 @@ function untimedTrack(count: number): TrackProducer { return producer; } -test("a null timescale declares an untimed track, like an omitted one", () => { - expect(infoDefaults({ timescale: null }).timescale).toBeUndefined(); - - const producer = new TrackProducer("test").accept({ timescale: null }); - const group = producer.appendGroup(); - expect(() => group.writeFrame({ payload: enc.encode("x"), timestamp: Timestamp.fromMillis(1) })).toThrow( - TimestampMismatch, - ); - group.writeFrame({ payload: enc.encode("y") }); - expect(() => producer.appendDatagram(Timestamp.fromMillis(1), enc.encode("x"))).toThrow(TimestampMismatch); - producer.appendDatagram(undefined, enc.encode("y")); -}); - test("a frame whose timedness disagrees with its track is refused", async () => { const timed = new TrackProducer("timed").accept({ timescale: Timescale.MILLI }); const group = timed.appendGroup(); @@ -3682,18 +3221,6 @@ test("a frame whose timedness disagrees with its track is refused", async () => expect(frame?.timestamp).toBeUndefined(); }); -test("a datagram whose timedness disagrees with its track is refused", async () => { - const timed = new TrackProducer("timed").accept({ timescale: Timescale.MILLI }); - expect(() => timed.appendDatagram(undefined, enc.encode("x"))).toThrow(TimestampMismatch); - - const untimed = new TrackProducer("untimed").accept({}); - const subscriber = untimed.subscribe(); - expect(() => untimed.insertDatagram(0n, Timestamp.fromMillis(1), enc.encode("x"))).toThrow(TimestampMismatch); - untimed.appendDatagram(undefined, enc.encode("y")); - const got = await subscriber.recvDatagram(); - expect(got?.timestamp).toBeUndefined(); -}); - test("unknown track metadata keeps presentation-time expiry until accepted", () => { const clock = manualTime(10_000); const producer = new TrackProducer("pending"); @@ -3751,18 +3278,6 @@ test("an untimed subscriber without a start begins at the latest group under its expect(drain(subscriber)).toEqual([2n]); }); -test("accept closes the track when something written before it disagrees", async () => { - const timed = new TrackProducer("groups"); - timed.appendGroup().writeFrame({ payload: enc.encode("x"), timestamp: Timestamp.fromMillis(1) }); - expect(() => timed.accept({})).toThrow(TimestampMismatch); - await expect(timed.info()).rejects.toBeInstanceOf(TimestampMismatch); - - const datagrams = new TrackProducer("datagrams"); - datagrams.subscribe(); - datagrams.appendDatagram(Timestamp.fromMillis(1), enc.encode("x")); - expect(() => datagrams.accept({})).toThrow(TimestampMismatch); -}); - test("a single-frame write that throws aborts its group", () => { const producer = new TrackProducer("test").accept({ maxAge: null }); const subscriber = producer.subscribe({ delay: { max: Milli(60_000) } }); @@ -4514,16 +4029,6 @@ test("an evicted group still fills a pending tail", async () => { } }); -test("a datagram fills its sequence in a pending tail", async () => { - const { producer, reader, write } = pendingTail(2n); - producer.insertDatagram(0n, undefined, enc.encode("x")); - write(1n); - expect((await reader.recvGroup())?.sequence).toBe(1n); - expect(await reader.recvGroup()).toBeUndefined(); - producer.finish(); - producer.close(); -}); - // SUBSCRIBE_DROP says the publisher gave the range up, so the tail owes nothing there: readers // end at the boundary, and a later abort cut nothing off. for (const abort of [false, true]) { diff --git a/js/net/src/track.ts b/js/net/src/track.ts index 2803e35e689..8c9f549eeda 100644 --- a/js/net/src/track.ts +++ b/js/net/src/track.ts @@ -1,12 +1,11 @@ /** - * Track role handles: a live stream of groups (and best-effort datagrams) within a broadcast. + * Track role handles: a live stream of groups within a broadcast. * * @module */ import { Instant, Milli, Nano, now, sleepUntil } from "@moq/clock"; import { type Dispose, type GetPromise, type Getter, Once, Signal } from "@moq/signals"; -import type { Datagram } from "./datagram.ts"; import { Delivered } from "./delivered.ts"; import { GroupTooLarge, @@ -18,7 +17,6 @@ import { toStreamCode, } from "./error.ts"; import { type Asker, FillState, type Requests, requestsOf } from "./fill.ts"; -import { datagramSlice, type Stored, sliceView } from "./flate/group.ts"; import { type Frame, type Consumer as GroupConsumer, Producer as GroupProducer } from "./group.ts"; import type { IetfVersion } from "./ietf/version.ts"; import { @@ -31,11 +29,9 @@ import { insertDrop, insertRun, type Land, - MAX_DATAGRAMS, maxDelayOf, NO_MAX_DELAY, type Recv, - type RecvDatagram, type ResumeSource, type Run, removeRun, @@ -63,7 +59,6 @@ import { } from "./window.ts"; import { type Broadcast as BroadcastWire, registerTrackConsumer } from "./wire.ts"; -export type { Datagram } from "./datagram.ts"; export { Fill, Requests } from "./fill.ts"; // The group a non-blocking read takes, skipping drops and throwing the track's error; @@ -88,13 +83,6 @@ const PRUNE_SLICES = 8; // moq-transport as 127, one urgency on both. const DEFAULT_PRIORITY = 0; -/** - * Sanity cap on a datagram payload: the QUIC DATAGRAM frame ceiling. The real limit is - * per-hop (the negotiated transport datagram size minus a small header) and oversize - * datagrams are dropped there; a payload above this cap could never fit anywhere. - */ -const MAX_DATAGRAM_BYTES = 65535; - // A group ended with one of these is a deliberate hole, never a cut readers see: cache policy, // a reader that fell behind, or a cancel such as a relay's own unsubscribe. Mirrors Rust's // `group::Handle::commit_abort`. @@ -144,7 +132,7 @@ export interface Info { compress?: boolean; /** * Units per second for this track's frame timestamps, or omitted (`undefined` or `null`, - * which mean the same) for an untimed track, whose frames and datagrams carry no timestamp. + * which mean the same) for an untimed track, whose frames carry no timestamp. * * There is no default: a timed track declares its scale, such as {@link Timescale | Timescale.MILLI}, * or {@link Timescale | Timescale.MICRO} for media that needs sub-millisecond timing. Reported in @@ -292,7 +280,7 @@ export interface Groups { * one that has, reading nothing of its range meanwhile. An end settles once a group starting * after it is known and every sequence below that one is accounted for; until then reads stop * at the first held group that may still start, so at most that one group past the end is read - * before its first frame shows where it starts. A datagram is judged by its own timestamp. A time anchor on an untimed track is refused + * before its first frame shows where it starts. A time anchor on an untimed track is refused * with {@link TimestampMismatch}, and on a moq-transport copy, whose timestamps may rewind. * * Older moq-lite versions carry only absolute anchors, the live start, and the oldest start, @@ -693,15 +681,12 @@ class TrackState { // Bumped when a wire subscription declares or resolves where its live feed starts, so a // publisher holding its first group for that answer wakes. startChanged = new Signal(0); - /** Best-effort datagram channel, parallel to {@link groups}; a bounded send buffer per subscriber. */ - datagrams = new Signal([]); latest?: bigint; - // One past the highest group or datagram received, including evicted content. + // One past the highest group received, including evicted content. received = new Signal(0n); /** * The exclusive end, declared by {@link Producer.finishAt} or stamped by a clean close - * at the head of the sequence after the highest produced. Groups and datagrams share - * the namespace, so it can sit past `latest + 1` (which only tracks groups). Mirrors + * at the head of the sequence after the highest produced. Mirrors * the Rust `end`. */ end = new Signal(undefined); @@ -718,7 +703,7 @@ class TrackState { update: Signal; /** Resolved once the producer commits the immutable properties. */ info = new Signal(undefined); - /** Newest group head, datagram, frame, or location reported by the publisher. */ + /** Newest group head, frame, or location reported by the publisher. */ largest = new Signal(undefined); // The group {@link largest} first named, where a relative anchor resolved before any group // existed starts. @@ -813,8 +798,6 @@ let goneAt: (producer: Producer, sequence: bigint) => bigint | undefined; let objectLargest: (producer: Producer) => Position | undefined; // The fills serving the producer's misses; assigned in its static block. let fillsOf: (producer: Producer) => FillState; -// The sequences of the newest datagrams the producer holds; assigned in its static block. -let heldDatagrams: (producer: Producer) => readonly bigint[]; // The sequence-order cursor lives inside `Subscriber` (it shares the group buffer and the // drift anchor with the arrival cursor), so `Ordered` reaches it through this bridge, @@ -828,7 +811,6 @@ let ordered_: { readString(subscriber: Subscriber): Promise; readJson(subscriber: Subscriber): Promise; readBool(subscriber: Subscriber): Promise; - recvDatagram(subscriber: Subscriber): Promise; }; // Constructs a Demand from within this module; assigned in the class's static block. @@ -865,8 +847,7 @@ export class Demand { * The group subscribers want: `undefined` while nobody subscribes, otherwise the sequence * the next group should have at least. A publisher opens a group whenever it passes its * largest. The first subscriber of a track with no group wants `0`, and a join judged by - * {@link Info.refresh} wants the next free sequence until a group lands there, so a - * datagram taking it moves the ask past. It never goes back. Watch it + * {@link Info.refresh} wants the next free sequence until a group lands there. It never goes back. Watch it * (`effect.get` / `.peek()`) to follow demand. */ get wanted(): Getter { @@ -907,7 +888,7 @@ export class Producer { // The producer's own state is the source of truth (info/closed); subscribers // read mirrored sinks, never this state directly. #state = new TrackState(); - // One past the highest group or datagram this producer received, like the Rust + // One past the highest group this producer received, like the Rust // `max_sequence`, and where the next append lands. #received = 0n; #subscribing = new Signal(false); @@ -945,12 +926,10 @@ export class Producer { // The lowest group they want while no refresh is pending, never lowered. #floor = 0n; // A join over the refresh ratio asked for a group past every sequence written, and none - // has landed yet. Read against `#received` rather than stored as a number, so a datagram - // taking the next sequence can never leave the ask behind. + // has landed yet. #refresh = false; #demand: Demand; - // The largest group's content may still grow: it has not closed cleanly, and it is not a - // datagram. An aborted one stays open, since it can be recreated. + // The largest group may still grow until it closes cleanly; an aborted one can be recreated. #largestOpen = false; // Only real objects or upstream metadata advance this, never the head of an empty group. #objectLargest?: Position; @@ -968,11 +947,7 @@ export class Producer { #starts: FillStarts = {}; // The misses its readers asked of a {@link Requests} handle. #fills: FillState; - // The sequences of the newest datagrams, as many as a reader buffers: the track's own record - // that a datagram took them, like the Rust track's datagram buffer, which a reader reading its - // copy does not empty. A fill never asks for one. - #datagramSequences: bigint[] = []; - // The sequences the live feed delivered, as a group or a datagram: what a pending tail is + // The sequences the live feed delivered, as a group: what a pending tail is // judged by. Eviction never touches it. A gap folds away a subscription's grace after a // later arrival opened it, as the session's tail does, so this holds only the gaps still // worth waiting for. @@ -1024,7 +999,6 @@ export class Producer { producer.#starts = starts; }; hooks.receiveGroup = (producer, { sequence, start }) => producer.#createGroup(sequence, start); - hooks.writeDatagram = (producer, datagram, inflated) => producer.#insertDatagram(datagram, inflated); sourceStart = (producer) => producer.#served && producer.#feedStart(); fillFloor = (producer, named) => producer.#fillFloor(named); startsOf = (producer) => producer.#bounds; @@ -1057,7 +1031,6 @@ export class Producer { hooks.reportedGroup = (producer, group) => producer.#advanceLargest({ group, frame: 0 }); objectLargest = (producer) => producer.#objectLargest; fillsOf = (producer) => producer.#fills; - heldDatagrams = (producer) => producer.#datagramSequences; cachedHolds = (producer, sequence) => { const entry = producer.#cached.get(sequence); return entry !== undefined && !(entry.group.closed.peek() instanceof Error); @@ -1219,34 +1192,13 @@ export class Producer { hooks.bindGroupTimed(group, resolved); hooks.bindGroupCompression(group, resolved.compress ?? false); } - for (const sink of this.#sinks) { - for (const datagram of sink.datagrams.peek()) { - rescale(resolved, datagram.timestamp); - } - } } catch (err) { this.abort(err instanceof Error ? err : new Error(String(err))); throw err; } this.#state.info.set(resolved); // Propagate to any sink handed out before accept (the on-demand path). - const compressed = new Map(); - for (const sink of this.#sinks) { - if (resolved.compress) { - sink.datagrams.mutate((datagrams) => { - for (let index = 0; index < datagrams.length; index++) { - const { sequence, timestamp, payload } = datagrams[index]; - let slice = compressed.get(datagrams[index]); - if (!slice) { - slice = datagramSlice(payload); - compressed.set(datagrams[index], slice); - } - datagrams[index] = Object.assign(sliceView(timestamp, slice), { sequence }); - } - }); - } - sink.info.set(resolved); - } + for (const sink of this.#sinks) sink.info.set(resolved); this.#updateSubscription(); return this; } @@ -1534,7 +1486,7 @@ export class Producer { this.#advanceLargest(largest); } - /** The track's largest location: its newest group or datagram at the newest frame, or undefined before any. */ + /** The track's largest location: its newest group at the newest frame, or undefined before any. */ largest(): Position | undefined { return this.#state.largest.peek(); } @@ -1692,12 +1644,6 @@ export class Producer { } } - // Refuse a datagram whose timedness disagrees with this track's. - #checkDatagram(timestamp: Timestamp | undefined): void { - const info = this.#state.info.peek(); - if (info) rescale(info, timestamp); - } - // Refuse a write once the track is closed, or at or past its declared end. #writable(sequence: bigint): void { if (this.#state.closed.peek() !== undefined) throw new Error("track is closed"); @@ -1773,87 +1719,13 @@ export class Producer { return info?.unchecked === true || info?.ietf !== undefined; } - // Fan a datagram out to every live subscriber, dropping the oldest once the ring is full. - // Late subscribers do NOT replay old datagrams (best-effort, unlike the group cache). A - // compressed track's rings share only `slice`: each holds its own lazy view of it. - #publishDatagram(datagram: Datagram, slice?: Stored): void { - this.#settle(); - const { sequence, timestamp } = datagram; - this.#advanceObject({ group: sequence, frame: 0 }); - this.#advance(sequence, false); - this.#arrive(sequence); - this.#finishLargest(sequence); - if (this.#datagramSequences.length === MAX_DATAGRAMS) this.#datagramSequences.shift(); - this.#datagramSequences.push(sequence); - for (const sink of this.#sinks) { - sink.received.set(this.#received); - const queued = slice ? Object.assign(sliceView(timestamp, slice), { sequence }) : datagram; - sink.datagrams.mutate((list) => { - if (list.length === MAX_DATAGRAMS) list.shift(); - list.push(queued); - }); - } - } - - /** - * Append a datagram with the next sequence number, returning the assigned sequence. - * - * A datagram is delivered best-effort over a single QUIC datagram, parallel to the track's - * groups but drawing from the same sequence namespace (interleaving with {@link appendGroup} - * never reuses a number). The payload must fit the negotiated transport datagram size minus - * a small header; an oversize payload is dropped at each hop (there is no group fallback), so - * keep datagram payloads small (e.g. a single audio frame). Datagrams are never delivered - * over stream-only transports (the WebSocket fallback). A payload over - * 65535 bytes (the QUIC datagram frame ceiling) throws. An origin publisher uses this; a - * relay preserving upstream numbering uses {@link insertDatagram}. - * - * Pass `undefined` for the timestamp on an untimed track; a timestamp whose presence - * disagrees with the track's {@link Info.timescale} throws {@link TimestampMismatch}. - */ - appendDatagram(timestamp: Timestamp | undefined, payload: Uint8Array): bigint { - const sequence = this.#nextSequence(); - if (payload.byteLength > MAX_DATAGRAM_BYTES) throw new Error("datagram payload too large"); - this.#checkDatagram(timestamp); - - this.#insertDatagram({ sequence, timestamp, payload }); - return sequence; - } - - /** - * Insert a datagram with an explicit sequence number. - * - * Preserves the supplied sequence (later appends continue past it) so a relay can - * forward a datagram without renumbering it. The size and timestamp rules of - * {@link appendDatagram} apply. Most origin publishers want {@link appendDatagram} instead. - */ - insertDatagram(sequence: bigint, timestamp: Timestamp | undefined, payload: Uint8Array) { - this.#insertDatagram({ sequence, timestamp, payload }); - } - - #insertDatagram(datagram: Datagram, inflated?: number): void { - const { sequence, timestamp, payload } = datagram; - checkSequence(sequence); - this.#writable(sequence); - this.#checkDatagram(timestamp); - if (payload.length > MAX_DATAGRAM_BYTES || (inflated ?? 0) > MAX_DATAGRAM_BYTES) - throw new Error("datagram payload too large"); - if ( - inflated !== undefined && - (!Number.isSafeInteger(inflated) || inflated < 0 || !this.#state.info.peek()?.compress) - ) - throw new ProtocolViolation("unexpected inflated datagram size"); - // A bad best-effort datagram is dropped on arrival, before any reader receives it. - const slice = this.#state.info.peek()?.compress ? datagramSlice(payload, inflated) : undefined; - this.#publishDatagram(datagram, slice); - } - /** * Declare the track's exclusive end, possibly ahead of the live edge, mirroring the Rust * `finish_at`. * * `end` is the first position that will never be produced: a track whose last group is * 89 finishes at `{ group: 90, frame: 0 }`, and `{ group: 89, frame: 5 }` also caps group - * 89 at frames 0 to 4. Groups, frames, and datagrams below it are still accepted; anything + * 89 at frames 0 to 4. Groups and frames below it are still accepted; anything * at or past it is refused. Unlike {@link close} it is not terminal: call `close()` once * the remaining groups are written. Readers end once the live edge reaches the end's * group, independently of closure. Throws if the track is closed, already has an end, or @@ -1927,8 +1799,8 @@ export class Producer { } // Whether the pending tail misses a sequence from where the feed starts up to `final`: one - // no live group or datagram delivered, in a gap younger than the grace. Judged by the - // arrivals, not the cache, so an evicted group or a datagram counts as delivered. + // no live group delivered, in a gap younger than the grace. Judged by the + // arrivals, not the cache, so an evicted group counts as delivered. // // The feed starts at the declared start while the demand keeps the floor its SUBSCRIBE asked // for, since an update gets no fresh START: a moved floor stands on its own, and moving @@ -1994,7 +1866,7 @@ export class Producer { this.#endLargest(); } - // Group `sequence` holds every frame it will have: it closed cleanly, or it is a datagram. + // Group `sequence` holds every frame it will have: it closed cleanly, its stream finished. // A group name fixes its content, so no legal write raises the largest past it. #finishLargest(sequence: bigint): void { if (this.#state.largest.peek()?.group === sequence) this.#largestOpen = false; @@ -2176,7 +2048,6 @@ export class Subscriber { // Which cursor owns this subscription. Both cursors draw from one buffer, so the // first group read commits the track to arrival order and {@link ordered} commits // it to sequence order; whichever wins is the only one allowed from here on. - // Datagrams are a separate cursor and never commit. #mode?: "arrival" | "ordered"; // What this cursor reports as drops: sequences given up and not reported yet, // the track's drops last folded in, and every sequence already reported. Drops are @@ -2684,16 +2555,12 @@ export class Subscriber { const asker = this.#askerOf(); this.#clipAsked(); const timeline = this.#state.timeline; - // A datagram the track still holds took its sequence, so no group is missed there, as - // the Rust judge skips its track's datagram buffer; the Rust fetch names it - // `NotFetchable`. const gone = [...this.#state.drops.peek()]; // Reordered streams still owed by the live feed are not cache misses. for (const run of owedRuns(producer, start, end)) { insertRun(gone, run.start, run.end); if (run.folds !== undefined) this.#wakeAt(run.folds); } - for (const sequence of heldDatagrams(producer)) insertRun(gone, sequence, sequence + 1n); // Below where the feed starts, only the history the copy's start rule leaves fillable. const floor = fillFloor(producer, this.#window.start()); for (const run of this.#wants( @@ -2738,14 +2605,12 @@ export class Subscriber { } // Ask a fill for a fetch's group at once, written yet or not, as the Rust `fetch_group` queues - // one; a group the live feed writes meanwhile answers it. `false` with no handler to ask, or - // for a sequence a buffered datagram took; `"pending"` until the track is accepted. + // one; a group the live feed writes meanwhile answers it. `false` with no handler to ask; `"pending"` until the track is accepted. #askFetch(sequence: bigint): boolean | "pending" { const producer = this.#state.producer; if (!producer) return false; if (this.#state.info.peek() === undefined && this.#state.closed.peek() === undefined) return "pending"; if (cachedHolds(producer, sequence)) return true; - if (heldDatagrams(producer).includes(sequence)) return false; // Joins a fill already bringing it, handler or not; asks a new one only while one is open. const asked = fillsOf(producer).ask(sequence, sequence + 1n, this.#askerOf()); if (asked.length === 0) return false; @@ -2923,23 +2788,8 @@ export class Subscriber { hooks.unreport = (subscriber, start, end) => { subscriber.#reported = removeRun(subscriber.#reported, start, end); }; - hooks.tryRecvDatagram = (subscriber) => subscriber.#tryDatagram(); hooks.deliveredRuns = (subscriber) => subscriber.#delivered.runs; hooks.handedOut = (subscriber, sequence) => subscriber.#delivered.has(sequence); - hooks.datagramChanged = (subscriber, fn) => { - if (subscriber.#resume) return subscriber.#groupChanged(fn); - const state = subscriber.#state; - const dispose = [ - state.datagrams.changed(fn), - state.closed.changed(fn), - state.end.changed(fn), - state.received.changed(fn), - state.owed.changed(fn), - ]; - return () => { - for (const close of dispose) close(); - }; - }; hooks.hasUnreadGroup = (subscriber, start, end, delivered) => subscriber.#state.groups .peek() @@ -2950,24 +2800,9 @@ export class Subscriber { !delivered.has(group.sequence) && !(group.closed.peek() instanceof Error), ); - hooks.spent = (subscriber, start, end, delivered) => { - const state = subscriber.#state; - // A datagram outside the cursor is gone for good, as a read would discard it. - const cursor = subscriber.#cursor.peek(); - const readable = state.datagrams - .peek() - .some( - (datagram) => - datagram.sequence >= cursor.start && - (cursor.end === undefined || datagram.sequence < cursor.end) && - subscriber.#window.admitsDatagram(datagram.sequence, datagram.timestamp), - ); - return ( - (state.closed.peek() !== undefined || subscriber.#complete()) && - !readable && - !hooks.hasUnreadGroup(subscriber, start, end, delivered) - ); - }; + hooks.spent = (subscriber, start, end, delivered) => + (subscriber.#state.closed.peek() !== undefined || subscriber.#complete()) && + !hooks.hasUnreadGroup(subscriber, start, end, delivered); hooks.staleAt = (subscriber, sequence) => { const held: Superseded = {}; return { @@ -3062,8 +2897,6 @@ export class Subscriber { readString: (subscriber) => subscriber.#readString(), readJson: (subscriber) => subscriber.#readJson(), readBool: (subscriber) => subscriber.#readBool(), - // Unordered either way, so both handles reach the same cursor. - recvDatagram: (subscriber) => subscriber.#recvDatagram(), }; } @@ -3119,7 +2952,7 @@ export class Subscriber { * The largest location within {@link withRange}: the current one at first, then each newer * one (skipping any in between), and `undefined` once the track ends. * - * The largest is the track's newest group or datagram at the newest frame (0 while empty), as + * The largest is the track's newest group at the newest frame (0 while empty), as * the publisher reported it and its feed delivered since, so it can name a frame before its * group arrives. A largest past the cap or below the floor reports nothing: the cache does not * stand in for it, so a cap moved below the largest stays pending. Moving the bounds wakes a @@ -3485,12 +3318,7 @@ export class Subscriber { this.#state.owed.changed(fn), ]; if (this.#state.waitingOrdered) - dispose.push( - this.#state.timelineChanged.changed(fn), - this.#wall.revision.changed(fn), - // A datagram shares the sequence namespace, so it can fill the owed hole. - this.#state.datagrams.changed(fn), - ); + dispose.push(this.#state.timelineChanged.changed(fn), this.#wall.revision.changed(fn)); // A bounded window ends once the feed stops owing its last hole, which only time decides, // and a fill's hole stops holding a read once the budget convicts it. else if (this.#window.end() !== undefined || this.#fillWait) dispose.push(this.#wall.revision.changed(fn)); @@ -3517,77 +3345,6 @@ export class Subscriber { ); } - /** - * Receive the next datagram in arrival order. - * - * Datagrams are a separate best-effort channel from groups (see - * {@link Producer.appendDatagram}); they share only the sequence namespace. Those outside - * the group range are skipped. A consumer that falls too far behind silently loses the - * oldest datagrams. Read this alongside - * {@link recvGroup} (e.g. in a separate loop) to receive both channels concurrently. - * The two cursors are independent: a datagram never moves the group cursor. - */ - async recvDatagram(): Promise { - this.#live(); - return this.#recvDatagram(); - } - - // The datagram cursor, reachable from either handle: unordered by construction, so the - // choice of group order says nothing about it. - async #recvDatagram(): Promise { - for (;;) { - const recv = this.#tryDatagram(); - switch (recv.kind) { - case "datagram": - return recv.datagram; - case "done": - return this.#ended(); - case "error": - throw recv.error; - } - if (this.#resume) await this.#wait(); - else - await Signal.race( - this.#state.datagrams, - // Either may refuse the ranges, with no datagram arriving to say so. - this.#state.info, - this.#state.dropsAnchors, - this.#state.closed, - this.#state.end, - this.#state.received, - this.#state.owed, - ); - } - } - - // The synchronous half of #recvDatagram. - #tryDatagram(): RecvDatagram { - const closed = this.#state.closed.peek(); - if (this.#resume && closed instanceof Error) return { kind: "error", error: closed }; - // The group range bounds datagrams too, judged when each is read. One outside it is - // gone for good: nothing holds it for a later raise, unlike a group past the cap. - const refused = this.#settle(); - if (refused) return { kind: "error", error: refused }; - const { start, end } = this.#cursor.peek(); - const inRange = (datagram: Datagram) => - datagram.sequence >= start && - (end === undefined || datagram.sequence < end) && - this.#window.admitsDatagram(datagram.sequence, datagram.timestamp); - if (this.#resume) { - for (;;) { - const recv = this.#resume.tryDatagram(this.#delivered); - if (recv.kind !== "datagram" || inRange(recv.datagram)) return recv; - } - } - const datagrams = this.#state.datagrams.peek(); - for (let datagram = datagrams.shift(); datagram; datagram = datagrams.shift()) { - if (inRange(datagram)) return { kind: "datagram", datagram }; - } - if (closed instanceof Error) return { kind: "error", error: closed }; - if (closed !== undefined || this.#complete()) return { kind: "done" }; - return { kind: "idle" }; - } - // The sequence cursor behind {@link Ordered}, which owns the only public door to it. async #nextGroup(): Promise { for (;;) { @@ -3799,11 +3556,9 @@ export class Subscriber { * * Both cursors draw from the same buffer, so a track is read one way or the other: this * hands the subscription to the returned {@link Ordered} and leaves this handle inert. - * {@link recvGroup} and {@link recvDatagram} throw afterwards. Throws once a + * {@link recvGroup} throw afterwards. Throws once a * {@link recvGroup} call has already committed the track to arrival order. * - * Datagrams come along: they are a separate cursor either way, so the choice of group - * order says nothing about them, and reading them commits nothing. */ ordered(): Ordered { if (this.#mode === "ordered") throw new Error("track is already read in sequence order"); @@ -3964,17 +3719,4 @@ export class Ordered { readBool(): Promise { return ordered_.readBool(this.#subscriber); } - - /** - * Receive the next datagram in arrival order. - * - * Datagrams are a separate best-effort channel from groups (see - * {@link Producer.appendDatagram}); they share only the sequence namespace, and - * neither cursor moves the other. Unordered by construction, so this behaves - * identically on either handle; it is here so a track carrying both channels needs - * one subscription rather than two. - */ - recvDatagram(): Promise { - return ordered_.recvDatagram(this.#subscriber); - } } diff --git a/js/net/src/untimed.test.ts b/js/net/src/untimed.test.ts index 2ec52a88be1..2ce19df361d 100644 --- a/js/net/src/untimed.test.ts +++ b/js/net/src/untimed.test.ts @@ -181,37 +181,3 @@ for (const wire of WIRES) { } }); } - -for (const wire of WIRES.filter((wire) => wire.name === "lite-06" || wire.name === "lite-07-wip")) { - test(`${wire.name}: an untimed datagram arrives ${wire.sends ? "with its send time" : "untimed"}`, async () => { - const pair = createMockTransportPair(wire.protocol); - const origin = new OriginProducer(); - const [client, server] = await Promise.all([ - connect({ url, transport: pair.client }), - accept({ transport: pair.server, url, publish: origin.consume() }), - ]); - - const broadcastPublication = origin.createBroadcast(Path.from("test")); - const broadcast = broadcastPublication.broadcast; - broadcastPublication.announce(); - const producer = broadcast.createTrack("video", {}); - const subscriber = wireOf(client).consume(Path.from("test")).track("video").subscribe(); - - // Datagrams aren't cached, so a group read back first proves the subscription is live. - const group = producer.appendGroup(); - group.writeFrame({ payload: new Uint8Array([0]) }); - group.finish(); - expect(await subscriber.recvGroup()).toBeDefined(); - - producer.appendDatagram(undefined, new Uint8Array([7])); - const datagram = await subscriber.recvDatagram(); - - expect(datagram?.payload).toEqual(new Uint8Array([7])); - expect(datagram?.timestamp !== undefined).toBe(wire.sends); - - broadcast.close(); - subscriber.close(); - client.abort(); - server.abort(); - }); -} diff --git a/js/net/src/window.test.ts b/js/net/src/window.test.ts index 245c41d5937..80fd7ce3011 100644 --- a/js/net/src/window.test.ts +++ b/js/net/src/window.test.ts @@ -313,40 +313,6 @@ test("a time end walks each group once behind a missing sequence", () => { expect(groups.asked).toBeLessThan(8 * count); }); -test("a datagram is judged by its own timestamp, one range at a time", () => { - const window = new Window([{ start: { absolute: 0n }, end: time(3500) }], undefined, new Groups()); - expect(window.admitsDatagram(9n, Timestamp.fromMillis(3500))).toBe(true); - expect(window.admitsDatagram(9n, Timestamp.fromMillis(3600))).toBe(false); - - // A start still waiting judges a datagram by its time alone. - expect(window.admitsDatagram(0n, Timestamp.fromMillis(0))).toBe(true); - const waiting = new Window([{ start: time(3000), end: time(3500) }], undefined, new Groups()); - expect(waiting.admitsDatagram(0n, Timestamp.fromMillis(2900))).toBe(false); - expect(waiting.admitsDatagram(1n, Timestamp.fromMillis(3000))).toBe(true); - expect(waiting.admitsDatagram(3n, Timestamp.fromMillis(3600))).toBe(false); - - // Group 1 arrived only as a datagram: the end settles before 2 by sequence, but 1 presents past it. - const groups = new Groups([ - [0n, started(0)], - [2n, started(10_000)], - ]); - groups.gones.push({ start: 1n, end: 2n }); - const settled = new Window([{ start: time(0), end: time(5000) }], undefined, groups); - expect(settled.resolved()).toEqual([{ start: 0n, end: 2n, floored: true }]); - expect(settled.admitsDatagram(1n, Timestamp.fromMillis(4000))).toBe(true); - expect(settled.admitsDatagram(1n, Timestamp.fromMillis(7000))).toBe(false); - - // The sequence and the time must pass the same range. - const both = new Window( - [{ start: { absolute: 6n } }, { start: time(5500) }], - 5n, - new Groups([[5n, started(5000)]]), - ); - expect(both.admitsDatagram(5n, Timestamp.fromMillis(5000))).toBe(false); - expect(both.admitsDatagram(5n, Timestamp.fromMillis(5500))).toBe(true); - expect(both.admitsDatagram(6n, Timestamp.fromMillis(5000))).toBe(true); -}); - test("a time anchor needs moq-lite-07, is resolved by the copy, and clamps to its group", () => { expect(isWire({ start: time(1) })).toBe(false); expect(isWire({ start: { absolute: 1n }, end: time(1) })).toBe(false); diff --git a/js/net/src/window.ts b/js/net/src/window.ts index 7e435bf0898..dad35c81daf 100644 --- a/js/net/src/window.ts +++ b/js/net/src/window.ts @@ -462,8 +462,7 @@ class TimeEnd { else this.limit = this.after; // Settle once every sequence up to the first group starting past `pts` can no longer - // start inside it. A sequence that arrived only as a datagram never indexes where its - // group starts, so it counts as gone here. + // start inside it. const after = this.after; if (after === undefined) return undefined; // Everything at or below `known` is in range, and `after` lies above it. @@ -599,38 +598,6 @@ export class Window { return run !== undefined && sequence >= run.start; } - /** - * Whether a datagram is in range, judged on its own and never through a time end's settling: - * its `timestamp` against each range's times, and its `sequence` against the range's absolute - * groups and the relative ones resolved so far. - */ - admitsDatagram(sequence: bigint, timestamp: Timestamp | undefined): boolean { - if (timestamp === undefined) { - return this.#pending || this.#spans.some((span) => sequence >= span.start && sequence < limitOf(span)); - } - return this.ranges.some((range, index) => { - if (isEmpty(range)) return false; - // A relative anchor still waiting on a latest group bounds nothing yet. - const span = this.#spans.find((span) => span.index === index); - const { start, end } = range; - const started = - start?.time !== undefined - ? !later(start.time, timestamp) - : start?.absolute !== undefined - ? sequence >= start.absolute - : start === undefined || !span || sequence >= span.start; - const ended = - end === undefined - ? true - : end.time !== undefined - ? !later(timestamp, end.time) - : end.absolute !== undefined - ? sequence < end.absolute - : !span || span.end === "open" || !("before" in span.end) || sequence < span.end.before; - return started && ended; - }); - } - /** The lowest group in range at or above `from`, undefined when none is. `from` itself while a relative anchor waits. */ next(from: bigint): bigint | undefined { if (this.#pending) return from; diff --git a/js/net/src/wire.ts b/js/net/src/wire.ts index f56f2c2adca..d8a7e59e99c 100644 --- a/js/net/src/wire.ts +++ b/js/net/src/wire.ts @@ -9,6 +9,7 @@ */ import type { Dispose, GetPromise, Getter } from "@moq/signals"; import type * as broadcast from "./broadcast.ts"; +import type * as datagram from "./datagram.ts"; import type { Consumer as GroupConsumer } from "./group.ts"; import type { Route } from "./hop.ts"; import type * as Instance from "./instance.ts"; @@ -19,6 +20,10 @@ import type * as track from "./track.ts"; /** The protocol-facing operations behind a broadcast handle. */ export interface Broadcast { + datagrams(name: string): datagram.Consumer; + liveDatagrams(name: string): datagram.Consumer | undefined; + createDatagrams(name: string, info: Partial): datagram.Producer; + requestedDatagrams(): Promise; subscribe(name: string, options?: track.Subscription): track.Subscriber; /** The track's properties; `hold` keeps the track wanted until it aborts, the way a held TRACK stream does, and aborting it before the answer abandons the lookup. */ resolveTrackInfo(name: string, hold?: AbortSignal): Promise; diff --git a/quest/m1/datagram-tracks/README.md b/quest/m1/datagram-tracks/README.md index 743e3b8b81f..e6da124a255 100644 --- a/quest/m1/datagram-tracks/README.md +++ b/quest/m1/datagram-tracks/README.md @@ -120,7 +120,6 @@ datagram track on lite-07, direct and through a relay. ## Required -- [JS datagram tracks](/quest/m1/datagram-tracks/js.md) - `@moq/net` datagram types and the lite-07 datagram request - [moq-transport datagram tracks](/quest/m1/datagram-tracks/ietf.md) - Rust and JS map a datagram track to SUBSCRIBE and OBJECT_DATAGRAM with object ID 0, keeping the requested kind - [A datagram sent right behind its answer reaches the subscriber](/quest/m1/datagram-tracks/answer-race.md) - receivers decode answers before routing datagrams, and count the ones they still can't route diff --git a/quest/m1/datagram-tracks/answer-race.md b/quest/m1/datagram-tracks/answer-race.md index 0fac692d68e..d413c2baf1b 100644 --- a/quest/m1/datagram-tracks/answer-race.md +++ b/quest/m1/datagram-tracks/answer-race.md @@ -24,9 +24,7 @@ Decided 2026-11-05 (maintainer, /quest-import of answer is acknowledged (about one RTT of datagrams lost per subscription instead), or accepting the loss with only a log. - ✅ **One quest for Rust and JS.** The JS part builds on the paths - [JS datagram tracks](/quest/m1/datagram-tracks/js.md) introduces - ([moq-dev/moq#7846](https://github.com/moq-dev/moq/pull/7846)), so it - waits for that quest. + [JS datagram tracks](https://github.com/moq-dev/moq/pull/7846) introduces. - ✅ m1, in the Datagram tracks epic. Verified on `main` at 246916845a, from the issue's report: @@ -54,10 +52,6 @@ arrives, and an unroutable one bumps the counter. The integration tests in `rs/moq-net/tests/integration/datagram.rs` all wait for `consumer.info()` before appending, so add a case that doesn't. -## Required - -- [JS datagram tracks](/quest/m1/datagram-tracks/js.md) - the JS paths this fix builds on - ## Closes - [#7882](https://github.com/moq-dev/moq/issues/7882) - the datagram served at subscribe time is dropped by the receiver when TRACK and SUBSCRIBE are pipelined diff --git a/quest/m1/datagram-tracks/ietf.md b/quest/m1/datagram-tracks/ietf.md index 37b3a70b460..1b797ce6182 100644 --- a/quest/m1/datagram-tracks/ietf.md +++ b/quest/m1/datagram-tracks/ietf.md @@ -51,7 +51,3 @@ when it starts accepting datagrams, and add a JS test that an undefined status closes the session. Public API: none beyond the model. Wire: no format change. - -## Required - -- [JS datagram tracks](/quest/m1/datagram-tracks/js.md) - the JS datagram types this maps diff --git a/quest/m1/datagram-tracks/js.md b/quest/m1/datagram-tracks/js.md deleted file mode 100644 index 851c6215870..00000000000 --- a/quest/m1/datagram-tracks/js.md +++ /dev/null @@ -1,30 +0,0 @@ -# [M] JS datagram tracks - -## Goal - -`@moq/net` has the same separate datagram types as Rust, and speaks the -lite-07 datagram request and datagram body against the Rust implementation. -Stream tracks lose `appendDatagram`, and datagram sequences no longer share -the group counter. Older lite versions keep today's group datagrams. - -## Plan - -Follow the [epic](/quest/m1/datagram-tracks/README.md)'s decisions and the -lite-07 wire that `drafts/draft-lcurley-moq-lite.md` specifies (Datagrams -Stream 0x9, DATAGRAMS, DATAGRAMS_UPDATE, DATAGRAMS_INFO, and the Track -Datagram body), which Rust speaks since -[moq-dev/moq#7752](https://github.com/moq-dev/moq/pull/7752). - -- Remove from `js/net/src/track.ts` and the lite subscriber and publisher: - - `appendDatagram` and the shared sequence counter (`track.test.ts` - "appendDatagram shares the group sequence counter"); - - the join and refresh rules for datagrams ("a join wants past a datagram - after the newest group"); - - the datagram cases in ranges. -- Map lite-01..06 group datagrams the same way Rust does. -- Update any first-party consumer of JS datagrams. -- Drop the sentence saying the JavaScript library still sends datagrams - within a stream track from the Datagrams section of `doc/concept/moq-lite.md`. - -Public API: breaking for `@moq/net` datagrams. Wire: lite-07-wip, matching -Rust. diff --git a/sh/gh/nightly.sh b/sh/gh/nightly.sh index 28699f54bdc..b4bd6fc31d4 100755 --- a/sh/gh/nightly.sh +++ b/sh/gh/nightly.sh @@ -160,7 +160,7 @@ case "$1" in done ;; js) - for benchmark in broadcasts requests fills fetch forward reader frames drift ledger bandwidth replay varint lite-varint track resume priority subscribe-drop refresh observed compression; do + for benchmark in broadcasts requests fills fetch forward reader frames drift ledger bandwidth replay varint lite-varint track resume priority subscribe-drop refresh observed compression datagrams; do javascript "$benchmark" just js bench "$benchmark" done javascript catalog just js catalog-bench