Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions doc/lib/js/flate.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,14 @@ time written as the frame timestamp. Nothing fills in now: a timed track, such
as `createTrack(name, { timescale: Time.Timescale.MILLI })`, needs `at` on
every write, and an untimed track (no `timescale`) takes none.

A Stream rides one group, so the whole log shares `@moq/net`'s group budget:
32 MiB of payload and 8192 payloads. An `append` that might not fit throws
`GroupTooLarge` before it is encoded and leaves the log intact, compressed or
not. With `"deflate"` the check counts the raw payload plus DEFLATE's
worst-case overhead. Once the budget is spent every `append` throws; start a
new track to keep going. Any other failed append aborts the track, so readers
see the error rather than a log that looks complete.

The codec underneath is exported as `Encoder`/`Decoder`. Create one pair per
group and feed frames in order.

Expand Down
7 changes: 7 additions & 0 deletions doc/lib/rs/moq-flate.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,13 @@ A payload is stamped when written, unless it carries its capture time:
`moq_net::Timed::from(bytes).at(captured)`. Writes return the encoded frame
size.

A stream rides one group, so the whole log shares moq-net's group budget:
32 MiB of payload and 8192 payloads. An `append` that might not fit returns
`moq_net::Error::GroupTooLarge` before it is encoded and leaves the log intact,
compressed or not. Once the budget is spent every `append` is refused; start a
new track to keep going. Any other failed append aborts the track, so readers
see the error rather than a log that looks complete.

The group-scoped codec underneath is exported as `Encoder`/`Decoder`, which
[`moq-json`](/lib/rs/moq-json) reuses for its merge-patch deltas. Create one
pair per group and feed frames in order.
Expand Down
24 changes: 24 additions & 0 deletions js/flate/src/codec.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,30 @@ test("a frame larger than pako's chunk size round-trips", () => {
expect(new Decoder().frame(slice)).toEqual(payload);
});

// The bound holds for incompressible input, the worst case, at sizes straddling the 16 KiB block
// boundary, and on a window already primed by earlier frames. Twin of the Rust test, since pako is a
// separate implementation from the zlib the Rust side checks.
test("the bound covers incompressible frames", () => {
// xorshift: incompressible enough to force stored blocks, and deterministic. Each frame is fresh
// noise; a repeat would match the window instead.
let state = 0x9e3779b9;
const noise = (len: number) => {
const out = new Uint8Array(len);
for (let i = 0; i < len; i++) {
state ^= state << 13;
state ^= state >>> 17;
state ^= state << 5;
out[i] = state & 0xff;
}
return out;
};

const encoder = new Encoder();
for (const len of [1, 2, 100, 16_383, 16_384, 16_385, 65_535, 65_536, 1 << 20, 3 << 20]) {
expect(encoder.frame(noise(len)).byteLength).toBeLessThanOrEqual(Encoder.bound(len));
}
});

test("cross-frame context shrinks a repeated frame", () => {
// A later frame identical to an earlier one compresses far smaller once the window holds it.
const encoder = new Encoder();
Expand Down
13 changes: 13 additions & 0 deletions js/flate/src/codec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,19 @@ export class Encoder {
};
}

/**
* The most bytes {@link frame} can return for a payload of `len` bytes, at any level and whatever
* the window already holds.
*
* zlib's `deflateBound` for its default window and memory level, which both pako and the Rust
* `moq-flate` use: incompressible input falls back to stored blocks, 5 bytes per 16 KiB. The
* constant covers the block headers and the flush, whose fixed 4-byte marker is stripped anyway.
*/
static bound(len: number): number {
// Division rather than shifts, which wrap past 2^31.
return len + Math.floor(len / 4096) + Math.floor(len / 16384) + Math.floor(len / 33554432) + 13;
}

/**
* Compress the next frame's `payload`, returning its slice of the stream: the DEFLATE bytes minus
* the fixed sync-flush marker. Empty in yields empty out. Slices must be produced in frame order.
Expand Down
4 changes: 2 additions & 2 deletions js/flate/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@
* - {@link Snapshot}: **lossy**. One value updated over time; a consumer only gets the most recent
* one. Older values are superseded and dropped.
* - {@link Stream}: **lossless**. An ordered append-log of self-contained payloads, delivered in
* order with nothing superseded. Bounded by the group cache: see {@link Stream} for what that
* costs a consumer that falls behind.
* order with nothing superseded. Bounded by the group budget: see {@link Stream} for what happens
* once it is spent.
*
* Pick {@link Snapshot} when consumers care about "what is the value now" (a poster image, a
* serialized state blob) and {@link Stream} when they care about every payload (an event log, a
Expand Down
14 changes: 8 additions & 6 deletions js/flate/src/stream/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,18 +5,20 @@
* sequence of samples). Nothing is ever superseded: a consumer yields each payload in the order it
* was appended. For a latest-value document, use the `Snapshot` module instead.
*
* Retention is bounded, which is the limit of "lossless" here. The group's cache is finite, so a
* log longer than it holds evicts its earliest frames, and a consumer that falls behind or
* subscribes late then fails its read with `Lagged` rather than silently resuming partway through:
* a partial log presented as a whole one is what this mode exists to prevent. Keep a log inside
* what the group retains, and split anything unbounded across successive tracks.
*
* On the wire the log rides a **single group** that is never rolled, one payload per frame. A
* payload that cannot be written ends the track rather than opening a second group: a log missing a
* record is not lossless, and a gap dressed up as a complete log is worse than a visible failure.
* With {@link Config.compression} `"deflate"`, that one group is one DEFLATE window, so each payload
* compresses against the earlier ones and a run of similar payloads shrinks sharply.
*
* That one group bounds the log at `@moq/net`'s group budget: 32 MiB of payload and 8192 frames. A
* consumer always starts at frame 0, so the cap is never met by dropping a prefix some readers
* missed. Instead an append that might not fit throws `GroupTooLarge` before it is encoded, and the
* log stays intact and writable. With compression the check counts the payload's raw size plus
* DEFLATE's worst-case overhead (`Encoder.bound`), since the compressed size is only known once the
* window has moved. The budget covers the whole log, so once it is spent every append throws and a
* publisher with more to say opens a new track.
*
* @module
*/

Expand Down
45 changes: 29 additions & 16 deletions js/flate/src/stream/producer.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import type * as Moq from "@moq/net";
import type { Timed } from "@moq/net";
import { DEFAULT_MAX_FRAME_SIZE, Encoder as Flate } from "../codec.ts";
import { Group, Error as NetError, type Timed } from "@moq/net";
import { Encoder as Flate } from "../codec.ts";

import { type Compression, isDeflate } from "../compression.ts";

Expand All @@ -25,6 +25,10 @@ export class Producer {
#flate?: Flate;
// The single group carrying the whole log, opened on the first append and never rolled.
#group?: Moq.Group.Producer;
// Frames and payload bytes written to the group, checked against the group budget before each
// payload is encoded.
#frames = 0;
#bytes = 0;

/** Wrap a track to publish a payload log into it. */
constructor(config: Producer.Config) {
Expand All @@ -36,26 +40,32 @@ export class Producer {
/**
* Append one payload to the log.
*
* A payload that cannot be written ends the track: a log missing a record is not the lossless
* log this mode promises, so the failure is surfaced rather than papered over with a second
* group. The group is aborted rather than closed cleanly, so a consumer sees the failure
* A payload that might not fit in what is left of the group's budget throws `GroupTooLarge`
* before anything is written, leaving the log intact. The budget covers the whole log, so once
* it is spent every append throws; a publisher with more to say opens a new track.
*
* Any other payload that cannot be written ends the track: a log missing a record is not the
* lossless log this mode promises, so the failure is surfaced rather than papered over with a
* second group. The group is aborted rather than closed cleanly, so a consumer sees the failure
* instead of a log that merely looks complete. Every later append fails on the closed track.
*
* `at` is when the payload was captured, written as its frame timestamp. Omit it only on an
* untimed track: it is never filled in.
*/
append({ value: payload, at }: Timed<Uint8Array>): void {
// A payload no consumer could decode is as terminal as one the track rejects: consumers all
// decode with `@moq/flate`'s default cap, so this would publish a record none of them could
// read. Ends the track like any other lost record, and aborts the group the same way the
// write path below does: an earlier append may already have opened one, and leaving the two
// paths to tear down differently only invites a reader to wonder which is authoritative.
if (this.#flate && payload.byteLength > DEFAULT_MAX_FRAME_SIZE) {
const err = new Error(`payload larger than the decoder's ${DEFAULT_MAX_FRAME_SIZE} byte limit`);
this.#group?.close(err);
this.#group = undefined;
this.#track.close(err);
throw err;
// A closed track refuses every append, so say so before the budget: `GroupTooLarge` promises
// the log is still writable.
const closed = this.#track.closed.peek();
if (closed !== undefined) throw closed ?? new Error("track is closed");

// Check before compressing: encoding advances the window, so a payload refused afterwards
// would leave the encoder ahead of every reader. The worst case is checked rather than the
// actual size for the same reason. Also checked before the group is opened, so a refused
// first payload publishes nothing. The budget is below the decoder's cap, so any payload that
// fits is one every consumer can inflate.
const bound = this.#flate ? Flate.bound(payload.byteLength) : payload.byteLength;
if (this.#frames >= Group.MAX_GROUP_FRAMES || this.#bytes + bound > Group.MAX_GROUP_CACHE_BYTES) {
throw new NetError.GroupTooLarge();
}

// Open the group before compressing: a failure here must not leave the window ahead of a
Expand Down Expand Up @@ -83,6 +93,9 @@ export class Producer {
this.#track.close(abort);
throw err;
}

this.#frames++;
this.#bytes += encoded.byteLength;
}

/** Finish the track, closing the group. */
Expand Down
101 changes: 93 additions & 8 deletions js/flate/src/stream/stream.test.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import { expect, spyOn, test } from "bun:test";
import { Time, Track } from "@moq/net";
import { Group, Error as NetError, Time, Track } from "@moq/net";
import { DEFAULT_MAX_FRAME_SIZE } from "../codec.ts";
import { Consumer, Producer, Rolled } from "./index.ts";

Expand Down Expand Up @@ -129,24 +129,109 @@ test("a second group is reported while the first is still open", async () => {
await expect(consumer.next()).rejects.toThrow(Rolled);
});

test("an undecodable payload ends the log for a reader already inside the group", async () => {
// The decoder-limit check ends the track like any other lost record, and by then an earlier
// append may already have opened the group. A reader that pulled that group has to see the
// failure rather than park on a group nothing will ever finish.
test("a failed write ends the log for a reader already inside the group", async () => {
// A reader that pulled the group holds its own handle, so it has to see the failure rather than
// park on a group nothing will ever finish.
const track = new Track.Producer("test");
const producer = new Producer({ track, compression: "deflate" });
producer.append({ value: payloads(1)[0] });

const consumer = new Consumer({ track: track.subscribe(), compression: "deflate" });
expect(await consumer.next()).toBeDefined();

const oversized = new Uint8Array(DEFAULT_MAX_FRAME_SIZE + 1);
expect(() => producer.append({ value: oversized })).toThrow("limit");
// The budget check refuses anything the group would, so stub the write to stand in for any
// rejection past it.
const failure = new Error("write rejected");
const write = spyOn(Group.Producer.prototype, "writeFrame").mockImplementation(() => {
throw failure;
});
try {
expect(() => producer.append({ value: payloads(2)[1] })).toThrow(failure);
} finally {
write.mockRestore();
}

// Surfaces the terminal error rather than hanging on the still-open group.
await expect(consumer.next()).rejects.toThrow("limit");
await expect(consumer.next()).rejects.toThrow(failure);
expect(() => producer.append({ value: payloads(3)[2] })).toThrow();
});

// An ended log reports why it ended, even for a payload past the budget: `GroupTooLarge` would tell
// the caller the log is still writable.
test("an ended log refuses with its own error", () => {
const track = new Track.Producer("test");
const producer = new Producer({ track, compression: "deflate" });

const failure = new Error("write rejected");
const write = spyOn(Group.Producer.prototype, "writeFrame").mockImplementation(() => {
throw failure;
});
try {
expect(() => producer.append({ value: payloads(1)[0] })).toThrow(failure);
} finally {
write.mockRestore();
}

expect(() => producer.append({ value: new Uint8Array(Group.MAX_GROUP_CACHE_BYTES + 1) })).toThrow(failure);
});

// A payload past the group budget is refused before anything is written, so the log carries on: the
// next payload lands and decodes, which with compression proves the window never moved. One past the
// decoder's cap is refused the same way rather than ending the track.
for (const compression of [false, true]) {
test(`an oversized payload is refused and the log continues (compression=${compression})`, async () => {
const track = new Track.Producer("test");
const producer = new Producer({ track, compression: compression ? "deflate" : "none" });
producer.append({ value: payloads(1)[0] });

const huge = new Uint8Array(DEFAULT_MAX_FRAME_SIZE + 1);
expect(() => producer.append({ value: huge.subarray(0, Group.MAX_GROUP_CACHE_BYTES) })).toThrow(
NetError.GroupTooLarge,
);
expect(() => producer.append({ value: huge })).toThrow(NetError.GroupTooLarge);

producer.append({ value: payloads(2)[1] });
producer.finish();
expect(await drain(track.subscribe(), compression)).toEqual(payloads(2));
});
}

// The budget check stands in for the decoder's cap: a payload that fits is one every consumer can
// inflate.
test("the group budget is within the decoder's cap", () => {
expect(Group.MAX_GROUP_CACHE_BYTES).toBeLessThanOrEqual(DEFAULT_MAX_FRAME_SIZE);
});

test("a refused first payload publishes nothing", async () => {
const track = new Track.Producer("test");
const producer = new Producer({ track, compression: "deflate" });
expect(() => producer.append({ value: new Uint8Array(DEFAULT_MAX_FRAME_SIZE + 1) })).toThrow(
NetError.GroupTooLarge,
);
producer.finish();

expect(await track.subscribe().ordered().nextGroup()).toBeUndefined();
});

// The budget covers the whole log, so once its frames are spent every append is refused, and the log
// written so far still finishes cleanly and reads back whole.
for (const compression of [false, true]) {
test(`a spent budget refuses every append (compression=${compression})`, async () => {
const track = new Track.Producer("test");
const producer = new Producer({ track, compression: compression ? "deflate" : "none" });
const frame = (n: number) => new Uint8Array(new Uint32Array([n]).buffer);
for (let n = 0; n < Group.MAX_GROUP_FRAMES; n++) producer.append({ value: frame(n) });

expect(() => producer.append({ value: frame(-1) })).toThrow(NetError.GroupTooLarge);
expect(() => producer.append({ value: frame(-2) })).toThrow(NetError.GroupTooLarge);
producer.finish();

const out = await drain(track.subscribe(), compression);
expect(out.length).toBe(Group.MAX_GROUP_FRAMES);
expect(out.at(-1)).toEqual(frame(Group.MAX_GROUP_FRAMES - 1));
});
}

// Counts the reactions `run` attaches to promises still pending once it returns. A promise holds each
// reaction until it settles, so one left per iteration on a promise that outlives the loop is a leak.
// Recorded by hand: Bun's `mock.contexts` misses the engine's own calls from `Promise.race`.
Expand Down
26 changes: 1 addition & 25 deletions js/json/src/stream/codec.test.ts
Original file line number Diff line number Diff line change
@@ -1,10 +1,9 @@
import { expect, spyOn, test } from "bun:test";
import { Encoder as Flate } from "@moq/flate";
import { Group, Track } from "@moq/net";
import * as z from "@zod/mini";
import { Desync } from "../error.ts";
import { Decoder } from "./decoder.ts";
import { deflateBound, Encoder } from "./encoder.ts";
import { Encoder } from "./encoder.ts";
import { Producer } from "./producer.ts";

type Rec = { n: number };
Expand Down Expand Up @@ -209,26 +208,3 @@ test("a lost compressed stream frame has a typed error", () => {
encoder.encode({ n: 1 });
expect(() => encoder.encode({ n: 2 })).toThrow(Desync);
});

// The bound holds for incompressible input, the worst case, at sizes straddling the 16 KiB block
// boundary, and on a window already primed by earlier frames. Twin of the Rust test, since pako is a
// separate implementation from the zlib the Rust side checks.
test("deflate bound covers incompressible frames", () => {
// xorshift: incompressible enough to force stored blocks, and deterministic.
let state = 0x9e3779b9;
const noise = (len: number) => {
const out = new Uint8Array(len);
for (let i = 0; i < len; i++) {
state ^= state << 13;
state ^= state >>> 17;
state ^= state << 5;
out[i] = state & 0xff;
}
return out;
};

const flate = new Flate();
for (const len of [1, 2, 100, 16_383, 16_384, 16_385, 65_535, 65_536, 1 << 20, 3 << 20]) {
expect(flate.frame(noise(len)).byteLength).toBeLessThanOrEqual(deflateBound(len));
}
});
10 changes: 1 addition & 9 deletions js/json/src/stream/encoder.ts
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ export class Encoder<T> {
// actual size for the same reason. The budget is also below `@moq/flate`'s per-frame decode
// cap, so any record that fits is one every consumer can inflate.
const budget = this.#budget;
const bound = this.#compress ? deflateBound(bytes.byteLength) : bytes.byteLength;
const bound = this.#compress ? Flate.bound(bytes.byteLength) : bytes.byteLength;
if (budget.frames >= Group.MAX_GROUP_FRAMES || budget.bytes + bound > Group.MAX_GROUP_CACHE_BYTES) {
throw new NetError.GroupTooLarge();
}
Expand Down Expand Up @@ -152,11 +152,3 @@ export class Encoder<T> {
};
}
}

// The largest a sync-flushed DEFLATE frame of `len` raw bytes can grow to: zlib's `deflateBound` for
// its default window and memory level, which both pako and moq-flate use. Incompressible input falls
// back to stored blocks, 5 bytes per 16 KiB; the constant covers the block headers and the flush,
// whose fixed 4-byte marker is stripped anyway. Division rather than shifts, which wrap past 2^31.
export function deflateBound(len: number): number {
return len + Math.floor(len / 4096) + Math.floor(len / 16384) + Math.floor(len / 33554432) + 13;
}
1 change: 0 additions & 1 deletion quest/m1/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,6 @@ were done or not worth their cost.
- [Cross-relay bursts re-run](/quest/m1/cross-relay-bursts.md) - condition: the #4349 reporter re-runs their A/B/C comparison against current cdn.moq.pro
- [Untimed lite-07](/quest/m1/lite-untimed.md) - lite-07 carries an untimed track in both languages; lite-05/06 write send time
- [Late lower groups](/quest/m1/lite-late-lower-group.md) - a moq-lite subscriber with a floor receives a group created below the first served one, as moq-transport does
- [Flate stream budget](/quest/m1/flate-stream-budget.md) - a flate stream refuses an oversized append without ending, sharing one DEFLATE bound with json
- [A watch and publish release ships assets()](/quest/m1/assets-release.md) - the release that lets the sites host the worklets
- [Dogfood hosted worklets](/quest/m1/dogfood-assets.md) - moq.dev and the moq.pro dashboard host the worklets and call `assets()` after the release
- [Media audio-tone check](/quest/m1/media-audio-tone.md) - the media lane's audio-tone check passes under load, fixed at its cause
Expand Down
Loading
Loading