Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions doc/concept/stats.md
Original file line number Diff line number Diff line change
Expand Up @@ -186,8 +186,8 @@ writes only when a track's frame changed, so silence means nothing moved, not
that the producer is gone. The track ends when the producer does, or at
depth 1 or more when its group's broadcast is unannounced; the group may
return later as a new broadcast. A frame too large to cache (32 MB) ends its
track with a `FrameTooLarge` error, while the `.z` flavor, which compresses
under the limit, keeps updating.
track with a `FrameTooLarge` error. The `.z` flavor meets the same limit by its
uncompressed size, since a reader inflates the whole group.

### Totals

Expand Down Expand Up @@ -260,7 +260,7 @@ real sums. Started minus ended stays exact. A child appears while a broadcast
under it has counters, and leaves after the frame carrying its last one; its
tail stays in the rollup, and a child that returns starts over. Children are
never capped, so roughly ten thousand across four tiers outgrow the 32 MB frame
limit: that encoding's track then ends with `FRAME_TOO_LARGE` instead of going
limit: both encodings' tracks then end with `FRAME_TOO_LARGE` instead of going
stale.

## Entries
Expand Down
6 changes: 6 additions & 0 deletions js/flate/MIGRATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,12 @@ How to update your code for each breaking change, newest first.

## Unreleased

### A compressed group is held to the group budget

`Decoder` holds its whole window, across every frame, to `Group.MAX_GROUP_CACHE_BYTES` (32 MiB) instead of capping each frame at 64 MiB.
`DEFAULT_MAX_FRAME_SIZE`, `DecoderOptions`, and `Decoder.maxFrameSize` are removed; construct with `new Decoder()`.
The producers charge a compressed payload its inflated size: a snapshot value over 32 MiB throws `FrameTooLarge` however small it deflates, and a stream log ends its budget at 32 MiB of inflated payload.

### @moq/binary is @moq/flate

The opaque `Snapshot` and `Stream` tracks moved beside the codec; the wire and the catalog's `binary` section are unchanged.
Expand Down
2 changes: 1 addition & 1 deletion js/flate/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,4 +50,4 @@ new TextDecoder().decode(decoder.frame(a)); // "the quick brown fox"
new TextDecoder().decode(decoder.frame(b)); // "the quick brown dog"
```

Create a fresh `Encoder`/`Decoder` pair per independent stream (in moq-net terms, per group). `new Encoder({ level })` sets the DEFLATE level (`0..=9`, default `6`); `new Decoder({ maxFrameSize })` caps how far a single frame may inflate (default 64 MiB), rejecting zip bombs.
Create a fresh `Encoder`/`Decoder` pair per independent stream (in moq-net terms, per group). `new Encoder({ level })` sets the DEFLATE level (`0..=9`, default `6`); a `Decoder` holds its whole window to the group budget (32 MiB), rejecting zip bombs: a compressed group never costs a reader more than a plain one.
22 changes: 11 additions & 11 deletions js/flate/src/codec.test.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import { expect, test } from "bun:test";
import { Group } from "@moq/net";
import { Deflate, Inflate } from "fflate";
import { Decoder, Encoder } from "./codec.ts";

Expand Down Expand Up @@ -64,19 +65,18 @@ test("codec rejects garbage", () => {
expect(() => decoder.frame(new Uint8Array(64).fill(0xff))).toThrow();
});

test("codec rejects a frame that inflates past the cap", () => {
// A tiny slice can inflate enormously, so the decoder bounds the output as it is produced. The
// cap's value doesn't change that path, so use a small one instead of paying for a real bomb.
const slice = new Encoder().frame(new Uint8Array(1024));
const decoder = new Decoder({ maxFrameSize: 512 });
expect(() => decoder.frame(slice)).toThrow(/exceeded 512/);
});
test("the decoder holds its whole window to the group budget", () => {
// Each frame fits on its own and deflates to almost nothing; together they inflate past what a
// plain group can carry.
const half = new Uint8Array(Group.MAX_GROUP_CACHE_BYTES / 2 + 1);
const encoder = new Encoder();
const first = encoder.frame(half);
const second = encoder.frame(half);
expect(second.length).toBeLessThan(64 * 1024);

test("a decoder's frame cap cannot be lifted after construction", () => {
const decoder = new Decoder();
expect(() => {
(decoder as { maxFrameSize: number }).maxFrameSize = Infinity;
}).toThrow(TypeError);
expect(decoder.frame(first).length).toBe(half.length);
expect(() => decoder.frame(second)).toThrow(/decompressed group exceeded/);
});

test("a frame larger than pako's chunk size round-trips", () => {
Expand Down
40 changes: 15 additions & 25 deletions js/flate/src/codec.ts
Original file line number Diff line number Diff line change
@@ -1,13 +1,11 @@
// The group-scoped DEFLATE codec: see the module docs in index.ts.

import { Group } from "@moq/net";
import * as pako from "pako";

/** The default DEFLATE level ({@link Encoder}): a good size/speed balance for small, repetitive payloads. */
export const DEFAULT_LEVEL = 6;

/** The default per-frame decompressed-size cap ({@link Decoder}): 64 MiB. */
export const DEFAULT_MAX_FRAME_SIZE = 64 * 1024 * 1024;

// The trailing bytes of a DEFLATE sync flush, stripped on the wire and re-appended to decode.
const SYNC_FLUSH_TAIL = new Uint8Array([0x00, 0x00, 0xff, 0xff]);

Expand Down Expand Up @@ -92,42 +90,33 @@ export class Encoder {
}
}

/** Options for a {@link Decoder}. */
export interface DecoderOptions {
/**
* Per-frame decompressed-size cap. A frame that inflates past this is rejected (zip-bomb guard).
* Defaults to {@link DEFAULT_MAX_FRAME_SIZE}.
*/
maxFrameSize?: number;
}

/**
* Decodes a stream's frame slices back into the original payloads. Hold one per stream; feed slices
* in frame order (each frame builds on the earlier ones).
*
* A peer could send tiny slices that inflate hugely, so the window's total output is held to the
* group budget, `Group.MAX_GROUP_CACHE_BYTES`: the most a plain group can carry. A compressed group
* then costs a reader no more than a plain one, however small its slices.
*
* @public
*/
export class Decoder {
#inflate = new pako.Inflate({ raw: true });
#chunks: Uint8Array[] = [];
#total = 0;
// Bytes the whole window has inflated to, across every frame.
#inflated = 0;
#tooLarge = false;
#maxFrameSize: number;

/** The per-frame decompressed-size cap in bytes, in effect for this decoder. */
get maxFrameSize(): number {
return this.#maxFrameSize;
}

/** Start a fresh decoder with a cold window. */
constructor(options: DecoderOptions = {}) {
this.#maxFrameSize = options.maxFrameSize ?? DEFAULT_MAX_FRAME_SIZE;
constructor() {
this.#inflate.onData = (chunk) => {
const bytes = chunk as Uint8Array;
this.#total += bytes.length;
this.#inflated += bytes.length;
// Bound the inflated output as it is produced; a tiny slice can expand enormously. Stop
// retaining past the cap, then reject once the push returns.
if (this.#total > this.#maxFrameSize) {
// retaining past the budget, then reject once the push returns.
if (this.#inflated > Group.MAX_GROUP_CACHE_BYTES) {
this.#tooLarge = true;
return;
}
Expand All @@ -137,21 +126,22 @@ export class Decoder {

/**
* Decompress the next frame's `slice` back into its payload. Empty in yields empty out. Throws if
* the input is malformed or inflates past the per-frame size cap.
* the input is malformed, or once the window's output across every frame so far passes the group
* budget.
*/
frame(slice: Uint8Array): Uint8Array {
if (slice.length === 0) return slice;
if (this.#tooLarge) throw new Error(`decompressed group exceeded ${Group.MAX_GROUP_CACHE_BYTES} bytes`);

this.#chunks = [];
this.#total = 0;
this.#tooLarge = false;

// Feed the slice then the re-appended sync-flush marker as two pushes, so no combined buffer is
// allocated. The marker delimits the frame and flushes its last bytes out of the inflate buffer.
this.#inflate.push(slice, false);
this.#inflate.push(SYNC_FLUSH_TAIL, pako.Z_SYNC_FLUSH);
if (this.#inflate.err) throw new Error(`decompression failed: ${this.#inflate.msg}`);
if (this.#tooLarge) throw new Error(`decompressed frame exceeded ${this.#maxFrameSize} bytes`);
if (this.#tooLarge) throw new Error(`decompressed group exceeded ${Group.MAX_GROUP_CACHE_BYTES} bytes`);

return concat(this.#chunks, this.#total);
}
Expand Down
14 changes: 4 additions & 10 deletions js/flate/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -34,20 +34,14 @@
* A sync flush always ends in the fixed 4-byte marker `00 00 ff ff`. {@link Encoder.frame} drops it
* and {@link Decoder.frame} re-appends it, saving 4 bytes per frame, the same trick
* [RFC 7692](https://www.rfc-editor.org/rfc/rfc7692.html#section-7.2.1) (permessage-deflate) uses. A
* small slice can still inflate enormously, so {@link Decoder.frame} caps the inflated output as it
* is produced. pako is synchronous, so the whole codec is synchronous.
* small slice can still inflate enormously, so a {@link Decoder} holds its whole window to the group
* budget as output is produced: a compressed group never costs a reader more than a plain one, and
* the producers refuse before a group would. pako is synchronous, so the whole codec is synchronous.
*
* @module
*/

export {
DEFAULT_LEVEL,
DEFAULT_MAX_FRAME_SIZE,
Decoder,
type DecoderOptions,
Encoder,
type EncoderOptions,
} from "./codec.ts";
export { DEFAULT_LEVEL, Decoder, Encoder, type EncoderOptions } from "./codec.ts";
export type { Compression } from "./compression.ts";
/** Lossy latest-value opaque publishing. */
export * as Snapshot from "./snapshot/index.ts";
Expand Down
17 changes: 7 additions & 10 deletions js/flate/src/snapshot/producer.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import type { Timed } from "@moq/net";
import * as Moq from "@moq/net";
import { DEFAULT_MAX_FRAME_SIZE, Encoder as Flate } from "../codec.ts";
import { Encoder as Flate } from "../codec.ts";

import { type Compression, isDeflate } from "../compression.ts";

Expand Down Expand Up @@ -43,19 +43,16 @@ export class Producer {
* untimed track: it is never filled in.
*/
update({ value: payload, at }: Timed<Uint8Array>): void {
// Consumers all decode with `@moq/flate`'s default cap, so publishing past it would advertise
// a value that always fails to read. Rejected before anything is published; unlike a stream
// this is not terminal, since the previous value still stands and the next update supersedes.
if (this.#compress && payload.byteLength > DEFAULT_MAX_FRAME_SIZE) {
throw new Error(`payload larger than the decoder's ${DEFAULT_MAX_FRAME_SIZE} byte limit`);
}
// Check before opening a group. `appendGroup` publishes immediately, so letting `writeFrame`
// reject the frame would leave an empty newest group behind: a snapshot consumer jumps to the
// newest, so the previous value would be lost even though this update threw. A compressed value
// is held to the same budget by its inflated size, since that is what every consumer's decoder
// allows: compression saves bandwidth, never what a reader holds.
if (payload.byteLength > Moq.Group.MAX_GROUP_CACHE_BYTES) throw new Moq.Error.FrameTooLarge();

// One frame per group, so the window spans a single value and starts cold every time.
const encoded = this.#compress ? new Flate().frame(payload) : payload;

// Check before opening a group. `appendGroup` publishes immediately, so letting `writeFrame`
// reject the frame would leave an empty newest group behind: a snapshot consumer jumps to the
// newest, so the previous value would be lost even though this update threw.
if (encoded.byteLength > Moq.Group.MAX_GROUP_CACHE_BYTES) throw new Moq.Error.FrameTooLarge();

const group = this.#track.appendGroup();
Expand Down
16 changes: 15 additions & 1 deletion js/flate/src/snapshot/snapshot.test.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { expect, test } from "bun:test";
import { Milli } from "@moq/clock";
import { Time, Track } from "@moq/net";
import { Group, Error as NetError, Time, Track } from "@moq/net";
import { Consumer, Producer } from "./index.ts";

const bytes = (...values: number[]) => new Uint8Array(values);
Expand Down Expand Up @@ -85,6 +85,20 @@ test("compression shrinks the frame on the wire", async () => {
expect(frame?.payload.byteLength).toBeLessThan(payload.byteLength / 4);
});

// Compression doesn't lift the group limit: a reader inflates the whole value, so one past the budget
// is refused however small it deflates, and the previous value stays readable.
test("a compressed value past the group budget is refused", async () => {
const track = new Track.Producer("test");
const producer = new Producer({ track, compression: "deflate" });
producer.update({ value: bytes(1) });
expect(() => producer.update({ value: new Uint8Array(Group.MAX_GROUP_CACHE_BYTES + 1) })).toThrow(
NetError.FrameTooLarge,
);
producer.finish();

expect(await drain(track.subscribe(), true)).toEqual([bytes(1)]);
});

test("a finished track ends the consumer", async () => {
const track = new Track.Producer("test");
const producer = new Producer({ track });
Expand Down
12 changes: 6 additions & 6 deletions js/flate/src/stream/producer.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,9 @@ export class Producer {
#flate?: Flate;
// The single group carrying the whole log, opened on the first append and never rolled.
#group?: Moq.Group.Producer;
// Frames and payload bytes written to the group, checked against the group budget before each
// payload is encoded.
// Frames and bytes charged to the group, checked against the group budget before each payload is
// encoded. A compressed payload is charged its inflated size (or its slice, if larger), the budget
// every consumer's `Decoder` holds its window to.
#frames = 0;
#bytes = 0;

Expand Down Expand Up @@ -60,9 +61,8 @@ export class Producer {

// Check before compressing: encoding advances the window, so a payload refused afterwards
// would leave the encoder ahead of every reader. The worst case is checked rather than the
// actual size for the same reason. Also checked before the group is opened, so a refused
// first payload publishes nothing. The budget is below the decoder's cap, so any payload that
// fits is one every consumer can inflate.
// actual size for the same reason, and it covers the inflated size too. Also checked before
// the group is opened, so a refused first payload publishes nothing.
const bound = this.#flate ? Flate.bound(payload.byteLength) : payload.byteLength;
if (this.#frames >= Group.MAX_GROUP_FRAMES || this.#bytes + bound > Group.MAX_GROUP_CACHE_BYTES) {
throw new NetError.GroupTooLarge();
Expand Down Expand Up @@ -95,7 +95,7 @@ export class Producer {
}

this.#frames++;
this.#bytes += encoded.byteLength;
this.#bytes += Math.max(payload.byteLength, encoded.byteLength);
}

/** Finish the track, closing the group. */
Expand Down
25 changes: 13 additions & 12 deletions js/flate/src/stream/stream.test.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
import { expect, spyOn, test } from "bun:test";
import { Milli } from "@moq/clock";
import { Group, Error as NetError, Time, Track } from "@moq/net";
import { DEFAULT_MAX_FRAME_SIZE } from "../codec.ts";
import { Consumer, Producer, Rolled } from "./index.ts";

// Ask for a replay window, so the superseded first group is delivered rather than skipped by the
Expand Down Expand Up @@ -173,18 +172,14 @@ test("an ended log refuses with its own error", () => {
});

// A payload past the group budget is refused before anything is written, so the log carries on: the
// next payload lands and decodes, which with compression proves the window never moved. One past the
// decoder's cap is refused the same way rather than ending the track.
// next payload lands and decodes, which with compression proves the window never moved.
for (const compression of [false, true]) {
test(`an oversized payload is refused and the log continues (compression=${compression})`, async () => {
const track = new Track.Producer("test");
const producer = new Producer({ track, compression: compression ? "deflate" : "none" });
producer.append({ value: payloads(1)[0] });

const huge = new Uint8Array(DEFAULT_MAX_FRAME_SIZE + 1);
expect(() => producer.append({ value: huge.subarray(0, Group.MAX_GROUP_CACHE_BYTES) })).toThrow(
NetError.GroupTooLarge,
);
const huge = new Uint8Array(Group.MAX_GROUP_CACHE_BYTES);
expect(() => producer.append({ value: huge })).toThrow(NetError.GroupTooLarge);

producer.append({ value: payloads(2)[1] });
Expand All @@ -193,16 +188,22 @@ for (const compression of [false, true]) {
});
}

// The budget check stands in for the decoder's cap: a payload that fits is one every consumer can
// inflate.
test("the group budget is within the decoder's cap", () => {
expect(Group.MAX_GROUP_CACHE_BYTES).toBeLessThanOrEqual(DEFAULT_MAX_FRAME_SIZE);
// A compressed payload is charged its inflated size, since that is what every reader's decoder holds
// the group to: two payloads that each fit, and deflate to almost nothing, still overflow together.
test("a compressed log is charged its inflated size", async () => {
const track = new Track.Producer("test");
const producer = new Producer({ track, compression: "deflate" });
const half = new Uint8Array(Group.MAX_GROUP_CACHE_BYTES / 2 + 1);
producer.append({ value: half });
expect(() => producer.append({ value: half })).toThrow(NetError.GroupTooLarge);
producer.finish();
expect(await drain(track.subscribe(), true)).toEqual([half]);
});

test("a refused first payload publishes nothing", async () => {
const track = new Track.Producer("test");
const producer = new Producer({ track, compression: "deflate" });
expect(() => producer.append({ value: new Uint8Array(DEFAULT_MAX_FRAME_SIZE + 1) })).toThrow(
expect(() => producer.append({ value: new Uint8Array(Group.MAX_GROUP_CACHE_BYTES + 1) })).toThrow(
NetError.GroupTooLarge,
);
producer.finish();
Expand Down
6 changes: 6 additions & 0 deletions js/json/MIGRATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,12 @@ How to update your code for each breaking change, newest first.

## Unreleased

### A compressed group is held to the group budget

A compressed group may inflate to at most `Group.MAX_GROUP_CACHE_BYTES` (32 MiB), the most a plain group carries.
The encoders charge each frame its plaintext: a compressed snapshot over 32 MiB throws `FrameTooLarge` (it used to pass up to 64 MiB), a window header over it throws, and deltas and ops roll a new group once a group's plaintext would pass the budget.
A compressed stream's budget is 32 MiB of record plaintext.

### Groups roll on joiner demand

`Snapshot` `deltaRatio` and `Window` `opRatio` are replaced by `deltas` and `ops` booleans (both default `true`).
Expand Down
Loading
Loading