Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions demo/web/src/publish.ts
Original file line number Diff line number Diff line change
Expand Up @@ -391,7 +391,7 @@ let activeMeta: Json.Snapshot.Producer<unknown> | undefined;

const setMeta = (value: unknown) => {
currentMeta = value;
activeMeta?.update(value);
activeMeta?.update({ value, at: Net.Time.Timestamp.now() });
};

const meta = new Signals.Effect();
Expand All @@ -408,7 +408,7 @@ meta.run((effect) => {
effect.cleanup(() => track.close());

const producer = new Json.Snapshot.Producer<unknown>({ track });
producer.update(currentMeta);
producer.update({ value: currentMeta, at: Net.Time.Timestamp.now() });
activeMeta = producer;
effect.cleanup(() => {
if (activeMeta === producer) activeMeta = undefined;
Expand Down
10 changes: 6 additions & 4 deletions doc/lib/js/flate.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,14 +19,16 @@ group-scoped DEFLATE `@moq/json` uses.

```ts
import { Snapshot } from "@moq/flate";
import { Time } from "@moq/net";

const producer = new Snapshot.Producer({ track, compression: "deflate" });
producer.update(payload);
producer.update({ value: payload, at: Time.Timestamp.now() });
```

A payload is stamped when written, unless you pass its capture time:
`producer.update(payload, at)`. Every frame carries a timestamp, so the track must
declare a timescale, such as `createTrack(name, { timescale: Time.Timescale.MILLI })`.
Producers take a `Timed` value, `{ value, at }`, where `at` is the capture
time written as the frame timestamp. Nothing fills in now: a timed track, such
as `createTrack(name, { timescale: Time.Timescale.MILLI })`, needs `at` on
every write, and an untimed track (no `timescale`) takes none.

The codec underneath is exported as `Encoder`/`Decoder`. Create one pair per
group and feed frames in order.
Expand Down
7 changes: 4 additions & 3 deletions doc/lib/js/json.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,10 @@ reader will not pick it up.
- **Stream**: lossless append-log in a single group. A reader that falls behind fails the read rather than resuming mid-log.
- **Window**: a bounded run of records a reader can join at any point. A new group restates what it keeps explicitly, so a reader that was keeping up is not handed a record twice.

Both sides choose the same compression, `"none"` or `"deflate"`. A value is
stamped when written, unless you pass its capture time, so the track must
declare a timescale.
Both sides choose the same compression, `"none"` or `"deflate"`. Producers take
a `Timed` value, `{ value, at }`, where `at` is the capture time written as
the frame timestamp. Nothing fills in now: a timed track needs `at` on every
write, and an untimed track (no `timescale`) takes none.

A stream rides one group, so the whole log shares `@moq/net`'s group budget:
32 MiB of payload and 8192 records. An append that might not fit is refused
Expand Down
4 changes: 2 additions & 2 deletions doc/lib/js/net.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ const connection = await Moq.Connection.connect({
const broadcast = origin.createBroadcast(Moq.Path.from("chat.room"));
const track = broadcast.createTrack("messages", { timescale: Moq.Time.Timescale.MILLI });
const group = track.appendGroup();
group.writeString("hello");
group.writeFrame({ payload: new TextEncoder().encode("hello"), timestamp: Moq.Time.Timestamp.now() });
group.close();
broadcast.announce();
```
Expand All @@ -39,7 +39,7 @@ broadcast.announce();
- **GOAWAY moves the session.** The connection dials the replacement at once while the old session keeps serving its groups, up to `goaway.handover` (default 10s, or the relay's deadline when that is sooner). New requests keep opening on the old session until the replacement's route outranks it. An empty GOAWAY redials the same URL. A redirect stays on the same host unless `goaway.redirect` is `"follow"`.
- **One send estimate per connection.** `Bandwidth.Allocator` divides it by track priority, max-min fair within a tier. An idle track claims nothing. Publishers reserve against it so their targets sum to the estimate.
- **One track per name.** Concurrent subscriptions share one request and producer. An on-demand producer that replaces an ended one continues the name's group and datagram sequences; only a new broadcast restarts them. `createTrack` and `insertTrack` start at 0.
- **Timing is per track, with no default.** A track that declares a `timescale` carries a `timestamp` on every frame; one without is [untimed](/concept/moq-lite#subscriptions) and its frames carry none. Unlike Rust, omitting it means untimed.
- **Timing is per track, with no default.** A track that declares a `timescale` carries a `timestamp` on every frame; one without is [untimed](/concept/moq-lite#subscriptions) and its frames carry none. Unlike Rust, omitting it means untimed. Nothing stamps a frame for you: the publisher passes the timestamp, and the `Timed` type (`{ value, at }`) carries one into the `@moq/json` and `@moq/flate` producers.
- **Subscriber staleness is `maxDelay`.** It is media time. Passing the old `maxAge` key throws a `TypeError` naming `maxDelay`. Publisher retention is the separate `Track.Info.maxAge`.
- **Hidden paths** stay out of discovery unless the announce request opts in. See [hidden broadcasts](/concept/moq-lite#hidden-broadcasts).
- **A graceful close waits.** `await connection.close()` withdraws announcements and gives finished tracks up to one second. `abort()` ends immediately.
Expand Down
6 changes: 3 additions & 3 deletions doc/lib/js/publish.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,16 +75,16 @@ signals.run((effect) => {
const net = effect.get(broadcast.net);
if (!net) return;

// A day-long retention so a late viewer still replays the last value. JSON
// values are stamped when written, so the track declares a timescale.
// A day-long retention so a late viewer still replays the last value. Each
// value is stamped with when it was written, so the track declares a timescale.
const track = net.createTrack("meta.json", {
timescale: Moq.Time.Timescale.MILLI,
maxAge: Moq.Time.Milli(86_400_000),
});
effect.cleanup(() => track.close());

const meta = new Json.Snapshot.Producer<Meta>({ track });
meta.update(current);
meta.update({ value: current, at: Moq.Time.Timestamp.now() });
});
```

Expand Down
5 changes: 3 additions & 2 deletions js/clock/src/main.ts
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,7 @@ async function publish(config: Config) {
async function publishTrack(track: Moq.Track.Producer) {
// Send timestamps over the wire, matching the Rust implementation format
console.log("✅ Publishing clock data on track:", track.name);
const encoder = new TextEncoder();

for (;;) {
const now = new Date();
Expand All @@ -100,7 +101,7 @@ async function publishTrack(track: Moq.Track.Producer) {

// Send the base timestamp (everything but seconds) - matching Rust format
const base = `${now.toISOString().slice(0, 16).replace("T", " ")}:`;
group.writeString(base);
group.writeFrame({ payload: encoder.encode(base), timestamp: Moq.Time.Timestamp.now() });

// Send individual seconds for this minute
const currentMinute = now.getMinutes();
Expand All @@ -109,7 +110,7 @@ async function publishTrack(track: Moq.Track.Producer) {
const secondsNow = new Date();
const seconds = secondsNow.getSeconds().toString().padStart(2, "0");

group.writeString(seconds);
group.writeFrame({ payload: encoder.encode(seconds), timestamp: Moq.Time.Timestamp.now() });

// Wait until next second
const nextSecond = new Date(secondsNow);
Expand Down
5 changes: 4 additions & 1 deletion js/flate/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,10 @@ The bytes are opaque: the tracks frame them onto a [`@moq/net`](../net) track an

```ts
import { Snapshot, Stream } from "@moq/flate";
import { Time } from "@moq/net";

const thumbnail = new Snapshot.Producer({ track, compression: "deflate" });
thumbnail.update(jpeg);
thumbnail.update({ value: jpeg, at: Time.Timestamp.now() });

const log = new Stream.Consumer({ track: subscriber, compression: "deflate" });
for (;;) {
Expand All @@ -29,6 +30,8 @@ for (;;) {

Compression is opt-in per track (`compression: "none" | "deflate"`, default `"none"`); a consumer must set the same value as the producer.

Producers take `{ value, at }`, where `at` is the capture time written as the frame timestamp. Nothing fills in now: a timed track needs `at` on every write, and an untimed track takes none.

## Codec

A sequence of frame payloads is compressed into a single raw DEFLATE ([RFC 1951](https://www.rfc-editor.org/rfc/rfc1951.html)) stream, sync-flushed at each frame boundary. Every frame is self-delimited (byte-aligned, the window retained) while later frames reuse the earlier ones as context, so a stream of similar payloads (a snapshot followed by deltas, repeated records, log lines) compresses far better than each payload alone.
Expand Down
7 changes: 4 additions & 3 deletions js/flate/src/snapshot/producer.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import type { Timed } from "@moq/net";
import * as Moq from "@moq/net";
import { Time } from "@moq/net";
import { DEFAULT_MAX_FRAME_SIZE, Encoder as Flate } from "../codec.ts";

import { type Compression, isDeflate } from "../compression.ts";
Expand Down Expand Up @@ -39,9 +39,10 @@ export class Producer {
* Unlike `@moq/json`, an identical value is republished rather than skipped: comparing two
* opaque blobs costs a full scan, and only the caller knows whether its bytes changed.
*
* `at` is when the payload was captured, written as its frame timestamp. Defaults to now.
* `at` is when the payload was captured, written as its frame timestamp. Omit it only on an
* untimed track: it is never filled in.
*/
update(payload: Uint8Array, at: Time.Timestamp = Time.Timestamp.now()): void {
update({ value: payload, at }: Timed<Uint8Array>): void {
// Consumers all decode with `@moq/flate`'s default cap, so publishing past it would advertise
// a value that always fails to read. Rejected before anything is published; unlike a stream
// this is not terminal, since the previous value still stands and the next update supersedes.
Expand Down
26 changes: 18 additions & 8 deletions js/flate/src/snapshot/snapshot.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,8 @@ async function drain(track: Track.Subscriber, compression: boolean): Promise<Uin
test("one single-frame group per update", async () => {
const track = new Track.Producer("test");
const producer = new Producer({ track });
producer.update(bytes(1));
producer.update(bytes(2));
producer.update({ value: bytes(1) });
producer.update({ value: bytes(2) });
producer.finish();

// Two updates => two self-contained groups, so a consumer never needs an older one: a reader
Expand All @@ -49,7 +49,7 @@ test("a live consumer sees each update", async () => {
const consumer = new Consumer({ track: track.subscribe() });

for (let n = 0; n < 3; n++) {
producer.update(bytes(n));
producer.update({ value: bytes(n) });
expect(await consumer.next()).toEqual(bytes(n));
}
producer.finish();
Expand All @@ -59,7 +59,7 @@ test("compressed roundtrip", async () => {
const track = new Track.Producer("test");
const producer = new Producer({ track, compression: "deflate" });
const payload = new TextEncoder().encode("the quick brown fox".repeat(64));
producer.update(payload);
producer.update({ value: payload });
producer.finish();

expect(await drain(track.subscribe(), true)).toEqual([payload]);
Expand All @@ -71,7 +71,7 @@ test("compression shrinks the frame on the wire", async () => {
const track = new Track.Producer("test");
const producer = new Producer({ track, compression: "deflate" });
const payload = new TextEncoder().encode("the quick brown fox".repeat(64));
producer.update(payload);
producer.update({ value: payload });
producer.finish();

const group = await track.subscribe().ordered().nextGroup();
Expand All @@ -82,7 +82,7 @@ test("compression shrinks the frame on the wire", async () => {
test("a finished track ends the consumer", async () => {
const track = new Track.Producer("test");
const producer = new Producer({ track });
producer.update(bytes(7));
producer.update({ value: bytes(7) });
producer.finish();

const consumer = new Consumer({ track: track.subscribe() });
Expand All @@ -97,7 +97,7 @@ test("a backlog collapses to the newest value", async () => {
const producer = new Producer({ track });
const consumer = new Consumer({ track: track.subscribe() });

for (let n = 0; n < 10; n++) producer.update(bytes(n));
for (let n = 0; n < 10; n++) producer.update({ value: bytes(n) });
producer.finish();

expect(await consumer.next()).toEqual(bytes(9));
Expand Down Expand Up @@ -139,11 +139,21 @@ test("an aborted track surfaces its error instead of spinning", async () => {
await expect(consumer.next()).rejects.toThrow("subscription aborted");
});

test("a payload without a timestamp goes out untimed", async () => {
const track = new Track.Producer("test").accept({});
const producer = new Producer({ track });
producer.update({ value: bytes(1) });
producer.finish();

const frame = await (await track.subscribe().ordered().nextGroup())?.readFrame();
expect(frame?.timestamp).toBeUndefined();
});

test("a capture timestamp is written as the frame timestamp", async () => {
const track = new Track.Producer("test");
const producer = new Producer({ track });
const captured = Time.Timestamp.fromMillis(1_234);
producer.update(bytes(1), captured);
producer.update({ value: bytes(1), at: captured });
producer.finish();

const frame = await (await track.subscribe().ordered().nextGroup())?.readFrame();
Expand Down
7 changes: 4 additions & 3 deletions js/flate/src/stream/producer.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import type * as Moq from "@moq/net";
import { Time } from "@moq/net";
import type { Timed } from "@moq/net";
import { DEFAULT_MAX_FRAME_SIZE, Encoder as Flate } from "../codec.ts";

import { type Compression, isDeflate } from "../compression.ts";
Expand Down Expand Up @@ -41,9 +41,10 @@ export class Producer {
* group. The group is aborted rather than closed cleanly, so a consumer sees the failure
* instead of a log that merely looks complete. Every later append fails on the closed track.
*
* `at` is when the payload was captured, written as its frame timestamp. Defaults to now.
* `at` is when the payload was captured, written as its frame timestamp. Omit it only on an
* untimed track: it is never filled in.
*/
append(payload: Uint8Array, at: Time.Timestamp = Time.Timestamp.now()): void {
append({ value: payload, at }: Timed<Uint8Array>): void {
// A payload no consumer could decode is as terminal as one the track rejects: consumers all
// decode with `@moq/flate`'s default cap, so this would publish a record none of them could
// read. Ends the track like any other lost record, and aborts the group the same way the
Expand Down
30 changes: 20 additions & 10 deletions js/flate/src/stream/stream.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ test("every payload survives in order", async () => {
const track = new Track.Producer("test");
const producer = new Producer({ track });
const expected = payloads(5);
for (const payload of expected) producer.append(payload);
for (const payload of expected) producer.append({ value: payload });
producer.finish();

expect(await drain(track.subscribe(), false)).toEqual(expected);
Expand All @@ -35,7 +35,7 @@ test("compressed roundtrip in order", async () => {
const track = new Track.Producer("test");
const producer = new Producer({ track, compression: "deflate" });
const expected = payloads(20);
for (const payload of expected) producer.append(payload);
for (const payload of expected) producer.append({ value: payload });
producer.finish();

expect(await drain(track.subscribe(), true)).toEqual(expected);
Expand All @@ -44,7 +44,7 @@ test("compressed roundtrip in order", async () => {
test("the whole log rides one group, never rolled", async () => {
const track = new Track.Producer("test");
const producer = new Producer({ track, compression: "deflate" });
for (const payload of payloads(50)) producer.append(payload);
for (const payload of payloads(50)) producer.append({ value: payload });
producer.finish();

const subscriber = track.subscribe().ordered();
Expand All @@ -56,7 +56,7 @@ test("the shared window shrinks repetitive payloads", async () => {
const track = new Track.Producer("test");
const producer = new Producer({ track, compression: "deflate" });
const payload = new TextEncoder().encode("the quick brown fox".repeat(16));
for (let n = 0; n < 8; n++) producer.append(payload);
for (let n = 0; n < 8; n++) producer.append({ value: payload });
producer.finish();

const group = await track.subscribe().ordered().nextGroup();
Expand Down Expand Up @@ -94,7 +94,7 @@ test("a second concurrent read is refused rather than served the first one's gro
// group for a second one and fail a perfectly good log. Rust gets this from `&mut self`.
const track = new Track.Producer("test");
const producer = new Producer({ track });
producer.append(payloads(1)[0]);
producer.append({ value: payloads(1)[0] });
producer.finish();

const consumer = new Consumer({ track: track.subscribe() });
Expand Down Expand Up @@ -135,13 +135,13 @@ test("an undecodable payload ends the log for a reader already inside the group"
// failure rather than park on a group nothing will ever finish.
const track = new Track.Producer("test");
const producer = new Producer({ track, compression: "deflate" });
producer.append(payloads(1)[0]);
producer.append({ value: payloads(1)[0] });

const consumer = new Consumer({ track: track.subscribe(), compression: "deflate" });
expect(await consumer.next()).toBeDefined();

const oversized = new Uint8Array(DEFAULT_MAX_FRAME_SIZE + 1);
expect(() => producer.append(oversized)).toThrow("limit");
expect(() => producer.append({ value: oversized })).toThrow("limit");

// Surfaces the terminal error rather than hanging on the still-open group.
await expect(consumer.next()).rejects.toThrow("limit");
Expand Down Expand Up @@ -176,7 +176,7 @@ test("blocked reads leave nothing behind on the pending group read", async () =>
const reactions = await pendingReactions(async () => {
for (let n = 0; n < 1000; n++) {
const next = consumer.next();
producer.append(new Uint8Array([n & 0xff]));
producer.append({ value: new Uint8Array([n & 0xff]) });
expect((await next)?.[0]).toBe(n & 0xff);
}
});
Expand All @@ -186,11 +186,21 @@ test("blocked reads leave nothing behind on the pending group read", async () =>
producer.finish();
});

test("a payload without a timestamp goes out untimed", async () => {
const track = new Track.Producer("test").accept({});
const producer = new Producer({ track });
producer.append({ value: new Uint8Array([1]) });
producer.finish();

const group = await track.subscribe().ordered().nextGroup();
expect((await group?.readFrame())?.timestamp).toBeUndefined();
});

test("each record keeps its capture timestamp", async () => {
const track = new Track.Producer("test");
const producer = new Producer({ track });
producer.append(new Uint8Array([1]), Time.Timestamp.fromMillis(1_000));
producer.append(new Uint8Array([2]), Time.Timestamp.fromMillis(2_000));
producer.append({ value: new Uint8Array([1]), at: Time.Timestamp.fromMillis(1_000) });
producer.append({ value: new Uint8Array([2]), at: Time.Timestamp.fromMillis(2_000) });
producer.finish();

const group = await track.subscribe().ordered().nextGroup();
Expand Down
Loading
Loading