Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion demo/web/src/stats.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,8 @@
* `.stats/node/<node>` carrying JSON tracks that snapshot current activity. We
* auto-discover all of those nodes (announcements under `.stats/node`), so this
* works for a single relay and for a cluster alike, then aggregate each node and
* let you drill into one.
* let you drill into one. A restarted relay announces a new epoch at the same
* path, which arrives as an end then a start, so its history starts over.
*
* Per-node tracks we read:
* publisher.json egress (relay -> downstream viewers)
Expand Down Expand Up @@ -146,6 +147,10 @@ discovery.run((effect) => {
nodeStats.mutate((s) => {
delete s[node];
});
// A restart ends the old epoch and starts a new one at the same path,
// counting from zero, so neither series may splice across it.
history.delete(node);
clusterMembership = "";
}
}
});
Expand Down
5 changes: 4 additions & 1 deletion doc/bin/relay/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -238,7 +238,10 @@ Each node publishes `publisher.json`, `subscriber.json`, and `sessions.json`
tracks (plus compressed `.json.z` twins) of cumulative counters per broadcast
and auth root, split by a **tier** label chosen by the auth server's grant or
`--cluster-tier`, which is what makes billing per customer or per region
possible. [Stats](/concept/stats) describes the paths, tracks, and encodings;
possible. Each run, and each group returning after its linger, announces under
a fresh [epoch](/concept/moq-lite#publisher-epochs) on its route, so a restart
is a new broadcast at the same path.
[Stats](/concept/stats) describes the paths, tracks, and encodings;
read them with the [`moq-stats`](https://docs.rs/moq-stats) crate.

## \[iroh]
Expand Down
28 changes: 21 additions & 7 deletions doc/concept/stats.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,16 +33,27 @@ generate more stats.
prefix, so a consumer skips any path without `node` where it expects one. A
group segment literally named `node` is ambiguous; don't use one.

Each announcement carries a fresh [publisher epoch](/concept/moq-lite#publisher-epochs)
on its route, so a restarted node or a group returning from idle is a new
broadcast at the same path: it replaces the old one, and no relay serves it
groups cached under the old epoch. A reader treats each epoch as its own set of
counters. The epoch rides moq-lite 07 announcements; over older versions and
moq-transport a reader sees the same change as an end and a start at the path.
An aggregating reader pins each node's subscription to its epoch, and when one
publisher is reached through both kinds of link (with and without an epoch), it
carries only that path's outgoing counters across the change.

At depth 0 the broadcast stays announced for the producer's life. At depth
1 or more, a group's broadcast is announced while that group has entries, and
for a linger (five minutes by default) after its last one leaves. A group that
returns within the linger keeps its broadcast, so viewer churn doesn't
unannounce and re-announce it across the mesh; while it lingers empty, its
tracks hold `{}`. Once the linger elapses with the group still empty, the
broadcast is unannounced. Group numbers keep increasing across recreated
tracks and group broadcasts for the producer's life; they may have gaps. A
recreated compressed track starts a new group with a full snapshot, never a
delta whose compression state belonged to its previous writer.
broadcast is unannounced and its counters dropped; a group that returns later
announces a new epoch counted from zero. Within one epoch, group numbers keep
increasing across recreated tracks; they may have gaps. A recreated compressed
track starts a new group with a full snapshot, never a delta whose compression
state belonged to its previous writer.

## Tracks

Expand Down Expand Up @@ -132,9 +143,12 @@ Every counter is a cumulative, monotonic unsigned integer. A rate is the
difference between two frames divided by the time between them, and a live
count is started minus ended. A frame never shows ended above started.

A counter going **down** means the relay restarted or the entry was dropped
and re-created. Treat it as the start of a fresh segment rather than a
negative rate.
A new epoch starts every counter from zero, so a reader summing a node over
time adds each epoch's counters rather than diffing across them. Within one
epoch, a counter going **down** means the entry was dropped and re-created.
Treat it as the start of a fresh segment rather than a negative rate. On a
route without an epoch (an older relay, or a session older than moq-lite 07),
a counter going down also means the node restarted.

A reader ignores unknown fields, so a newer relay can add counters, and
defaults a missing field to zero, so it can read an older relay.
Expand Down
1 change: 1 addition & 0 deletions doc/lib/js/net.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ for (;;) {
```

- **Origins** hold the broadcasts, not the connection: closing a session unannounces them but leaves them created for the next one. `origin.request(path)` resolves an announced local broadcast with no round trip, so a page that watches what it publishes reads its own copy, unless a cheaper route announces the same path. A refusal from the most specific route ends the request: `request.closed` settles with the handler's error and no broader route is asked. Create, populate, then `announce()` for an exact path; use `dynamic(prefix, route)` when the set of paths is not known: an exact-path subscribe before the tracks exist is refused, and nobody, local or remote, can see or reach a broadcast until it announces.
- **Epoch pinning**: `consumer.request(path, { epoch })` pins an announced publisher instance. The request stays unresolved and reports `unroutable` when the winning route names a different or missing epoch, including while an asynchronous handler answers. Omit `epoch` for unpinned lookup. The resolved broadcast's `epoch` names the route it actually resolved through, pinned or not.
- **Connections** race WebTransport against WebSocket. `new Connection({ url })` pools one connection per relay URL and reconnects with backoff, which the elements use. Supplying WebTransport/WebSocket options, discovery, delay, `goaway`, or a caller-owned origin selects a private loop; explicit `share: true` refuses those options. On a relay's GOAWAY the connection dials the replacement at once while the old session keeps serving its groups until it closes or the `goaway.handover` cap (default 10s, lowered to the relay's deadline) passes; the handle and its origin stay the same. An empty GOAWAY redials the same URL through a fresh DNS resolve. A redirect is followed onto the same host by default (`goaway.redirect: "follow"` lets the relay pick the host, `"ignore"` never moves), and moves the pooled entry to the new URL; a malformed or refused one, including a host change under a pinned certificate, ends the connection with `Error.RefusedRedirect` instead of reconnecting. `closed` settles when the handle is released (`null` on a clean close); the failure that stopped retrying the current URL is `error`, and a new URL recovers the same handle. A connection owns one send-rate sampler and one `Bandwidth.Allocator`; publishers reserve against it so their encoder targets sum to the estimate instead of each matching it.
- **Bandwidth** (`Bandwidth.Allocator`) divides the connection's send-rate estimate by track priority, max-min fair within a tier. An idle track claims nothing. The receive side is untouched.
- **Discovery** by any pattern scope (`origin.announced(scope)`, such as `room/*/chat`; default everything). A route event's `kind` says whether the route started, updated, or ended; its `prefix` is the covered prefix relative to the origin, and `captures` reports what the scope's wildcards matched when the prefix pins them. The consumer is an async iterable. `origin.broadcasts(scope)` is a live `Getter<ReadonlyMap<Path.Valid, Route>>` of the same covered prefixes for UIs that need the current set. A borrowed `Connection.origin` also exposes `dynamic(prefix, route)` for serving paths on demand.
Expand Down
2 changes: 1 addition & 1 deletion doc/lib/rs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ let mut announced = consumer.announced();
while let Some(event) = announced.next().await {
// Skip updates and retractions.
let moq_net::announce::Event::Start(update) = event else { continue };
let broadcast = consumer.request_broadcast(&update.prefix).await?;
let broadcast = consumer.request_broadcast(&update.prefix, update.route.epoch).await?;
let catalog = broadcast
.track(hang::Catalog::DEFAULT_NAME)?
.subscribe(hang::Catalog::default_subscription())
Expand Down
3 changes: 2 additions & 1 deletion doc/lib/rs/moq-net.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,8 @@ above ([hang](/lib/rs/hang)); relays and CDNs implement only this.

- **Origins** scope what a session can see, and merge duplicate subscriptions so a broadcast is pulled upstream once no matter how many local readers.
- **Broadcasts** are created unannounced and invisible to everyone, then announced as an exact route, or served below a prefix with `dynamic`. A consumer of the same origin sees exactly what a peer sees. Discovery accepts pattern unions; events carry the advertised prefix and captures for a complete match.
- **Epochs** identify publisher instances with a canonical UUIDv7 carried on each route (`Route::epoch`): the newest wins a path, and only routes with the same epoch resume a subscription. A resolved `broadcast::Consumer` names the epoch it came through in `info().epoch`; see [publisher epochs](/concept/moq-lite#publisher-epochs).
- **Resolution** uses `consumer.request_broadcast(path, epoch)` with an owned `Epoch` to pin a publisher instance, or `None` for unpinned lookup. A pinned request refuses a different or missing epoch at request time and asynchronous resolution.
- **Epochs** identify publisher instances with a canonical UUIDv7 carried on each route (`Route::epoch`): the newest wins a path, and only routes with the same epoch resume a subscription. A resolved `broadcast::Consumer` names the epoch it came through in `info().epoch`, pinned or not; see [publisher epochs](/concept/moq-lite#publisher-epochs).
- **Patterns** (`Pattern`, `Patterns`) are re-exported from [`moq-pattern`](https://docs.rs/moq-pattern). Literal `Path` stays a coordinate.
- **Tracks** carry groups with a priority, an optional publisher retention window (`Info::max_age`), and a timescale, or none for an untimed track whose frames carry no timestamp. Subscribers set their own priority and max delay and can change them live.
- **Groups** are written frame by frame and delivered on independent streams. Old groups are cached for fetch-by-sequence; stale groups are skipped per the subscriber's budget.
Expand Down
9 changes: 8 additions & 1 deletion doc/setup/upgrade.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,13 @@ These land with the next breaking release, not the 2026-09-23 train.
`producer.demand().used` and `.unused()`, as are `Group.Producer`'s, and
`Allocator.reserve` takes `producer.demand()`, replacing the
`Bandwidth.Demand` interface.
- **`request_broadcast` takes an epoch.** In Rust,
`origin::Consumer::request_broadcast(path)` is
`request_broadcast(path, None)`; pass an `Epoch` instead to refuse any other
publisher instance. In TypeScript, `RequestOptions.epoch` does the same. A
dynamic route updated to another epoch now refuses the requests its handler
still holds with `Unroutable`, so a handler answering them late sees its
answer dropped.
- **`moq --hop` is removed; `--epoch` replaces it.** A redundant pair shares an
epoch, a UUIDv7 such as `uuidgen -7` prints, instead of a Hop ID: pass the
same `--epoch` (or `MOQ_EPOCH`) to both. Unlike a rename, `--hop` is now an
Expand Down Expand Up @@ -335,7 +342,7 @@ Other changes to a deployment:
`origin::Config`.
- **Announcements are prefix routes** (#3225, #3770). `announce::Update` is
`{ prefix, route, kind, captures }`: skip `!update.kind.is_active()` and
resolve the broadcast with `consumer.request_broadcast(&update.prefix)`.
resolve the broadcast with `consumer.request_broadcast(&update.prefix, update.route.epoch)`.
Serve a subtree on demand with `origin.dynamic(prefix, route)`.
- **Tracks.** `with_latency_max` / `latency_max` is `with_max_age` / `max_age`.
`write_datagram(Datagram)` is `insert_datagram(sequence, timestamp, payload)`
Expand Down
35 changes: 35 additions & 0 deletions js/net/bench/broadcasts.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
/** Sweep observers against routes for one route update, and against tracks for one demand edge. */

import { Producer as BroadcastProducer } from "../src/broadcast.ts";
import * as Epoch from "../src/epoch.ts";
import { Producer } from "../src/origin.ts";
import * as Path from "../src/path.ts";

Expand Down Expand Up @@ -85,3 +86,37 @@ for (const trackCount of [8, 32, 128]) {
broadcast.close();
}
}

// Resolve one touched path while both the table and that path's readers grow.
console.log("request,routes,readers,update_and_read_us");
const epoch = Epoch.parse("01900000-0000-7000-8000-000000000001");
for (const pinned of [false, true]) {
for (const routeCount of routeCounts) {
for (const readerCount of observerCounts) {
const origin = new Producer();
const reader = origin.consume();
const options = pinned ? { epoch } : {};
const publishers = Array.from({ length: routeCount }, (_, index) => {
const source = origin.createBroadcast(Path.from(`room/${index}`));
source.announce({ epoch });
return source;
});
const background = publishers.map((_, index) => reader.request(Path.from(`room/${index}`), options));
const requests = Array.from({ length: readerCount }, () => reader.request(Path.from("room/0"), options));
const start = performance.now();
for (let index = 0; index < updates; index++) {
publishers[0].announce({ epoch, cost: BigInt(index + 1) });
await Promise.resolve();
for (const request of requests) {
if (!request.active.peek()) throw new Error("same-epoch request lost its front");
checksum++;
}
}
console.log(
`${pinned ? "pinned" : "unpinned"},${routeCount},${readerCount},${(((performance.now() - start) * 1000) / updates).toFixed(1)}`,
);
for (const request of [...requests, ...background]) request.close();
origin.close();
}
}
}
20 changes: 19 additions & 1 deletion js/net/src/broadcast.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
* @module
*/
import { type Dispose, type GetPromise, type Getter, Once, Signal } from "@moq/signals";
import type * as Epoch from "./epoch.ts";
import { NotFound } from "./error.ts";
import type { Consumer as GroupConsumer } from "./group.ts";
import { Route } from "./hop.ts";
Expand Down Expand Up @@ -415,6 +416,7 @@ export class Producer {
interface Shared {
state: BroadcastState;
path: Path.Valid;
epoch?: Epoch.Valid;
}

// Constructs a Consumer from within this module without exposing a public constructor
Expand All @@ -432,13 +434,15 @@ let makeConsumer: (shared: Shared) => Consumer;
export class Consumer {
#state: BroadcastState;
#path: Path.Valid;
#epoch?: Epoch.Valid;

// Guards against a double close() on this handle over-decrementing the consumer count.
#closed = false;

protected constructor(shared?: never);
protected constructor(shared?: Shared) {
this.#path = shared?.path ?? Path.empty();
this.#epoch = shared?.epoch;
if (shared) {
this.#state = shared.state;
} else {
Expand All @@ -461,6 +465,9 @@ export class Consumer {
if (target instanceof Consumer) target.#path = path;
else stampProducer(target, path);
};
hooks.stampEpoch = (target, epoch) => {
target.#epoch = epoch;
};
}

/**
Expand All @@ -475,6 +482,17 @@ export class Consumer {
return this.#path;
}

/**
* The publisher epoch of the route an origin resolved this handle through, if it has one.
*
* An origin stamps it with each request's result, so it names the publisher
* instance actually serving this handle even when the request named none. `undefined` for a
* route without an epoch or a handle not handed out by an origin.
*/
get epoch(): Epoch.Valid | undefined {
return this.#epoch;
}

/**
* Settles with `null` once the broadcast closes; a broadcast end carries no cause.
* Peek it synchronously (`undefined` while open), observe it reactively, or `await` it.
Expand All @@ -501,7 +519,7 @@ export class Consumer {
// Hand this consumer's backing state and path to a clone. Opaque (`never`) so the state type
// stays unexported; a subclass passes it straight back into its own `super(...)`.
protected shareState(): never {
return { state: this.#state, path: this.#path } satisfies Shared as never;
return { state: this.#state, path: this.#path, epoch: this.#epoch } satisfies Shared as never;
}

/** Get a lazy handle for a track on this broadcast. Repeat subscriptions dedupe onto one upstream subscription. */
Expand Down
6 changes: 6 additions & 0 deletions js/net/src/internal.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
*/
import type { Dispose, Getter } from "@moq/signals";
import type { Consumer as BroadcastConsumer, Producer as BroadcastProducer } from "./broadcast.ts";
import type * as Epoch from "./epoch.ts";
import type { Frame, Consumer as GroupConsumer, Producer as GroupProducer } from "./group.ts";
import type { Route } from "./hop.ts";
import * as Path from "./path.ts";
Expand Down Expand Up @@ -186,6 +187,8 @@ export const hooks: {
) => void;
/** Name a broadcast handle by the path an origin created or resolved it at. */
stampPath: (target: BroadcastProducer | BroadcastConsumer, path: Path.Valid) => void;
/** Name the epoch of the route an origin resolved a broadcast handle through. */
stampEpoch: (target: BroadcastConsumer, epoch: Epoch.Valid | undefined) => void;
} = {
makeRequest: () => {
throw new Error("track.ts not loaded");
Expand Down Expand Up @@ -232,6 +235,9 @@ export const hooks: {
stampPath: () => {
throw new Error("broadcast.ts not loaded");
},
stampEpoch: () => {
throw new Error("broadcast.ts not loaded");
},
};

/**
Expand Down
64 changes: 64 additions & 0 deletions js/net/src/origin.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1885,3 +1885,67 @@ test("a peer naming another epoch is refused rather than served a different inst
local.close();
origin.close();
});

test("request pins its epoch while an unpinned sibling follows the newest publisher", () => {
const origin = new Producer();
const path = Path.from("room");
const first = origin.createBroadcast(path);
first.announce({ epoch: EPOCH });
const reader = origin.consume();
const pinned = reader.request(path, { epoch: EPOCH });
const unpinned = reader.request(path);
expect(pinned.active.peek()?.epoch).toBe(EPOCH);
expect(unpinned.active.peek()?.epoch).toBe(EPOCH);
const newerEpoch = Epoch.parse("01900000-0000-7000-8000-000000000002");
const second = origin.createBroadcast(path);
second.announce({ epoch: newerEpoch });
expect(pinned.active.peek()).toBeUndefined();
expect(pinned.unroutable.peek()).toBe(true);
// The unpinned handle names the publisher it actually resolved.
expect(unpinned.active.peek()?.epoch).toBe(newerEpoch);
const stale = reader.request(path, { epoch: EPOCH });
expect(stale.active.peek()).toBeUndefined();
expect(stale.unroutable.peek()).toBe(true);
stale.close();
pinned.close();
unpinned.close();
origin.close();
});

test("request never accepts an old handler after its epoch changes asynchronously", async () => {
const origin = new Producer();
const prefix = Path.from("room");
const path = Path.from("room/cam");
const dynamic = origin.dynamic(prefix, { epoch: EPOCH });
const pending = origin.consume().request(path, { epoch: EPOCH });
const requests = dynamic.requested();
const old = (await requests.next()).value;
const epoch = Epoch.parse("01900000-0000-7000-8000-000000000002");
dynamic.update({ ...dynamic.route, epoch });
const current = origin.consume().request(path, { epoch });
old?.accept(new BroadcastProducer().consume());
expect(pending.active.peek()).toBeUndefined();
expect(pending.unroutable.peek()).toBe(true);
expect(current.active.peek()).toBeUndefined();
const next = (await requests.next()).value;
next?.accept(new BroadcastProducer().consume());
expect(current.active.peek()?.epoch).toBe(epoch);
expect(pending.active.peek()).toBeUndefined();
await requests.return?.();
pending.close();
current.close();
dynamic.close();
origin.close();
});

test("a route without an epoch resolves a handle without one", () => {
const origin = new Producer();
const path = Path.from("room");
const broadcast = origin.createBroadcast(path);
broadcast.announce();
const request = origin.consume().request(path);
expect(request.active.peek()).toBeDefined();
expect(request.active.peek()?.epoch).toBeUndefined();
request.close();
origin.close();
});
Loading
Loading