Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion doc/lib/js/publish.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,10 @@ for a permission, device, or settings change instead of prompting again.
The video encoder follows its share of the connection's send-rate estimate, so
several publishers on one connection do not each target the whole uplink.
Codec, resolution, framerate, and bitrate are tunable through
`el.video.config`, and `el.video.cut()` asks for an extra keyframe. Size,
`el.video.config`, and `el.video.cut()` asks for an extra keyframe. A viewer
who joins once a group's deltas outweigh twice its keyframe gets a fresh one,
so `keyframeInterval` is only the longest a group runs. Through a moq-lite
relay, only the join that opens the relay's upstream subscription counts. Size,
bitrate, and framerate change in place, while a codec or audio format change
keeps the rendition's catalog ID under a fresh `track` name that viewers
follow through the catalog. For
Expand Down
8 changes: 7 additions & 1 deletion doc/lib/rs/moq-video.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,13 @@ hardware-only, AV1 decodes through NVDEC and MediaCodec, and VP8 and VP9
decode in software through the opt-in `vpx` feature.

Capture is pull-driven: `encode::publish_capture` advertises the track up
front and opens the camera only while someone subscribes. Backends that can
front and opens the camera only while someone subscribes. A viewer who joins
once the current group's deltas outweigh twice its keyframe gets a fresh
keyframe, so the GOP is only the longest a group runs. Through a moq-lite
relay, only the join that opens the relay's upstream subscription counts. A
build with the `v4l2` feature keeps every `Auto` or `Hardware` selection to its
GOP, whichever encoder it lands on, since a V4L2 driver may refuse the request.
Backends that can
change bitrate live follow the connection's send estimate. Where the platform
allows it, frames stay on the GPU between capture, decode, encode, and render.

Expand Down
7 changes: 7 additions & 0 deletions js/publish/MIGRATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,13 @@ How to update your code for each breaking change, newest first.

## Unreleased

### `Encoder.cut()` is not rate limited, and video tracks advertise refresh (#6048)

`Encoder.cut()` no longer spaces keyframes 500 ms apart: every call forces one on the next frame, so frequent calls produce all-keyframe video.
Debounce your own calls if you relied on the spacing.
Every `Broadcast.video()` track now advertises a refresh ratio of 2, so a producer registered there must open a group with a keyframe whenever `track.demand().wanted` passes the largest group.
The bundled `Encoder` already does; a custom producer has to as well.

### A disabled rendition stays in the catalog (#4915)

`enabled: false` keeps the rendition in the catalog with `enabled: false` instead of removing it, and viewers skip it.
Expand Down
14 changes: 12 additions & 2 deletions js/publish/src/broadcast.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@ import { errors } from "./internal";
import { Baseline } from "./jitter";
import { type Kind, Rendition } from "./rendition";

// The refresh ratio video tracks advertise: a joiner forces a keyframe once a group's deltas
// outweigh twice its keyframe. Matches moq-video's capture `REFRESH`; see it for the measurements.
const VIDEO_REFRESH = 2;

// Signals the broadcast reads. Whoever owns the backing Signal (the element, or another component
// whose output is wired in, e.g. a Video.Capture's `display`) does the writing.
export type BroadcastInput = {
Expand Down Expand Up @@ -133,7 +137,12 @@ export class Broadcast {
this.#signals.run(this.#run.bind(this));
}

/** Register a video rendition under a full track name (e.g. `"video/hd"`). Throws if the name is taken. */
/**
* Register a video rendition under a full track name (e.g. `"video/hd"`). Throws if the name is taken.
*
* Its track advertises a refresh ratio, so its producer opens a group with a keyframe whenever
* `track.demand().wanted` passes the largest group, as the video `Encoder` does.
*/
video(name: string): Rendition<Catalog.VideoConfig> {
return this.#register<Catalog.VideoConfig>(name, "video");
}
Expand Down Expand Up @@ -413,9 +422,10 @@ export class Broadcast {
continue;
}
if (!entry) {
const info = Container.trackInfo({ maxAge, priority: Catalog.PRIORITY[rendition.kind] });
const track = broadcast.createTrack(
name,
Container.trackInfo({ maxAge, priority: Catalog.PRIORITY[rendition.kind] }),
rendition.kind === "video" ? { ...info, refresh: VIDEO_REFRESH } : info,
);
entry = { track, owner: rendition };
tracks.set(rendition.name, entry);
Expand Down
113 changes: 78 additions & 35 deletions js/publish/src/video/encoder.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -849,8 +849,30 @@ test("a re-probe swaps the catalog entry in one update", async () => {
}
});

test("cut forces a keyframe, coalescing requests and spacing them at least 500ms apart", async () => {
// Drive a live encoder over a recording `VideoEncoder` that outputs every frame it is handed, so the
// track holds real groups: a keyframe of 100 bytes, each delta 60. `run` sends one frame per call
// at `millis` of media time and reads back which frames were forced to be keyframes.
async function recordKeyframes(
info: Partial<Moq.Track.Info>,
run: (ctx: {
encoder: Encoder;
track: Moq.Track.Producer;
send: (millis: number) => Promise<void>;
}) => Promise<void>,
): Promise<number[]> {
const keys: number[] = [];

class Frame {
readonly timestamp: number;
constructor(timestamp: number) {
this.timestamp = timestamp;
}
clone(): Frame {
return new Frame(this.timestamp);
}
close(): void {}
}

class RecordingVideoEncoder {
state: CodecState = "unconfigured";
#output: VideoEncoderInit["output"];
Expand All @@ -869,9 +891,13 @@ test("cut forces a keyframe, coalescing requests and spacing them at least 500ms
this.#codec = config.codec;
}

// Records the test's frames only, not the probe's.
// Records the test's frames only, not the probe's, and outputs each a task later as WebCodecs does.
encode(frame: VideoFrame, options?: VideoEncoderEncodeOptions): void {
if (options?.keyFrame && frame instanceof Frame) keys.push(frame.timestamp / 1000);
if (!(frame instanceof Frame)) return;
const key = !!options?.keyFrame;
if (key) keys.push(frame.timestamp / 1000);
const chunk = { type: key ? "key" : "delta", timestamp: frame.timestamp, byteLength: key ? 100 : 60 };
queueMicrotask(() => this.#output({ ...chunk, copyTo: () => {} } as never, {} as never));
}

// Only the probe flushes; report the configured codec, as Chrome does.
Expand All @@ -894,17 +920,6 @@ test("cut forces a keyframe, coalescing requests and spacing them at least 500ms
const videoFrame = Object.getOwnPropertyDescriptor(globalThis, "VideoFrame");
Object.defineProperty(globalThis, "VideoFrame", { configurable: true, value: FakeVideoFrame, writable: true });

class Frame {
readonly timestamp: number;
constructor(timestamp: number) {
this.timestamp = timestamp;
}
clone(): Frame {
return new Frame(this.timestamp);
}
close(): void {}
}

const { Fanout } = await import("../fanout");
let controller!: ReadableStreamDefaultController<VideoFrame>;
const fanout = new Fanout(
Expand All @@ -916,7 +931,7 @@ test("cut forces a keyframe, coalescing requests and spacing them at least 500ms
{ clone: (frame) => frame.clone(), release: (frame) => frame.close() },
);

const track = new Moq.Track.Producer("video").accept({ timescale: Moq.Time.Timescale.MILLI });
const track = new Moq.Track.Producer("video").accept({ timescale: Moq.Time.Timescale.MICRO, ...info });
const rendition = {
config: new Signal(undefined),
track: new Signal<Moq.Track.Producer | undefined>(track),
Expand All @@ -928,35 +943,19 @@ test("cut forces a keyframe, coalescing requests and spacing them at least 500ms
};
const encoder = new Encoder("video", {
enabled: true,
broadcast: { video: () => rendition } as never,
broadcast: { video: () => rendition, baseline: new Baseline() } as never,
capture: capture as never,
});

// One frame per 100ms of media time, after calling cut() when `cut` says so.
const send = async (millis: number, cut = false) => {
if (cut) encoder.cut();
const send = async (millis: number) => {
controller.enqueue(new Frame(millis * 1000) as unknown as VideoFrame);
await new Promise((resolve) => setTimeout(resolve, 5));
};

try {
await settle();

// The opening keyframe serves a request made before it.
await send(0, true);
await send(100);
// Several requests before one frame produce one keyframe.
encoder.cut();
encoder.cut();
await send(600, true);
await send(700);
// Too soon after the last: deferred until 500ms have passed, not dropped.
await send(800, true);
await send(1000);
await send(1100);
await send(1200);

expect(keys).toEqual([0, 600, 1100]);
await run({ encoder, track, send });
return keys;
} finally {
encoder.close();
fanout.close();
Expand All @@ -966,6 +965,50 @@ test("cut forces a keyframe, coalescing requests and spacing them at least 500ms
if (videoFrame) Object.defineProperty(globalThis, "VideoFrame", videoFrame);
else Reflect.deleteProperty(globalThis, "VideoFrame");
}
}

test("cut forces a keyframe on the next frame, coalescing requests without spacing them", async () => {
const keys = await recordKeyframes({}, async ({ encoder, send }) => {
// The opening keyframe serves a request made before it.
encoder.cut();
await send(0);
await send(100);
// Several requests before one frame produce one keyframe.
encoder.cut();
encoder.cut();
await send(200);
// The very next frame can be cut again.
encoder.cut();
await send(300);
await send(400);
});
expect(keys).toEqual([0, 200, 300]);
});

test("a join over the refresh ratio forces a keyframe on the next frame", async () => {
const keys = await recordKeyframes({ refresh: 2 }, async ({ track, send }) => {
const watching = track.subscribe();
const joined: Moq.Track.Subscriber[] = [];
try {
await send(0);
await send(50);
// One 60 byte delta is under twice the 100 byte keyframe.
joined.push(track.subscribe());
await send(100);
await send(150);
await send(200);
// Four deltas are over it, 200ms after the keyframe.
joined.push(track.subscribe());
await send(250);
// A join after the fresh keyframe reads its group.
joined.push(track.subscribe());
await send(300);
} finally {
watching.close();
for (const subscriber of joined) subscriber.close();
}
});
expect(keys).toEqual([0, 250]);
});

test("a failed video encoder disables only its rendition and recovers on a fresh track", async () => {
Expand Down
36 changes: 24 additions & 12 deletions js/publish/src/video/encoder.ts
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,8 @@ export interface Config {
// When combined with maxPixels, the smaller resulting cap wins.
maxScale?: number;

// The interval at which to insert keyframes. (default: 2000 milliseconds)
// The longest interval between keyframes. A joining subscriber forces one sooner once the
// group's deltas outweigh the track's refresh ratio. (default: 2000 milliseconds)
keyframeInterval?: Milli;

// If not provided, the encoder will use the best bitrate for the given width, height, and framerate.
Expand Down Expand Up @@ -368,9 +369,17 @@ export class Encoder {
if (!capture) return;

const producer = new Container.Legacy.Producer(track, new Container.Legacy.Format("video"));
const demand = track.demand();

let lastKeyframe: Micro | undefined;
let lastEncoded: Micro | undefined;
// The largest group demand asked for that a keyframe already answers. A join over the refresh
// ratio asks for one past the largest, which the next frame opens.
let answered: number | undefined;
const answer = () => {
const wanted = demand.wanted.peek();
if (wanted !== undefined) answered = Math.max(answered ?? 0, wanted);
};
let tearingDown = false;

effect.spawn(async () => {
Expand All @@ -388,6 +397,8 @@ export class Encoder {
}));

producer.encode(frame, frame.timestamp as Micro, key);
// Including a keyframe the encoder placed on its own, which opened a group too.
if (key) answer();
if (this.#estimator.flush(frame.timestamp, baseline)) {
this.#estimate.set({ ...this.#estimator.estimate });
}
Expand Down Expand Up @@ -458,19 +469,23 @@ export class Encoder {

const interval = config?.keyframeInterval ?? Milli.fromSecond(2 as Second);

// Force a keyframe if this is the first frame (no group yet), the GOP elapsed, or
// the caller asked for one and the last is old enough.
// Force a keyframe if this is the first frame (no group yet), the GOP elapsed, the
// caller asked for one, or a join wants a group past the largest. No spacing: the
// refresh ratio already rate-limits joins by the bytes a group holds.
const since = lastKeyframe === undefined ? undefined : frame.timestamp - lastKeyframe;
const wanted = demand.wanted.peek();
const keyFrame =
resized ||
since === undefined ||
since >= Micro.fromMilli(interval) ||
(this.#cut && since >= Micro.fromMilli(MIN_CUT_INTERVAL));
this.#cut ||
(wanted !== undefined && (answered === undefined || wanted > answered));
if (keyFrame) {
lastKeyframe = frame.timestamp as Micro;
resized = false;
// Any keyframe serves an outstanding request.
// Any keyframe serves an outstanding request and every join before it.
this.#cut = false;
answer();
}

encoder.encode(frame, { keyFrame });
Expand Down Expand Up @@ -804,9 +819,10 @@ export class Encoder {
* Request a keyframe, opening a new group at a frame no earlier than this call.
*
* For a resume, a recording cut, or a known tune-in moment; {@link Config.keyframeInterval} is the
* cadence. Requests coalesce into the next keyframe, and forced keyframes land at least 500ms
* apart so a caller in a loop cannot pin the encoder at all-keyframe. A request while not encoding
* is served by the keyframe every encode starts with.
* longest a group runs, and a joining subscriber already opens one through the track's refresh
* ratio. Requests between two frames coalesce into one keyframe on the next, with no spacing: a
* caller in a loop pins the encoder at all-keyframe. A request while not encoding is served by the
Comment thread
kixelated marked this conversation as resolved.
Resolved
* keyframe every encode starts with.
*/
cut(): void {
this.#cut = true;
Expand All @@ -817,10 +833,6 @@ export class Encoder {
}
}

// The closest two requested keyframes may land. A keyframe costs several times a predicted frame,
// and this stays well under the default two-second GOP so a request still beats the cadence.
const MIN_CUT_INTERVAL = 500 as Milli;

// The source's nominal frame rate: what the capture device settled on, or what a frame stream
// declared. Undefined when nothing reports one.
function sourceFrameRate(source: Source): number | undefined {
Expand Down
1 change: 0 additions & 1 deletion quest/m1/refresh/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,6 @@ Work owned by this README, after the children land:

- [Lite wire](/quest/m1/refresh/lite.md) - lite-07 TRACK_INFO advertises the ratio and SUBSCRIBE_UPDATE carries the wanted group, so a join at any relay reaches the origin
- [moq-json](/quest/m1/refresh/json.md) - snapshot and window groups roll on a qualifying join, reading the ratio from the track; `delta_ratio` and `op_ratio` become `deltas` and `ops` switches in every binding
- [Video](/quest/m1/refresh/video.md) - video publishers advertise refresh and force a keyframe on demand, keeping the encoder's max GOP

## Related

Expand Down
44 changes: 0 additions & 44 deletions quest/m1/refresh/video.md

This file was deleted.

6 changes: 6 additions & 0 deletions rs/moq-video/MIGRATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,12 @@ How to update your code for each breaking change, newest first.

## Unreleased

### `Control::cut` is not rate limited, and capture advertises refresh (#6048)

`encode::Control::cut` no longer spaces keyframes 500 ms apart: every request forces one on the next frame, so frequent calls produce all-keyframe video.
Debounce your own calls if you relied on the spacing.
Capture tracks now advertise `refresh: Some(2)` when every backend the selection may open honors a cut, and the driver forces a keyframe when `Demand::wanted` passes the largest group.

### OpenH264 is opt-in (#5771)

The default features drop `openh264`, so a default build has no software H.264 encoder or decoder and a GPU-less host fails to open one.
Expand Down
Loading