From 498ea4db7dae9df10149d864a0985d8e595e8cb9 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 13:54:24 -0800 Subject: [PATCH 1/5] chore(quest): claim quest/m1/webgpu-safari.md Quest-Claim: 8a1a9342701afc0d From 208fa97f37a197bd8b3ad94f1c65c1261d90d231 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 14:14:29 -0800 Subject: [PATCH 2/5] docs(quest): record Safari verification blockers Co-Authored-By: GPT-6.1 Sol --- quest/m1/webgpu-safari.md | 64 ++++++++++++++++++++++++++++++++++----- 1 file changed, 56 insertions(+), 8 deletions(-) diff --git a/quest/m1/webgpu-safari.md b/quest/m1/webgpu-safari.md index 6a26683e024..5f36b7d0caa 100644 --- a/quest/m1/webgpu-safari.md +++ b/quest/m1/webgpu-safari.md @@ -9,24 +9,72 @@ evidence. ## Plan -Planned 2026-10-10 as a follow-up of #5138, which ships the renderer -untested on Safari. Known Safari 26 issues with `importExternalTexture`: H.264 -frames, biplanar formats, and iOS orientation. +Planned 2026-10-10 as a follow-up of +[moq-dev/moq#5138](https://github.com/moq-dev/moq/pull/5138), which shipped the +renderer without Safari verification. The original concerns were H.264 +frames, biplanar formats, and iOS orientation with `importExternalTexture`. - Run `` and the `` preview with `backend` set to `auto` and `webgpu` on real Apple hardware or a hosted Safari session, over H.264 and the other catalog codecs, in each orientation. -- #5138 decided an unimportable frame stays loud (it throws and that frame is +- [moq-dev/moq#5138](https://github.com/moq-dev/moq/pull/5138) decided an + unimportable frame stays loud (it throws and that frame is skipped). Revisit with what Safari actually does: keep it, or report it - through `out.error` under its own error name. + through `out.error` under its own error name. No valid macOS frame failed to + import in the run below, so this decision remains unchanged. - Fix what breaks at its source. If the probe passes but frames fail, consider - extending the probe rather than a runtime fallback, which #5138 rejected. + extending the probe rather than a runtime fallback, which that PR rejected. -Needs a human-run or hosted Safari session; a background agent reports -instead of guessing. +Verification on 2026-11-03 used physical Apple M4 hardware (Mac16,13), macOS +26.6.2 build 25G83, and Safari 26.6.2 through its existing WebDriver setup. +The source was `976fece17349666bb2ab6dad93109408ac272e8d`. The adapter reported +`vendor: apple` and `isFallbackAdapter: false`; neither was overridden. +Pixel readback checked four colored quadrants, with no renderer changes. + +- Direct WebGPU rendering passed 80/80 cases: RGBA, NV12, I420, H.264 High, + Main and Baseline, HEVC, VP8, VP9 and AV1, each rotated 0/90/180/270 degrees + and mirrored or unmirrored. Decoded frames were NV12. VP9 and AV1 used + independent 640x360 FFmpeg fixtures; Safari's own encoders failed, and its + decoders also refused the earlier 64x32 fixtures. +- `` passed 112/112 cases: those seven encoded codec/profile + variants, both `auto` and `webgpu`, all four catalog rotations and both + mirror settings. Broadcasts used a page-local origin, so this is rendering + evidence, not transport interoperability evidence. +- `` passed 10/12 landscape image-source preview cases, covering + `source` and `encoded` with both backends for H.264 High, HEVC and VP8. Both + HEVC `encoded` cases failed before rendering with + `preview: decode error EncodingError: Decoder failure`. +- The HEVC reproduction requested `hevc: { format: "annexb" }`, but Safari + returned a length-prefixed chunk (prefix `00 00 00 3b 4e 01`) and an hvcC + decoder description. `preview.ts` configures its decoder without that + description. The publishing probe also assumes the requested Annex B + format was honored, so a preview-only decoder change would leave the + publishing path unresolved. + +Remaining evidence includes real iOS capture and physical orientation, +publisher preview orientation, and a valid frame that actually fails import. +A deliberately closed RGBA frame did not induce an import exception or +validation error in this Safari session; it does not verify failure handling. +Keep the renderer's existing error behavior pending such evidence. Public API: possibly a new `out.error` name. Wire: none. +## Waiting + +- Maintainer: connect real iOS hardware running Safari 26 to an interactive + or hosted session. The available device tools expose only a Linux host and + no iOS devices; SSH/WebDriver supplied macOS evidence only. Recommend a + physical iPhone or iPad session that permits capture and rotation. A + simulator does not clear this wait. Resume the remaining matrix once that + session is reachable. +- Maintainer: decide how to handle Safari ignoring HEVC Annex B encoding. + Recommend verifying the output format in the existing encoder setup probe + and refusing HEVC when unsupported, covering both publishing and encoded + preview. The alternative is supporting Safari's length-prefixed output + while preserving the publisher's inline parameter sets and adaptive + resizing. This changes supported publishing behavior, so the verification + attempt leaves the choice open rather than adding a runtime fallback. + ## Related - [WebGPU HDR](/quest/m2/webgpu-hdr.md) - builds on the same renderer From 389c01685427c225ce760b940458b70d88682e57 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 17:14:31 -0800 Subject: [PATCH 3/5] fix(publish): refuse H.264 or HEVC encoders that ignore Annex B The encoder setup probe now checks the first chunk for an Annex B start code when it requested Annex B. Safari 26 accepts `hevc: { format: "annexb" }` but emits length-prefixed NAL units with an hvcC description, which no subscriber decodes, so publishing moves on to the next codec instead. No runtime fallback. Co-Authored-By: Claude Opus 5.5 --- js/publish/src/video/capture.test.ts | 7 +- js/publish/src/video/encoder-stall.test.ts | 5 +- js/publish/src/video/encoder.test.ts | 75 ++++++++++++++++++++-- js/publish/src/video/encoder.ts | 39 +++++++++-- 4 files changed, 112 insertions(+), 14 deletions(-) diff --git a/js/publish/src/video/capture.test.ts b/js/publish/src/video/capture.test.ts index 002522bb18b..deca0d179ef 100644 --- a/js/publish/src/video/capture.test.ts +++ b/js/publish/src/video/capture.test.ts @@ -5,6 +5,9 @@ import { Effect, Signal } from "@moq/signals"; import { Order } from "../order"; import type { StreamTrack } from "./types"; +// A probe chunk's bytes: an Annex B start code, which the probe requires of H.264 and HEVC. +const ANNEX_B = { byteLength: 4, copyTo: (dest: Uint8Array) => dest.set([0, 0, 0, 1]) }; + mock.module("./capture-worker.ts?worklet", () => ({ default: async () => "blob:fake-worker" })); const { Capture } = await import("./capture"); const { Encoder } = await import("./encoder"); @@ -438,7 +441,7 @@ test("an encoder resuming on a still source encodes a keyframe of the current pi // Only the probe flushes; report the configured codec, as Chrome does. async flush(): Promise { - const chunk = { type: "key", timestamp: 0, byteLength: 1, copyTo: () => {} }; + const chunk = { type: "key", timestamp: 0, ...ANNEX_B }; this.#output(chunk as never, { decoderConfig: { codec: this.#codec } } as never); } @@ -558,7 +561,7 @@ test("an encoder torn down while a frame is in flight closes that frame", async // Only the probe flushes; report the configured codec, as Chrome does. async flush(): Promise { - const chunk = { type: "key", timestamp: 0, byteLength: 1, copyTo: () => {} }; + const chunk = { type: "key", timestamp: 0, ...ANNEX_B }; this.#output(chunk as never, { decoderConfig: { codec: this.#codec } } as never); } diff --git a/js/publish/src/video/encoder-stall.test.ts b/js/publish/src/video/encoder-stall.test.ts index d1f9eee1566..59daefd620c 100644 --- a/js/publish/src/video/encoder-stall.test.ts +++ b/js/publish/src/video/encoder-stall.test.ts @@ -7,6 +7,9 @@ import { Fanout } from "../fanout"; import { Estimator, Order } from "../order"; import { Encoder } from "./encoder"; +// A probe chunk's bytes: an Annex B start code, which the probe requires of H.264 and HEVC. +const ANNEX_B = { byteLength: 4, copyTo: (dest: Uint8Array) => dest.set([0, 0, 0, 1]) }; + // Model an encoder that cannot accept more work until the test completes its pending input. // The codec probe completes immediately, independently of the live encoder's service rate. class SlowEncoder { @@ -55,7 +58,7 @@ class SlowEncoder { async flush() { this.output( - { type: "key", timestamp: 0, byteLength: 1, copyTo: () => {} } as never, + { type: "key", timestamp: 0, ...ANNEX_B } as never, { decoderConfig: { codec: this.codec } } as never, ); } diff --git a/js/publish/src/video/encoder.test.ts b/js/publish/src/video/encoder.test.ts index 27e472fe206..039a4111e08 100644 --- a/js/publish/src/video/encoder.test.ts +++ b/js/publish/src/video/encoder.test.ts @@ -10,6 +10,9 @@ import { Estimator, Order } from "../order"; import { Slot } from "../publication"; import { Encoder } from "./encoder"; +// A probe chunk's bytes: an Annex B start code, which the probe requires of H.264 and HEVC. +const ANNEX_B = { byteLength: 4, copyTo: (dest: Uint8Array) => dest.set([0, 0, 0, 1]) }; + // How long a fake hardware probe takes on the installed clock. const PROBE = Nano.fromMilli(Milli(5)); @@ -37,6 +40,10 @@ class FakeVideoEncoder { // software H.264 does for High and Main. static reorders: (codec: string) => boolean = () => false; + // Whether the encoder ignores a configured codec's requested Annex B format and emits + // length-prefixed NAL units instead, as Safari 26's HEVC encoder does. + static lengthPrefixed: (codec: string) => boolean = () => false; + state: CodecState = "unconfigured"; #output: VideoEncoderInit["output"]; #codec?: string; @@ -74,7 +81,16 @@ class FakeVideoEncoder { // Only the probe flushes, so only it sees an output; the live encoder holds every frame. async flush(): Promise { const codec = this.#codec && FakeVideoEncoder.report(this.#codec); - const chunk = { type: "key", timestamp: 0, byteLength: 1, copyTo: () => {} }; + // A slice NAL unit behind an Annex B start code, or behind Safari's four-byte length. + const data = new Uint8Array( + this.#codec && FakeVideoEncoder.lengthPrefixed(this.#codec) ? [0, 0, 0, 2, 0x26, 0x01] : [0, 0, 0, 1, 0x26, 0x01], + ); + const chunk = { + type: "key", + timestamp: 0, + byteLength: data.byteLength, + copyTo: (dest: Uint8Array) => dest.set(data), + }; this.#output(chunk as never, (codec ? { decoderConfig: { codec } } : {}) as never); if (this.#codec && FakeVideoEncoder.reorders(this.#codec)) { for (const timestamp of [66_666, 33_333]) @@ -107,6 +123,7 @@ function installFakeVideoEncoder() { [Symbol.dispose]() { FakeVideoEncoder.report = (codec) => codec; FakeVideoEncoder.reorders = () => false; + FakeVideoEncoder.lengthPrefixed = () => false; if (original) { Object.defineProperty(globalThis, "VideoEncoder", original); } else { @@ -438,6 +455,56 @@ test("an encoder that reorders frames is skipped for the next codec", async () = } }); +test("an encoder that ignores the requested Annex B format is skipped for the next codec", async () => { + using _videoEncoder = installFakeVideoEncoder(); + // Safari 26: HEVC is supported, but its chunks come out length-prefixed. H.264 honors Annex B. + FakeVideoEncoder.lengthPrefixed = (codec) => codec.startsWith("hev1"); + const probe = spyOn(FakeVideoEncoder, "isConfigSupported").mockImplementation(async (config) => ({ + supported: config.codec.startsWith("hev1") || config.codec === "vp8", + })); + const warn = spyOn(console, "warn").mockImplementation(() => {}); + const capture = { + in: { source: new Signal({ getSettings: () => ({ frameRate: 30 }), getConstraints: () => ({}) }) }, + out: { display: new Signal({ width: 1280, height: 720 }) }, + }; + const encoder = new Encoder("video", { enabled: true, capture: capture as never }); + try { + await settle(); + expect(probe.mock.calls.some(([config]) => config.codec.startsWith("hev1"))).toBe(true); + expect(encoder.out.resolved.peek()?.codec).toBe("vp8"); + expect(warn).toHaveBeenCalledWith(expect.stringContaining("Annex B")); + } finally { + encoder.close(); + probe.mockRestore(); + warn.mockRestore(); + } +}); + +test("an HEVC-only encoder that ignores Annex B leaves no codec rather than falling back at runtime", async () => { + using _videoEncoder = installFakeVideoEncoder(); + FakeVideoEncoder.lengthPrefixed = (codec) => codec.startsWith("hev1"); + const probe = spyOn(FakeVideoEncoder, "isConfigSupported").mockImplementation(async (config) => ({ + supported: config.codec.startsWith("hev1"), + })); + const warn = spyOn(console, "warn").mockImplementation(() => {}); + const error = spyOn(console, "error").mockImplementation(() => {}); + const capture = { + in: { source: new Signal({ getSettings: () => ({ frameRate: 30 }), getConstraints: () => ({}) }) }, + out: { display: new Signal({ width: 1280, height: 720 }) }, + }; + const encoder = new Encoder("video", { enabled: true, capture: capture as never, config: { codec: "hev1" } }); + try { + await settle(); + expect(encoder.out.resolved.peek()).toBeUndefined(); + expect(encoder.settled.peek()).toBe(true); + } finally { + encoder.close(); + probe.mockRestore(); + warn.mockRestore(); + error.mockRestore(); + } +}); + test("software-only AV1 is refused even when explicitly requested", async () => { using _videoEncoder = installFakeVideoEncoder(); const probe = spyOn(FakeVideoEncoder, "isConfigSupported").mockImplementation(async (config) => ({ @@ -626,7 +693,7 @@ test("disabling encodes one black keyframe and keeps the rendition with enabled: // Emits the newest frame, and reports the configured codec as Chrome does. async flush(): Promise { - const chunk = { type: "key", timestamp: this.#last, byteLength: 1, copyTo: () => {} }; + const chunk = { type: "key", timestamp: this.#last, ...ANNEX_B }; this.#output(chunk as never, { decoderConfig: { codec: this.#codec } } as never); } @@ -986,7 +1053,7 @@ class RecordingVideoEncoder { // Only the probe flushes; report the configured codec, as Chrome does. async flush(): Promise { - const chunk = { type: "key", timestamp: 0, byteLength: 1, copyTo: () => {} }; + const chunk = { type: "key", timestamp: 0, ...ANNEX_B }; this.#output(chunk as never, { decoderConfig: { codec: this.#codec } } as never); } @@ -1475,7 +1542,7 @@ test("a shrinking capture resizes in band below the advertised ceiling", async ( // Only the probe flushes; report the configured codec, as Chrome does. async flush(): Promise { - const chunk = { type: "key", timestamp: 0, byteLength: 1, copyTo: () => {} }; + const chunk = { type: "key", timestamp: 0, ...ANNEX_B }; this.#output(chunk as never, { decoderConfig: { codec: this.#config?.codec } } as never); } diff --git a/js/publish/src/video/encoder.ts b/js/publish/src/video/encoder.ts index fe9acd558de..af079d550ff 100644 --- a/js/publish/src/video/encoder.ts +++ b/js/publish/src/video/encoder.ts @@ -875,6 +875,8 @@ export class Encoder { bitrate: ceiling(codec, dimensions, target), latencyMode: "realtime", hardwareAcceleration, + // Annex B keeps the parameter sets in band, so the catalog carries no description and a + // keyframe can change the size. The probe refuses an encoder that ignores this. avc: codec.startsWith("avc1") ? { format: "annexb" } : undefined, // @ts-expect-error TypeScript's DOM lib lacks the HEVC codec registration's `hevc` config. hevc: codec.startsWith("hev1") ? { format: "annexb" } : undefined, @@ -883,9 +885,9 @@ export class Encoder { const { supported } = await VideoEncoder.isConfigSupported(config); if (!supported) continue; - const reported = await reportedCodec(config); - if (reported !== undefined) return { config, reported }; - console.warn(`moq-publish: skipping ${codec}: its encoder reorders frames despite realtime latency`); + const probed = await probeEncoder(config); + if ("reported" in probed) return { config, reported: probed.reported }; + console.warn(`moq-publish: skipping ${codec}: ${probed.refused}`); } throw new Error("no supported codec"); @@ -998,17 +1000,25 @@ const PROBE_FRAMES = 4; // claim the first keyframe would contradict. It closes before the config resolves, so it never // overlaps the first real encoder; only a re-probe while serving briefly holds two sessions. // -// Returns undefined when the encoder reorders its output. The catalog declares none, but Firefox's -// software H.264 still emits B-frames for High and Main under realtime latency. -async function reportedCodec(config: VideoEncoderConfig): Promise { +// Refuses an encoder whose output the track can't carry, so the caller moves on to the next codec: +// - Output that reorders. The catalog declares none, but Firefox's software H.264 still emits +// B-frames for High and Main under realtime latency. +// - H.264 or HEVC that isn't Annex B. Safari 26 accepts `hevc: { format: "annexb" }` but emits +// length-prefixed NAL units with an hvcC description, which no subscriber can decode. +async function probeEncoder(config: VideoEncoderConfig): Promise<{ reported: string } | { refused: string }> { let reported: string | undefined; let reordered = false; let last: number | undefined; + let first: Uint8Array | undefined; const encoder = new VideoEncoder({ output: (chunk, metadata) => { reported ??= metadata?.decoderConfig?.codec; if (last !== undefined && chunk.timestamp <= last) reordered = true; last = chunk.timestamp; + if (!first) { + first = new Uint8Array(chunk.byteLength); + chunk.copyTo(first); + } }, // flush() rejects with the same error. error: () => {}, @@ -1040,7 +1050,22 @@ async function reportedCodec(config: VideoEncoderConfig): Promise Date: Tue, 3 Nov 2026 17:25:03 -0800 Subject: [PATCH 4/5] docs(quest): record the Safari HEVC decision and the iOS device steps Co-Authored-By: Claude Opus 5.5 --- js/publish/src/video/encoder.test.ts | 4 ++- quest/m1/webgpu-safari.md | 52 +++++++++++++++++----------- 2 files changed, 34 insertions(+), 22 deletions(-) diff --git a/js/publish/src/video/encoder.test.ts b/js/publish/src/video/encoder.test.ts index 04b0104a0aa..29e169c8769 100644 --- a/js/publish/src/video/encoder.test.ts +++ b/js/publish/src/video/encoder.test.ts @@ -83,7 +83,9 @@ class FakeVideoEncoder { const codec = this.#codec && FakeVideoEncoder.report(this.#codec); // A slice NAL unit behind an Annex B start code, or behind Safari's four-byte length. const data = new Uint8Array( - this.#codec && FakeVideoEncoder.lengthPrefixed(this.#codec) ? [0, 0, 0, 2, 0x26, 0x01] : [0, 0, 0, 1, 0x26, 0x01], + this.#codec && FakeVideoEncoder.lengthPrefixed(this.#codec) + ? [0, 0, 0, 2, 0x26, 0x01] + : [0, 0, 0, 1, 0x26, 0x01], ); const chunk = { type: "key", diff --git a/quest/m1/webgpu-safari.md b/quest/m1/webgpu-safari.md index 5f36b7d0caa..124df479ea0 100644 --- a/quest/m1/webgpu-safari.md +++ b/quest/m1/webgpu-safari.md @@ -43,37 +43,47 @@ Pixel readback checked four colored quadrants, with no renderer changes. - `` passed 10/12 landscape image-source preview cases, covering `source` and `encoded` with both backends for H.264 High, HEVC and VP8. Both HEVC `encoded` cases failed before rendering with - `preview: decode error EncodingError: Decoder failure`. -- The HEVC reproduction requested `hevc: { format: "annexb" }`, but Safari - returned a length-prefixed chunk (prefix `00 00 00 3b 4e 01`) and an hvcC - decoder description. `preview.ts` configures its decoder without that - description. The publishing probe also assumes the requested Annex B - format was honored, so a preview-only decoder change would leave the - publishing path unresolved. + `preview: decode error EncodingError: Decoder failure`: Safari accepts + `hevc: { format: "annexb" }` but returns length-prefixed chunks (prefix + `00 00 00 3b 4e 01`) with a 109-byte hvcC description. + +✅ Safari HEVC: probe and refuse (maintainer, 2026-11-03, on +[moq-dev/moq#7765](https://github.com/moq-dev/moq/pull/7765)). The encoder +setup probe in `@moq/publish` now checks that H.264 and HEVC output opens with +an Annex B start code and otherwise skips that codec, so publishing and the +encoded preview pick the next one. No runtime fallback. On the same Safari +26.6.2 Mac, the probe refused `hev1.1.6.L93.B0` and resolved `avc1.640028`, +whose output was Annex B. Remaining evidence includes real iOS capture and physical orientation, publisher preview orientation, and a valid frame that actually fails import. A deliberately closed RGBA frame did not induce an import exception or -validation error in this Safari session; it does not verify failure handling. +validation error in the macOS session; it does not verify failure handling. Keep the renderer's existing error behavior pending such evidence. Public API: possibly a new `out.error` name. Wire: none. ## Waiting -- Maintainer: connect real iOS hardware running Safari 26 to an interactive - or hosted session. The available device tools expose only a Linux host and - no iOS devices; SSH/WebDriver supplied macOS evidence only. Recommend a - physical iPhone or iPad session that permits capture and rotation. A - simulator does not clear this wait. Resume the remaining matrix once that - session is reachable. -- Maintainer: decide how to handle Safari ignoring HEVC Annex B encoding. - Recommend verifying the output format in the existing encoder setup probe - and refusing HEVC when unsupported, covering both publishing and encoded - preview. The alternative is supporting Safari's length-prefixed output - while preserving the publisher's inline parameter sets and adaptive - resizing. This changes supported publishing behavior, so the verification - attempt leaves the choice open rather than adding a runtime fallback. +- Maintainer: run Safari 26 on a physical iPhone or iPad (a simulator does not + count), with Web Inspector attached from a Mac. Serve the `main` demo + (`just demo`) to the device over HTTPS, since camera capture needs a secure + context. Then: + 1. Open `publish.html`, publish the rear camera, then the front camera. In + the console, run `document.querySelector("moq-publish").backend = "webgpu"`, + then `"auto"`. For each, rotate the device through portrait, landscape + left, landscape right, and upside down. Check that the preview stays + upright and matches the scene. + 2. Open `watch.html` on the same broadcast and set + `document.querySelector("moq-watch").backend` to `"webgpu"`, then `"auto"`. + Repeat the rotations and check that the picture stays upright. + 3. In the console, run `(await navigator.gpu.requestAdapter()).info`. Record + its result, the iOS and Safari versions, any `moq-publish: skipping` + warning, and any `importExternalTexture` or WebGPU validation error in + this quest. + + Done when those results are recorded here, so the remaining work can be + fixed at its source or the quest finished. ## Related From 8d595701ee09969ef3a404f7a2c7e14e1f113f1a Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Tue, 3 Nov 2026 18:22:14 -0800 Subject: [PATCH 5/5] fix(publish): refuse H.264 or HEVC output that carries a description A length-prefixed first NAL of 256-511 bytes starts `00 00 01 xx`, which the start-code check alone read as Annex B. Per the AVC and HEVC WebCodecs registrations, a decoder description means length-prefixed output, so the probe now refuses it too. Safari's Annex B H.264 output carries none. Co-Authored-By: Claude Opus 5.5 --- js/publish/src/video/encoder.test.ts | 58 +++++++++++++++++++++------- js/publish/src/video/encoder.ts | 8 +++- 2 files changed, 50 insertions(+), 16 deletions(-) diff --git a/js/publish/src/video/encoder.test.ts b/js/publish/src/video/encoder.test.ts index 29e169c8769..078ce712714 100644 --- a/js/publish/src/video/encoder.test.ts +++ b/js/publish/src/video/encoder.test.ts @@ -40,9 +40,10 @@ class FakeVideoEncoder { // software H.264 does for High and Main. static reorders: (codec: string) => boolean = () => false; - // Whether the encoder ignores a configured codec's requested Annex B format and emits - // length-prefixed NAL units instead, as Safari 26's HEVC encoder does. - static lengthPrefixed: (codec: string) => boolean = () => false; + // The first NAL unit's size when the encoder ignores a configured codec's requested Annex B + // format and emits length-prefixed NAL units with a description instead, as Safari 26's HEVC + // encoder does. Undefined for Annex B. + static lengthPrefixed: (codec: string) => number | undefined = () => undefined; state: CodecState = "unconfigured"; #output: VideoEncoderInit["output"]; @@ -81,19 +82,21 @@ class FakeVideoEncoder { // Only the probe flushes, so only it sees an output; the live encoder holds every frame. async flush(): Promise { const codec = this.#codec && FakeVideoEncoder.report(this.#codec); - // A slice NAL unit behind an Annex B start code, or behind Safari's four-byte length. - const data = new Uint8Array( - this.#codec && FakeVideoEncoder.lengthPrefixed(this.#codec) - ? [0, 0, 0, 2, 0x26, 0x01] - : [0, 0, 0, 1, 0x26, 0x01], - ); + // A slice NAL unit behind an Annex B start code, or behind Safari's four-byte big-endian length. + const size = this.#codec ? FakeVideoEncoder.lengthPrefixed(this.#codec) : undefined; + const data = size === undefined ? new Uint8Array([0, 0, 0, 1, 0x26, 0x01]) : new Uint8Array(4 + size); + if (size !== undefined) { + new DataView(data.buffer).setUint32(0, size); + data.set([0x26, 0x01], 4); + } + const description = size === undefined ? undefined : new Uint8Array([1]); const chunk = { type: "key", timestamp: 0, byteLength: data.byteLength, copyTo: (dest: Uint8Array) => dest.set(data), }; - this.#output(chunk as never, (codec ? { decoderConfig: { codec } } : {}) as never); + this.#output(chunk as never, (codec ? { decoderConfig: { codec, description } } : {}) as never); if (this.#codec && FakeVideoEncoder.reorders(this.#codec)) { for (const timestamp of [66_666, 33_333]) this.#output({ ...chunk, type: "delta", timestamp } as never, {} as never); @@ -125,7 +128,7 @@ function installFakeVideoEncoder() { [Symbol.dispose]() { FakeVideoEncoder.report = (codec) => codec; FakeVideoEncoder.reorders = () => false; - FakeVideoEncoder.lengthPrefixed = () => false; + FakeVideoEncoder.lengthPrefixed = () => undefined; if (original) { Object.defineProperty(globalThis, "VideoEncoder", original); } else { @@ -459,8 +462,35 @@ test("an encoder that reorders frames is skipped for the next codec", async () = test("an encoder that ignores the requested Annex B format is skipped for the next codec", async () => { using _videoEncoder = installFakeVideoEncoder(); - // Safari 26: HEVC is supported, but its chunks come out length-prefixed. H.264 honors Annex B. - FakeVideoEncoder.lengthPrefixed = (codec) => codec.startsWith("hev1"); + // Safari 26: HEVC is supported, but its chunks come out length-prefixed (a 59-byte first NAL in the + // recorded sample). H.264 honors Annex B. + FakeVideoEncoder.lengthPrefixed = (codec) => (codec.startsWith("hev1") ? 59 : undefined); + const probe = spyOn(FakeVideoEncoder, "isConfigSupported").mockImplementation(async (config) => ({ + supported: config.codec.startsWith("hev1") || config.codec === "vp8", + })); + const warn = spyOn(console, "warn").mockImplementation(() => {}); + const capture = { + in: { source: new Signal({ getSettings: () => ({ frameRate: 30 }), getConstraints: () => ({}) }) }, + out: { display: new Signal({ width: 1280, height: 720 }) }, + }; + const encoder = new Encoder("video", { enabled: true, capture: capture as never }); + try { + await settle(); + expect(probe.mock.calls.some(([config]) => config.codec.startsWith("hev1"))).toBe(true); + expect(encoder.out.resolved.peek()?.codec).toBe("vp8"); + expect(warn).toHaveBeenCalledWith(expect.stringContaining("Annex B")); + } finally { + encoder.close(); + probe.mockRestore(); + warn.mockRestore(); + } +}); + +// A 256-511 byte first NAL has a length prefix of `00 00 01 xx`, which reads as an Annex B start code. +// The description still marks the output as length-prefixed. +test("a length prefix that reads as a start code is still refused", async () => { + using _videoEncoder = installFakeVideoEncoder(); + FakeVideoEncoder.lengthPrefixed = (codec) => (codec.startsWith("hev1") ? 300 : undefined); const probe = spyOn(FakeVideoEncoder, "isConfigSupported").mockImplementation(async (config) => ({ supported: config.codec.startsWith("hev1") || config.codec === "vp8", })); @@ -484,7 +514,7 @@ test("an encoder that ignores the requested Annex B format is skipped for the ne test("an HEVC-only encoder that ignores Annex B leaves no codec rather than falling back at runtime", async () => { using _videoEncoder = installFakeVideoEncoder(); - FakeVideoEncoder.lengthPrefixed = (codec) => codec.startsWith("hev1"); + FakeVideoEncoder.lengthPrefixed = (codec) => (codec.startsWith("hev1") ? 59 : undefined); const probe = spyOn(FakeVideoEncoder, "isConfigSupported").mockImplementation(async (config) => ({ supported: config.codec.startsWith("hev1"), })); diff --git a/js/publish/src/video/encoder.ts b/js/publish/src/video/encoder.ts index 88bd6221adb..f2055e8706e 100644 --- a/js/publish/src/video/encoder.ts +++ b/js/publish/src/video/encoder.ts @@ -1010,17 +1010,21 @@ const PROBE_FRAMES = 4; // - Output that reorders. The catalog declares none, but Firefox's software H.264 still emits // B-frames for High and Main under realtime latency. // - H.264 or HEVC that isn't Annex B. Safari 26 accepts `hevc: { format: "annexb" }` but emits -// length-prefixed NAL units with an hvcC description, which no subscriber can decode. +// length-prefixed NAL units with an hvcC description, which no subscriber can decode. A +// description means length-prefixed output (per the AVC and HEVC WebCodecs registrations), which +// also catches a four-byte length that happens to read as a start code, such as `00 00 01 2c`. async function probeEncoder(config: VideoEncoderConfig): Promise<{ reported: string } | { refused: string }> { let reported: string | undefined; let reordered = false; let last: number | undefined; let first: Uint8Array | undefined; + let described = false; const encoder = new VideoEncoder({ output: (chunk, metadata) => { reported ??= metadata?.decoderConfig?.codec; if (last !== undefined && chunk.timestamp <= last) reordered = true; last = chunk.timestamp; + if (metadata?.decoderConfig?.description) described = true; if (!first) { first = new Uint8Array(chunk.byteLength); chunk.copyTo(first); @@ -1057,7 +1061,7 @@ async function probeEncoder(config: VideoEncoderConfig): Promise<{ reported: str if (!reported) throw new Error(`${config.codec} encoder reported no codec string`); if (reordered) return { refused: "its encoder reorders frames despite realtime latency" }; - if (requestsAnnexB(config.codec) && !(first && startsWithStartCode(first))) { + if (requestsAnnexB(config.codec) && (described || !(first && startsWithStartCode(first)))) { return { refused: "its encoder ignored the requested Annex B format" }; } return { reported };