From 944169b8c1d2993c5080ce5d1175ffab233f11d1 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 8 Oct 2026 15:34:22 -0700 Subject: [PATCH 1/3] quest: plan a WebGPU renderer, HDR output, and catalog colour in m1 Re-plans issue #703. A new @moq/video package renders frames through WebGPU with Canvas2D where WebGPU is missing, for watch and the publish preview. HDR output follows, and the catalog colour model moves from m3 to m1 as its metadata source. Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 3 ++ quest/{m3 => m1}/color-catalog.md | 7 +-- quest/m1/color-model.md | 10 ++--- quest/m1/watch-worker.md | 1 + quest/m1/webgpu-hdr.md | 44 ++++++++++++++++++ quest/m1/webgpu-renderer.md | 75 +++++++++++++++++++++++++++++++ quest/m3/README.md | 1 - 7 files changed, 132 insertions(+), 9 deletions(-) rename quest/{m3 => m1}/color-catalog.md (90%) create mode 100644 quest/m1/webgpu-hdr.md create mode 100644 quest/m1/webgpu-renderer.md diff --git a/quest/m1/README.md b/quest/m1/README.md index 501a7ffaf78..6547d36b065 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -32,6 +32,7 @@ were done or not worth their cost. ## Required +- [WebGPU renderer](/quest/m1/webgpu-renderer.md) - a new `@moq/video` renders frames through WebGPU in their own colour space, with Canvas2D where WebGPU is missing, for watch and the publish preview - [One max_age meaning](/quest/m1/cache-max-age.md) - a superseded group goes stale on wall clock since its successor arrived or on media time, whichever is first, in Rust and js/net; fixes the untimed failover stall - [Held group wakes](/quest/m1/held-group-wakes.md) - a group held across a route switch wakes when its successor's first timestamp or abort makes it stale; cache-max-age's wall-clock budget rides the same wake - [Upstream position regression](/quest/m1/largest-regression.md) - a relay copy that sees upstream's largest group go backwards on moq-transport or epochless lite-07 fails loud instead of serving the old instance's cache @@ -172,3 +173,5 @@ were done or not worth their cost. - [JS pending tail](/quest/m1/js-pending-tail.md) - a JS reader holds for a track's pending tail like Rust, once #4225 lands - [In-band CMAF follow-ups](/quest/m1/cmaf-inline-followups.md) - MSF, h264/h265 export, and gst caps handle avc3/hev1 CMAF - [Datagram replay bound](/quest/m1/datagram-replay-bound.md) - a new Rust datagram subscriber starts within its max delay of the newest datagram, not at a minutes-old buffer +- [WebGPU HDR](/quest/m1/webgpu-hdr.md) - HDR renditions play as HDR on Chromium desktop and tone-map to SDR elsewhere +- [Catalog colour](/quest/m1/color-catalog.md) - the catalog describes a rendition's colour and HDR properties, which the WebGPU HDR renderer reads diff --git a/quest/m3/color-catalog.md b/quest/m1/color-catalog.md similarity index 90% rename from quest/m3/color-catalog.md rename to quest/m1/color-catalog.md index d786a726823..054ff3312f3 100644 --- a/quest/m3/color-catalog.md +++ b/quest/m1/color-catalog.md @@ -9,9 +9,9 @@ instead of guessing or reparsing the bitstream. ## Plan -Deferred to m2 in the 2026-09-30 audit: no renderer consumes colour today, so -the model waits for one. Decided 2026-10-08: moved to m3; no renderer -consumes colour. Encoder colour correctness stays in m1 as +Decided 2026-10-08 (WebGPU renderer interview): moved to m1. The +[WebGPU HDR](/quest/m1/webgpu-hdr.md) renderer is the consumer this model +waited for. Encoder colour correctness stays in m1 as [Encoder colour](/quest/m1/color-model.md). `rs/hang/src/catalog/video/mod.rs` has carried a bare `// TODO color space` since @@ -56,3 +56,4 @@ stays byte-identical. ## Related - [Encoder colour](/quest/m1/color-model.md) - encoders signal the colour they actually produce +- [WebGPU renderer](/quest/m1/webgpu-renderer.md) - renders SDR and wide gamut from `VideoFrame.colorSpace` before this lands diff --git a/quest/m1/color-model.md b/quest/m1/color-model.md index a6a9f868e94..66d3f4d870f 100644 --- a/quest/m1/color-model.md +++ b/quest/m1/color-model.md @@ -15,11 +15,11 @@ warn about a known color mismatch and then label unchanged pixels as the requested color: convert correctly or refuse. Preserve color information through CUDA, D3D, and Android paths and test the signaled VUI against pixels. -Decided in the 2026-09-30 audit: split from the catalog colour model, which -moved to [Catalog colour](/quest/m3/color-catalog.md) because no renderer -consumes colour from the catalog today. Encoder correctness stays in m1 -because a mislabelled stream is wrong for every consumer. +Decided in the 2026-09-30 audit: split from the catalog colour model, +[Catalog colour](/quest/m1/color-catalog.md), which the WebGPU HDR renderer +now consumes (2026-10-08). Encoder correctness stays separate because a +mislabelled stream is wrong for every consumer. ## Related -- [Catalog colour](/quest/m3/color-catalog.md) - the catalog describes a rendition's colour and HDR properties +- [Catalog colour](/quest/m1/color-catalog.md) - the catalog describes a rendition's colour and HDR properties diff --git a/quest/m1/watch-worker.md b/quest/m1/watch-worker.md index 9690a09e4aa..b2170648572 100644 --- a/quest/m1/watch-worker.md +++ b/quest/m1/watch-worker.md @@ -35,4 +35,5 @@ Gate on the plan's jank harness and N-player sweep, both nightly. ## Related +- [WebGPU renderer](/quest/m1/webgpu-renderer.md) - the renderer this moves into the worker; it already takes an `OffscreenCanvas` - [Time stretch](/quest/m1/watch-audio-time-stretch.md) - changes the worklet this feeds diff --git a/quest/m1/webgpu-hdr.md b/quest/m1/webgpu-hdr.md new file mode 100644 index 00000000000..e7503517fbb --- /dev/null +++ b/quest/m1/webgpu-hdr.md @@ -0,0 +1,44 @@ +# [M] HDR video renders as HDR + +## Goal + +An HDR rendition (PQ or HLG) plays as HDR in the browser where the display +and browser can show it: the `@moq/video` WebGPU renderer converts the frame +itself and presents through an `rgba16float` canvas with extended tone +mapping. Elsewhere it is tone-mapped to SDR deliberately, instead of being +drawn as if it were SDR as today. + +## Plan + +Decided 2026-10-08 in the WebGPU renderer `/quest-plan` interview: SDR and +wide gamut first in [WebGPU renderer](/quest/m1/webgpu-renderer.md), HDR here, +both in m1. + +Facts (2026-10): + +- HDR canvas output ships only on Chromium desktop: + `configure({ format: "rgba16float", toneMapping: { mode: "extended" } })` + since Chrome 129. Safari 26 accepts the configuration on macOS but does not + display it extended; Firefox has none. Canvas2D has no HDR output. +- `importExternalTexture` and `copyExternalImageToTexture` carry no HDR + headroom (gpuweb#5236 is open), so imported video is SDR. HDR needs the + renderer to convert the frame itself: read the planes, apply the matrix, + range, transfer (PQ or HLG), and primaries, and tone-map when the display + has less headroom. +- `VideoFrame.colorSpace` gives primaries, transfer, matrix, and range + (Chrome 94, Firefox 130, Safari 16.4). Mastering display and content light + levels come from [Catalog colour](/quest/m1/color-catalog.md). + +Settle while building: whether the HDR path gives up zero copy (planes via +`copyTo` and a custom conversion shader), and how the renderer detects +display headroom. + +Tests: an HDR10 rendition renders extended on Chromium desktop, tone-maps to +SDR elsewhere, and an SDR rendition is unchanged. + +Public API: none beyond the renderer. Wire: none. + +## Required + +- [WebGPU renderer](/quest/m1/webgpu-renderer.md) - the renderer and `@moq/video` package this extends +- [Catalog colour](/quest/m1/color-catalog.md) - the mastering display and content light levels diff --git a/quest/m1/webgpu-renderer.md b/quest/m1/webgpu-renderer.md new file mode 100644 index 00000000000..22b7ef5149f --- /dev/null +++ b/quest/m1/webgpu-renderer.md @@ -0,0 +1,75 @@ +# [L] Video renders through WebGPU + +## Goal + +A new `@moq/video` package renders video frames through WebGPU where the +browser has it, importing each `VideoFrame` with `importExternalTexture` +(zero copy where the browser allows), in the frame's own colour space +(sRGB or Display P3). Where WebGPU or a `VideoFrame` external texture is +missing, it renders through the existing Canvas2D path. `@moq/watch` and the +`@moq/publish` preview both use it, so the preview gains rotation too. + +Not here: HDR output, which is [WebGPU HDR](/quest/m1/webgpu-hdr.md); moving +playback into a worker, which is [watch worker](/quest/m1/watch-worker.md); +and moving the WebCodecs decoder or encoder out of watch and publish. + +## Plan + +Decided 2026-10-08 in a `/quest-plan` interview (paper trail in the PR that +added this quest). Re-planned from issue #703, whose stub quest the +2026-09-28 audit (#4393) dropped: + +- Why: correct colour (and later HDR), and draw cost. Measured on a 1080p + microbenchmark (webcodecsfundamentals.org), Canvas2D vs WebGPU frames per + second: Firefox 70 vs 430, Safari 230 vs 610, Chrome 960 vs 1230. Rejected + as reasons on their own: an effects hook, and replacing Canvas2D outright. +- Coverage (2026-10): WebGPU is Baseline, but Firefox on Linux, Intel Mac, + and Android, and Chrome on Linux with GPUs other than Intel Gen12+ or NVIDIA + on Wayland, still lack it, and `importExternalTexture(VideoFrame)` is + missing on Firefox Android. So Canvas2D stays as an explicit second + renderer. Rejected: WebGPU only, refusing elsewhere. +- Package: a new `@moq/video`, mirroring Rust's moq-video, holding only the + renderer for now (the WebGPU and Canvas2D paths, the presentation + transform from `js/watch/src/video/presentation.ts`, and selection). + `@moq/watch` and `@moq/publish` depend on it. Rejected: putting it in + `@moq/hang` (mixes a DOM and GPU concern into the media layer), and making + publish depend on watch. Adds the release scaffolding and a + `doc/lib/js/video.md` page. +- A public `renderer` option, `"auto" | "webgpu" | "2d"`: an attribute on + `` and `` and an input on the renderer. `"auto"` (the + default) picks WebGPU where supported, else Canvas2D, and logs the choice. + An explicit `"webgpu"` where it is missing refuses loudly; only `"auto"` + falls back. +- Device loss: request a new device and rebuild the pipeline. If no adapter + comes back, `"auto"` switches to Canvas2D (logged) rather than going black. +- The renderer takes an `HTMLCanvasElement` or an `OffscreenCanvas`, so the + [watch worker](/quest/m1/watch-worker.md) move transfers the canvas and + carries this renderer over unchanged. This lands first, on the main thread, + behind the existing `Renderer` API. +- Keep today's behaviour: rotation and flip parity with + `canvasPresentationTransform`, the canvas sized to the video's display size + (no devicePixelRatio scaling), `out.frame`, `out.timestamp`, and + `out.visible`. Import, bind, encode, and submit each frame with no `await` + in between, since the external texture expires when the frame closes. + Adapter and device setup is async, so the first paint waits on it. +- Watch for known Safari 26 issues with `importExternalTexture` (H.264 + frames, biplanar formats, iOS orientation) and test on Safari. +- Measurement: ship, then measure. Extending the browser benchmarks with a + per-browser Canvas2D vs WebGPU comparison is a follow-up, not a gate. +- Tests: selection (auto picks WebGPU or Canvas2D by support, an explicit + `"webgpu"` refuses where missing), rotation and flip parity between the two + paths, device-loss recovery and fallback, and the preview's rotation. + +Public API: new `@moq/video` package; a `renderer` attribute and option on +`` and ``; the publish preview's renderer moves to +the shared one. Wire: none. + +## Closes + +- [#703](https://github.com/moq-dev/moq/issues/703) - close this issue when the quest finishes + +## Related + +- [WebGPU HDR](/quest/m1/webgpu-hdr.md) - HDR output on top of this renderer +- [Watch worker](/quest/m1/watch-worker.md) - moves playback, and this renderer, into a worker +- [Catalog colour](/quest/m1/color-catalog.md) - colour metadata the HDR follow-up reads diff --git a/quest/m3/README.md b/quest/m3/README.md index a1782fd6129..1e0f42fbd3e 100644 --- a/quest/m3/README.md +++ b/quest/m3/README.md @@ -23,7 +23,6 @@ m4's upstream waits in here and moved m2 work with no consumer. - [Video hardware validation](/quest/m3/video-hardware.md) - run the encode, capture, and zero-copy paths that were never run on real machines, including PipeWire on KDE, the camera portal, and a Pi - [An Ada NVIDIA GPU is available](/quest/m3/ada-gpu.md) - the hardware AV1 NVENC is verified on - [NVENC AV1](/quest/m3/nvenc-av1.md) - AV1 encode through NVENC, once an Ada-generation GPU is available -- [Catalog colour model](/quest/m3/color-catalog.md) - the catalog describes a rendition's colour and HDR properties once a renderer consumes them - [#2147](/quest/m3/2147-moq-video-10-bit-hevc-and-av1-support-in-the-nvidia-codec.md) - NVIDIA Main10 encode, once a 10-bit source exists in the pipeline - [Embedded video](/quest/m3/video-embedded.md) - verify a Pi 4/5 presents through the existing Vulkan and CPU paths, adding EGL import only if that fails - [Multi-plane PipeWire cameras](/quest/m3/pipewire-camera-planes.md) - I420 and NV12 cameras that deliver one memory block per plane capture From 2a1ef846310eaab2ce2a38a75cf86fff4168ffda Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 8 Oct 2026 15:41:05 -0700 Subject: [PATCH 2/3] quest: record the device-loss fallback surface as open A canvas that acquired a webgpu context never returns a 2d one, so the Canvas2D fallback after device loss needs a replacement surface. Record who supplies it as an open question and require a real-browser test. Co-Authored-By: Claude Opus 5.5 --- quest/m1/webgpu-renderer.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/quest/m1/webgpu-renderer.md b/quest/m1/webgpu-renderer.md index 22b7ef5149f..2e9b17c4895 100644 --- a/quest/m1/webgpu-renderer.md +++ b/quest/m1/webgpu-renderer.md @@ -42,6 +42,11 @@ added this quest). Re-planned from issue #703, whose stub quest the falls back. - Device loss: request a new device and rebuild the pipeline. If no adapter comes back, `"auto"` switches to Canvas2D (logged) rather than going black. + A canvas that acquired a `webgpu` context never returns a `2d` one, even + after device loss, and the canvas is caller-owned (the `` child of + `` or ``, or a transferred `OffscreenCanvas`). Open: + who supplies the replacement surface for this fallback, including in the + worker. Settle it with the maintainer before building the fallback. - The renderer takes an `HTMLCanvasElement` or an `OffscreenCanvas`, so the [watch worker](/quest/m1/watch-worker.md) move transfers the canvas and carries this renderer over unchanged. This lands first, on the main thread, @@ -58,7 +63,9 @@ added this quest). Re-planned from issue #703, whose stub quest the per-browser Canvas2D vs WebGPU comparison is a follow-up, not a gate. - Tests: selection (auto picks WebGPU or Canvas2D by support, an explicit `"webgpu"` refuses where missing), rotation and flip parity between the two - paths, device-loss recovery and fallback, and the preview's rotation. + paths, device-loss recovery, and the preview's rotation. The fallback + test runs in a real browser: paint with WebGPU, lose the device with no + adapter to recover, then check Canvas2D actually paints. Public API: new `@moq/video` package; a `renderer` attribute and option on `` and ``; the publish preview's renderer moves to From 0c2224ee2f5181f63660b8c35afe5b46c022af57 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 8 Oct 2026 15:48:27 -0700 Subject: [PATCH 3/3] quest: order catalog colour before WebGPU HDR, mark Safari HDR unverified Catalog colour blocks WebGPU HDR, so it ranks first in m1. Safari 26 lists HDR WebGPU canvases, so HDR targets any browser that shows extended output and tests Safari rather than ruling it out. Co-Authored-By: Claude Opus 5.5 --- quest/m1/README.md | 2 +- quest/m1/webgpu-hdr.md | 13 ++++++++----- 2 files changed, 9 insertions(+), 6 deletions(-) diff --git a/quest/m1/README.md b/quest/m1/README.md index 6547d36b065..c5939aeb530 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -173,5 +173,5 @@ were done or not worth their cost. - [JS pending tail](/quest/m1/js-pending-tail.md) - a JS reader holds for a track's pending tail like Rust, once #4225 lands - [In-band CMAF follow-ups](/quest/m1/cmaf-inline-followups.md) - MSF, h264/h265 export, and gst caps handle avc3/hev1 CMAF - [Datagram replay bound](/quest/m1/datagram-replay-bound.md) - a new Rust datagram subscriber starts within its max delay of the newest datagram, not at a minutes-old buffer -- [WebGPU HDR](/quest/m1/webgpu-hdr.md) - HDR renditions play as HDR on Chromium desktop and tone-map to SDR elsewhere - [Catalog colour](/quest/m1/color-catalog.md) - the catalog describes a rendition's colour and HDR properties, which the WebGPU HDR renderer reads +- [WebGPU HDR](/quest/m1/webgpu-hdr.md) - HDR renditions play as HDR where the browser and display can show it, and tone-map to SDR elsewhere diff --git a/quest/m1/webgpu-hdr.md b/quest/m1/webgpu-hdr.md index e7503517fbb..7058328ec45 100644 --- a/quest/m1/webgpu-hdr.md +++ b/quest/m1/webgpu-hdr.md @@ -16,10 +16,12 @@ both in m1. Facts (2026-10): -- HDR canvas output ships only on Chromium desktop: +- HDR canvas output: `configure({ format: "rgba16float", toneMapping: { mode: "extended" } })` - since Chrome 129. Safari 26 accepts the configuration on macOS but does not - display it extended; Firefox has none. Canvas2D has no HDR output. + ships on Chromium desktop since Chrome 129. Safari 26's release notes list + HDR for WebGPU canvases without a device or display matrix, and an earlier + WebKit build accepted `extended` while displaying SDR (WebKit bug 272702), + so Safari is unverified. Firefox has none. Canvas2D has no HDR output. - `importExternalTexture` and `copyExternalImageToTexture` carry no HDR headroom (gpuweb#5236 is open), so imported video is SDR. HDR needs the renderer to convert the frame itself: read the planes, apply the matrix, @@ -33,8 +35,9 @@ Settle while building: whether the HDR path gives up zero copy (planes via `copyTo` and a custom conversion shader), and how the renderer detects display headroom. -Tests: an HDR10 rendition renders extended on Chromium desktop, tone-maps to -SDR elsewhere, and an SDR rendition is unchanged. +Tests: an HDR10 rendition renders extended on Chromium desktop and on Safari +26 with an HDR display if it does show extended output, tone-maps to SDR +where extended output is unavailable, and an SDR rendition is unchanged. Public API: none beyond the renderer. Wire: none.