diff --git a/quest/m1/README.md b/quest/m1/README.md index 501a7ffaf78..c5939aeb530 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -32,6 +32,7 @@ were done or not worth their cost. ## Required +- [WebGPU renderer](/quest/m1/webgpu-renderer.md) - a new `@moq/video` renders frames through WebGPU in their own colour space, with Canvas2D where WebGPU is missing, for watch and the publish preview - [One max_age meaning](/quest/m1/cache-max-age.md) - a superseded group goes stale on wall clock since its successor arrived or on media time, whichever is first, in Rust and js/net; fixes the untimed failover stall - [Held group wakes](/quest/m1/held-group-wakes.md) - a group held across a route switch wakes when its successor's first timestamp or abort makes it stale; cache-max-age's wall-clock budget rides the same wake - [Upstream position regression](/quest/m1/largest-regression.md) - a relay copy that sees upstream's largest group go backwards on moq-transport or epochless lite-07 fails loud instead of serving the old instance's cache @@ -172,3 +173,5 @@ were done or not worth their cost. - [JS pending tail](/quest/m1/js-pending-tail.md) - a JS reader holds for a track's pending tail like Rust, once #4225 lands - [In-band CMAF follow-ups](/quest/m1/cmaf-inline-followups.md) - MSF, h264/h265 export, and gst caps handle avc3/hev1 CMAF - [Datagram replay bound](/quest/m1/datagram-replay-bound.md) - a new Rust datagram subscriber starts within its max delay of the newest datagram, not at a minutes-old buffer +- [Catalog colour](/quest/m1/color-catalog.md) - the catalog describes a rendition's colour and HDR properties, which the WebGPU HDR renderer reads +- [WebGPU HDR](/quest/m1/webgpu-hdr.md) - HDR renditions play as HDR where the browser and display can show it, and tone-map to SDR elsewhere diff --git a/quest/m3/color-catalog.md b/quest/m1/color-catalog.md similarity index 90% rename from quest/m3/color-catalog.md rename to quest/m1/color-catalog.md index d786a726823..054ff3312f3 100644 --- a/quest/m3/color-catalog.md +++ b/quest/m1/color-catalog.md @@ -9,9 +9,9 @@ instead of guessing or reparsing the bitstream. ## Plan -Deferred to m2 in the 2026-09-30 audit: no renderer consumes colour today, so -the model waits for one. Decided 2026-10-08: moved to m3; no renderer -consumes colour. Encoder colour correctness stays in m1 as +Decided 2026-10-08 (WebGPU renderer interview): moved to m1. The +[WebGPU HDR](/quest/m1/webgpu-hdr.md) renderer is the consumer this model +waited for. Encoder colour correctness stays in m1 as [Encoder colour](/quest/m1/color-model.md). `rs/hang/src/catalog/video/mod.rs` has carried a bare `// TODO color space` since @@ -56,3 +56,4 @@ stays byte-identical. ## Related - [Encoder colour](/quest/m1/color-model.md) - encoders signal the colour they actually produce +- [WebGPU renderer](/quest/m1/webgpu-renderer.md) - renders SDR and wide gamut from `VideoFrame.colorSpace` before this lands diff --git a/quest/m1/color-model.md b/quest/m1/color-model.md index a6a9f868e94..66d3f4d870f 100644 --- a/quest/m1/color-model.md +++ b/quest/m1/color-model.md @@ -15,11 +15,11 @@ warn about a known color mismatch and then label unchanged pixels as the requested color: convert correctly or refuse. Preserve color information through CUDA, D3D, and Android paths and test the signaled VUI against pixels. -Decided in the 2026-09-30 audit: split from the catalog colour model, which -moved to [Catalog colour](/quest/m3/color-catalog.md) because no renderer -consumes colour from the catalog today. Encoder correctness stays in m1 -because a mislabelled stream is wrong for every consumer. +Decided in the 2026-09-30 audit: split from the catalog colour model, +[Catalog colour](/quest/m1/color-catalog.md), which the WebGPU HDR renderer +now consumes (2026-10-08). Encoder correctness stays separate because a +mislabelled stream is wrong for every consumer. ## Related -- [Catalog colour](/quest/m3/color-catalog.md) - the catalog describes a rendition's colour and HDR properties +- [Catalog colour](/quest/m1/color-catalog.md) - the catalog describes a rendition's colour and HDR properties diff --git a/quest/m1/watch-worker.md b/quest/m1/watch-worker.md index 9690a09e4aa..b2170648572 100644 --- a/quest/m1/watch-worker.md +++ b/quest/m1/watch-worker.md @@ -35,4 +35,5 @@ Gate on the plan's jank harness and N-player sweep, both nightly. ## Related +- [WebGPU renderer](/quest/m1/webgpu-renderer.md) - the renderer this moves into the worker; it already takes an `OffscreenCanvas` - [Time stretch](/quest/m1/watch-audio-time-stretch.md) - changes the worklet this feeds diff --git a/quest/m1/webgpu-hdr.md b/quest/m1/webgpu-hdr.md new file mode 100644 index 00000000000..7058328ec45 --- /dev/null +++ b/quest/m1/webgpu-hdr.md @@ -0,0 +1,47 @@ +# [M] HDR video renders as HDR + +## Goal + +An HDR rendition (PQ or HLG) plays as HDR in the browser where the display +and browser can show it: the `@moq/video` WebGPU renderer converts the frame +itself and presents through an `rgba16float` canvas with extended tone +mapping. Elsewhere it is tone-mapped to SDR deliberately, instead of being +drawn as if it were SDR as today. + +## Plan + +Decided 2026-10-08 in the WebGPU renderer `/quest-plan` interview: SDR and +wide gamut first in [WebGPU renderer](/quest/m1/webgpu-renderer.md), HDR here, +both in m1. + +Facts (2026-10): + +- HDR canvas output: + `configure({ format: "rgba16float", toneMapping: { mode: "extended" } })` + ships on Chromium desktop since Chrome 129. Safari 26's release notes list + HDR for WebGPU canvases without a device or display matrix, and an earlier + WebKit build accepted `extended` while displaying SDR (WebKit bug 272702), + so Safari is unverified. Firefox has none. Canvas2D has no HDR output. +- `importExternalTexture` and `copyExternalImageToTexture` carry no HDR + headroom (gpuweb#5236 is open), so imported video is SDR. HDR needs the + renderer to convert the frame itself: read the planes, apply the matrix, + range, transfer (PQ or HLG), and primaries, and tone-map when the display + has less headroom. +- `VideoFrame.colorSpace` gives primaries, transfer, matrix, and range + (Chrome 94, Firefox 130, Safari 16.4). Mastering display and content light + levels come from [Catalog colour](/quest/m1/color-catalog.md). + +Settle while building: whether the HDR path gives up zero copy (planes via +`copyTo` and a custom conversion shader), and how the renderer detects +display headroom. + +Tests: an HDR10 rendition renders extended on Chromium desktop and on Safari +26 with an HDR display if it does show extended output, tone-maps to SDR +where extended output is unavailable, and an SDR rendition is unchanged. + +Public API: none beyond the renderer. Wire: none. + +## Required + +- [WebGPU renderer](/quest/m1/webgpu-renderer.md) - the renderer and `@moq/video` package this extends +- [Catalog colour](/quest/m1/color-catalog.md) - the mastering display and content light levels diff --git a/quest/m1/webgpu-renderer.md b/quest/m1/webgpu-renderer.md new file mode 100644 index 00000000000..2e9b17c4895 --- /dev/null +++ b/quest/m1/webgpu-renderer.md @@ -0,0 +1,82 @@ +# [L] Video renders through WebGPU + +## Goal + +A new `@moq/video` package renders video frames through WebGPU where the +browser has it, importing each `VideoFrame` with `importExternalTexture` +(zero copy where the browser allows), in the frame's own colour space +(sRGB or Display P3). Where WebGPU or a `VideoFrame` external texture is +missing, it renders through the existing Canvas2D path. `@moq/watch` and the +`@moq/publish` preview both use it, so the preview gains rotation too. + +Not here: HDR output, which is [WebGPU HDR](/quest/m1/webgpu-hdr.md); moving +playback into a worker, which is [watch worker](/quest/m1/watch-worker.md); +and moving the WebCodecs decoder or encoder out of watch and publish. + +## Plan + +Decided 2026-10-08 in a `/quest-plan` interview (paper trail in the PR that +added this quest). Re-planned from issue #703, whose stub quest the +2026-09-28 audit (#4393) dropped: + +- Why: correct colour (and later HDR), and draw cost. Measured on a 1080p + microbenchmark (webcodecsfundamentals.org), Canvas2D vs WebGPU frames per + second: Firefox 70 vs 430, Safari 230 vs 610, Chrome 960 vs 1230. Rejected + as reasons on their own: an effects hook, and replacing Canvas2D outright. +- Coverage (2026-10): WebGPU is Baseline, but Firefox on Linux, Intel Mac, + and Android, and Chrome on Linux with GPUs other than Intel Gen12+ or NVIDIA + on Wayland, still lack it, and `importExternalTexture(VideoFrame)` is + missing on Firefox Android. So Canvas2D stays as an explicit second + renderer. Rejected: WebGPU only, refusing elsewhere. +- Package: a new `@moq/video`, mirroring Rust's moq-video, holding only the + renderer for now (the WebGPU and Canvas2D paths, the presentation + transform from `js/watch/src/video/presentation.ts`, and selection). + `@moq/watch` and `@moq/publish` depend on it. Rejected: putting it in + `@moq/hang` (mixes a DOM and GPU concern into the media layer), and making + publish depend on watch. Adds the release scaffolding and a + `doc/lib/js/video.md` page. +- A public `renderer` option, `"auto" | "webgpu" | "2d"`: an attribute on + `` and `` and an input on the renderer. `"auto"` (the + default) picks WebGPU where supported, else Canvas2D, and logs the choice. + An explicit `"webgpu"` where it is missing refuses loudly; only `"auto"` + falls back. +- Device loss: request a new device and rebuild the pipeline. If no adapter + comes back, `"auto"` switches to Canvas2D (logged) rather than going black. + A canvas that acquired a `webgpu` context never returns a `2d` one, even + after device loss, and the canvas is caller-owned (the `` child of + `` or ``, or a transferred `OffscreenCanvas`). Open: + who supplies the replacement surface for this fallback, including in the + worker. Settle it with the maintainer before building the fallback. +- The renderer takes an `HTMLCanvasElement` or an `OffscreenCanvas`, so the + [watch worker](/quest/m1/watch-worker.md) move transfers the canvas and + carries this renderer over unchanged. This lands first, on the main thread, + behind the existing `Renderer` API. +- Keep today's behaviour: rotation and flip parity with + `canvasPresentationTransform`, the canvas sized to the video's display size + (no devicePixelRatio scaling), `out.frame`, `out.timestamp`, and + `out.visible`. Import, bind, encode, and submit each frame with no `await` + in between, since the external texture expires when the frame closes. + Adapter and device setup is async, so the first paint waits on it. +- Watch for known Safari 26 issues with `importExternalTexture` (H.264 + frames, biplanar formats, iOS orientation) and test on Safari. +- Measurement: ship, then measure. Extending the browser benchmarks with a + per-browser Canvas2D vs WebGPU comparison is a follow-up, not a gate. +- Tests: selection (auto picks WebGPU or Canvas2D by support, an explicit + `"webgpu"` refuses where missing), rotation and flip parity between the two + paths, device-loss recovery, and the preview's rotation. The fallback + test runs in a real browser: paint with WebGPU, lose the device with no + adapter to recover, then check Canvas2D actually paints. + +Public API: new `@moq/video` package; a `renderer` attribute and option on +`` and ``; the publish preview's renderer moves to +the shared one. Wire: none. + +## Closes + +- [#703](https://github.com/moq-dev/moq/issues/703) - close this issue when the quest finishes + +## Related + +- [WebGPU HDR](/quest/m1/webgpu-hdr.md) - HDR output on top of this renderer +- [Watch worker](/quest/m1/watch-worker.md) - moves playback, and this renderer, into a worker +- [Catalog colour](/quest/m1/color-catalog.md) - colour metadata the HDR follow-up reads diff --git a/quest/m3/README.md b/quest/m3/README.md index a1782fd6129..1e0f42fbd3e 100644 --- a/quest/m3/README.md +++ b/quest/m3/README.md @@ -23,7 +23,6 @@ m4's upstream waits in here and moved m2 work with no consumer. - [Video hardware validation](/quest/m3/video-hardware.md) - run the encode, capture, and zero-copy paths that were never run on real machines, including PipeWire on KDE, the camera portal, and a Pi - [An Ada NVIDIA GPU is available](/quest/m3/ada-gpu.md) - the hardware AV1 NVENC is verified on - [NVENC AV1](/quest/m3/nvenc-av1.md) - AV1 encode through NVENC, once an Ada-generation GPU is available -- [Catalog colour model](/quest/m3/color-catalog.md) - the catalog describes a rendition's colour and HDR properties once a renderer consumes them - [#2147](/quest/m3/2147-moq-video-10-bit-hevc-and-av1-support-in-the-nvidia-codec.md) - NVIDIA Main10 encode, once a 10-bit source exists in the pipeline - [Embedded video](/quest/m3/video-embedded.md) - verify a Pi 4/5 presents through the existing Vulkan and CPU paths, adding EGL import only if that fails - [Multi-plane PipeWire cameras](/quest/m3/pipewire-camera-planes.md) - I420 and NV12 cameras that deliver one memory block per plane capture