Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions quest/m1/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ were done or not worth their cost.

## Required

- [WebGPU renderer](/quest/m1/webgpu-renderer.md) - a new `@moq/video` renders frames through WebGPU in their own colour space, with Canvas2D where WebGPU is missing, for watch and the publish preview
- [One max_age meaning](/quest/m1/cache-max-age.md) - a superseded group goes stale on wall clock since its successor arrived or on media time, whichever is first, in Rust and js/net; fixes the untimed failover stall
- [Held group wakes](/quest/m1/held-group-wakes.md) - a group held across a route switch wakes when its successor's first timestamp or abort makes it stale; cache-max-age's wall-clock budget rides the same wake
- [Upstream position regression](/quest/m1/largest-regression.md) - a relay copy that sees upstream's largest group go backwards on moq-transport or epochless lite-07 fails loud instead of serving the old instance's cache
Expand Down Expand Up @@ -172,3 +173,5 @@ were done or not worth their cost.
- [JS pending tail](/quest/m1/js-pending-tail.md) - a JS reader holds for a track's pending tail like Rust, once #4225 lands
- [In-band CMAF follow-ups](/quest/m1/cmaf-inline-followups.md) - MSF, h264/h265 export, and gst caps handle avc3/hev1 CMAF
- [Datagram replay bound](/quest/m1/datagram-replay-bound.md) - a new Rust datagram subscriber starts within its max delay of the newest datagram, not at a minutes-old buffer
- [Catalog colour](/quest/m1/color-catalog.md) - the catalog describes a rendition's colour and HDR properties, which the WebGPU HDR renderer reads
- [WebGPU HDR](/quest/m1/webgpu-hdr.md) - HDR renditions play as HDR where the browser and display can show it, and tone-map to SDR elsewhere
7 changes: 4 additions & 3 deletions quest/m3/color-catalog.md → quest/m1/color-catalog.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@ instead of guessing or reparsing the bitstream.

## Plan

Deferred to m2 in the 2026-09-30 audit: no renderer consumes colour today, so
the model waits for one. Decided 2026-10-08: moved to m3; no renderer
consumes colour. Encoder colour correctness stays in m1 as
Decided 2026-10-08 (WebGPU renderer interview): moved to m1. The
[WebGPU HDR](/quest/m1/webgpu-hdr.md) renderer is the consumer this model
waited for. Encoder colour correctness stays in m1 as
[Encoder colour](/quest/m1/color-model.md).

`rs/hang/src/catalog/video/mod.rs` has carried a bare `// TODO color space` since
Expand Down Expand Up @@ -56,3 +56,4 @@ stays byte-identical.
## Related

- [Encoder colour](/quest/m1/color-model.md) - encoders signal the colour they actually produce
- [WebGPU renderer](/quest/m1/webgpu-renderer.md) - renders SDR and wide gamut from `VideoFrame.colorSpace` before this lands
10 changes: 5 additions & 5 deletions quest/m1/color-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,11 @@ warn about a known color mismatch and then label unchanged pixels as the
requested color: convert correctly or refuse. Preserve color information
through CUDA, D3D, and Android paths and test the signaled VUI against pixels.

Decided in the 2026-09-30 audit: split from the catalog colour model, which
moved to [Catalog colour](/quest/m3/color-catalog.md) because no renderer
consumes colour from the catalog today. Encoder correctness stays in m1
because a mislabelled stream is wrong for every consumer.
Decided in the 2026-09-30 audit: split from the catalog colour model,
[Catalog colour](/quest/m1/color-catalog.md), which the WebGPU HDR renderer
now consumes (2026-10-08). Encoder correctness stays separate because a
mislabelled stream is wrong for every consumer.

## Related

- [Catalog colour](/quest/m3/color-catalog.md) - the catalog describes a rendition's colour and HDR properties
- [Catalog colour](/quest/m1/color-catalog.md) - the catalog describes a rendition's colour and HDR properties
1 change: 1 addition & 0 deletions quest/m1/watch-worker.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,4 +35,5 @@ Gate on the plan's jank harness and N-player sweep, both nightly.

## Related

- [WebGPU renderer](/quest/m1/webgpu-renderer.md) - the renderer this moves into the worker; it already takes an `OffscreenCanvas`
- [Time stretch](/quest/m1/watch-audio-time-stretch.md) - changes the worklet this feeds
47 changes: 47 additions & 0 deletions quest/m1/webgpu-hdr.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# [M] HDR video renders as HDR

## Goal

An HDR rendition (PQ or HLG) plays as HDR in the browser where the display
and browser can show it: the `@moq/video` WebGPU renderer converts the frame
itself and presents through an `rgba16float` canvas with extended tone
mapping. Elsewhere it is tone-mapped to SDR deliberately, instead of being
drawn as if it were SDR as today.

## Plan

Decided 2026-10-08 in the WebGPU renderer `/quest-plan` interview: SDR and
wide gamut first in [WebGPU renderer](/quest/m1/webgpu-renderer.md), HDR here,
both in m1.

Facts (2026-10):

- HDR canvas output:
`configure({ format: "rgba16float", toneMapping: { mode: "extended" } })`
ships on Chromium desktop since Chrome 129. Safari 26's release notes list
HDR for WebGPU canvases without a device or display matrix, and an earlier
WebKit build accepted `extended` while displaying SDR (WebKit bug 272702),
so Safari is unverified. Firefox has none. Canvas2D has no HDR output.
- `importExternalTexture` and `copyExternalImageToTexture` carry no HDR
headroom (gpuweb#5236 is open), so imported video is SDR. HDR needs the
renderer to convert the frame itself: read the planes, apply the matrix,
range, transfer (PQ or HLG), and primaries, and tone-map when the display
has less headroom.
- `VideoFrame.colorSpace` gives primaries, transfer, matrix, and range
(Chrome 94, Firefox 130, Safari 16.4). Mastering display and content light
levels come from [Catalog colour](/quest/m1/color-catalog.md).

Settle while building: whether the HDR path gives up zero copy (planes via
`copyTo` and a custom conversion shader), and how the renderer detects
display headroom.

Tests: an HDR10 rendition renders extended on Chromium desktop and on Safari
26 with an HDR display if it does show extended output, tone-maps to SDR
where extended output is unavailable, and an SDR rendition is unchanged.

Public API: none beyond the renderer. Wire: none.

## Required

- [WebGPU renderer](/quest/m1/webgpu-renderer.md) - the renderer and `@moq/video` package this extends
- [Catalog colour](/quest/m1/color-catalog.md) - the mastering display and content light levels
82 changes: 82 additions & 0 deletions quest/m1/webgpu-renderer.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,82 @@
# [L] Video renders through WebGPU

## Goal

A new `@moq/video` package renders video frames through WebGPU where the
browser has it, importing each `VideoFrame` with `importExternalTexture`
(zero copy where the browser allows), in the frame's own colour space
(sRGB or Display P3). Where WebGPU or a `VideoFrame` external texture is
missing, it renders through the existing Canvas2D path. `@moq/watch` and the
`@moq/publish` preview both use it, so the preview gains rotation too.

Not here: HDR output, which is [WebGPU HDR](/quest/m1/webgpu-hdr.md); moving
playback into a worker, which is [watch worker](/quest/m1/watch-worker.md);
and moving the WebCodecs decoder or encoder out of watch and publish.

## Plan

Decided 2026-10-08 in a `/quest-plan` interview (paper trail in the PR that
added this quest). Re-planned from issue #703, whose stub quest the
2026-09-28 audit (#4393) dropped:

- Why: correct colour (and later HDR), and draw cost. Measured on a 1080p
microbenchmark (webcodecsfundamentals.org), Canvas2D vs WebGPU frames per
second: Firefox 70 vs 430, Safari 230 vs 610, Chrome 960 vs 1230. Rejected
as reasons on their own: an effects hook, and replacing Canvas2D outright.
- Coverage (2026-10): WebGPU is Baseline, but Firefox on Linux, Intel Mac,
and Android, and Chrome on Linux with GPUs other than Intel Gen12+ or NVIDIA
on Wayland, still lack it, and `importExternalTexture(VideoFrame)` is
missing on Firefox Android. So Canvas2D stays as an explicit second
renderer. Rejected: WebGPU only, refusing elsewhere.
- Package: a new `@moq/video`, mirroring Rust's moq-video, holding only the
renderer for now (the WebGPU and Canvas2D paths, the presentation
transform from `js/watch/src/video/presentation.ts`, and selection).
`@moq/watch` and `@moq/publish` depend on it. Rejected: putting it in
`@moq/hang` (mixes a DOM and GPU concern into the media layer), and making
publish depend on watch. Adds the release scaffolding and a
`doc/lib/js/video.md` page.
- A public `renderer` option, `"auto" | "webgpu" | "2d"`: an attribute on
`<moq-watch>` and `<moq-publish>` and an input on the renderer. `"auto"` (the
default) picks WebGPU where supported, else Canvas2D, and logs the choice.
An explicit `"webgpu"` where it is missing refuses loudly; only `"auto"`
falls back.
- Device loss: request a new device and rebuild the pipeline. If no adapter
comes back, `"auto"` switches to Canvas2D (logged) rather than going black.
A canvas that acquired a `webgpu` context never returns a `2d` one, even
after device loss, and the canvas is caller-owned (the `<canvas>` child of
`<moq-watch>` or `<moq-publish>`, or a transferred `OffscreenCanvas`). Open:
who supplies the replacement surface for this fallback, including in the
worker. Settle it with the maintainer before building the fallback.
- The renderer takes an `HTMLCanvasElement` or an `OffscreenCanvas`, so the
[watch worker](/quest/m1/watch-worker.md) move transfers the canvas and
carries this renderer over unchanged. This lands first, on the main thread,
behind the existing `Renderer` API.
- Keep today's behaviour: rotation and flip parity with
`canvasPresentationTransform`, the canvas sized to the video's display size
(no devicePixelRatio scaling), `out.frame`, `out.timestamp`, and
`out.visible`. Import, bind, encode, and submit each frame with no `await`
in between, since the external texture expires when the frame closes.
Adapter and device setup is async, so the first paint waits on it.
- Watch for known Safari 26 issues with `importExternalTexture` (H.264
frames, biplanar formats, iOS orientation) and test on Safari.
- Measurement: ship, then measure. Extending the browser benchmarks with a
per-browser Canvas2D vs WebGPU comparison is a follow-up, not a gate.
- Tests: selection (auto picks WebGPU or Canvas2D by support, an explicit
`"webgpu"` refuses where missing), rotation and flip parity between the two
paths, device-loss recovery, and the preview's rotation. The fallback
test runs in a real browser: paint with WebGPU, lose the device with no
adapter to recover, then check Canvas2D actually paints.

Public API: new `@moq/video` package; a `renderer` attribute and option on
`<moq-watch>` and `<moq-publish>`; the publish preview's renderer moves to
the shared one. Wire: none.

## Closes

- [#703](https://github.com/moq-dev/moq/issues/703) - close this issue when the quest finishes

## Related

- [WebGPU HDR](/quest/m1/webgpu-hdr.md) - HDR output on top of this renderer
- [Watch worker](/quest/m1/watch-worker.md) - moves playback, and this renderer, into a worker
- [Catalog colour](/quest/m1/color-catalog.md) - colour metadata the HDR follow-up reads
1 change: 0 additions & 1 deletion quest/m3/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@ m4's upstream waits in here and moved m2 work with no consumer.
- [Video hardware validation](/quest/m3/video-hardware.md) - run the encode, capture, and zero-copy paths that were never run on real machines, including PipeWire on KDE, the camera portal, and a Pi
- [An Ada NVIDIA GPU is available](/quest/m3/ada-gpu.md) - the hardware AV1 NVENC is verified on
- [NVENC AV1](/quest/m3/nvenc-av1.md) - AV1 encode through NVENC, once an Ada-generation GPU is available
- [Catalog colour model](/quest/m3/color-catalog.md) - the catalog describes a rendition's colour and HDR properties once a renderer consumes them
- [#2147](/quest/m3/2147-moq-video-10-bit-hevc-and-av1-support-in-the-nvidia-codec.md) - NVIDIA Main10 encode, once a 10-bit source exists in the pipeline
- [Embedded video](/quest/m3/video-embedded.md) - verify a Pi 4/5 presents through the existing Vulkan and CPU paths, adding EGL import only if that fails
- [Multi-plane PipeWire cameras](/quest/m3/pipewire-camera-planes.md) - I420 and NV12 cameras that deliver one memory block per plane capture
Expand Down
Loading