diff --git a/doc/bin/studio.md b/doc/bin/studio.md index 766c8f5b2df..12dac2b8779 100644 --- a/doc/bin/studio.md +++ b/doc/bin/studio.md @@ -108,3 +108,10 @@ studio passes over it for an encoder that reads each frame back to the CPU (the opt-in `vaapi` or `openh264` features), which the Output panel counts. Frames the encoder cannot keep up with are dropped rather than delaying the stream. + +Sources make the opposite trip. A frame already on the GPU is drawn where it +lies: PipeWire camera and VA-API decode DMA-BUFs on Linux, given a Vulkan +driver that imports them, and NV12 capture and VideoToolbox decode surfaces on +macOS. Every other source, every source on Windows or on a GPU that cannot +import, and every still image delivers its frames from the CPU, which the +compositor uploads. diff --git a/quest/m1/README.md b/quest/m1/README.md index b9b188c8e3a..93bf9235570 100644 --- a/quest/m1/README.md +++ b/quest/m1/README.md @@ -105,8 +105,7 @@ cut ships a relay that compiles no C. It follows moq-quic's crypto reshape, whose surface its QUIC adapter plugs into. Added 2026-10-25 (maintainer): the studio's -GPU encode and -[source import](/quest/m1/studio-gpu/import.md) quests are in m1 so +GPU encode and source import quests are in m1 so moq-video's GPU encode and import API changes, mainly encoders taking a GPU surface and zero-copy texture import, land before the cut. @@ -132,7 +131,6 @@ surface and zero-copy texture import, land before the cut. - [More tests under load](/quest/m1/test-flakes-2/README.md) - the second round of load-only failures, one quest per flake, fixed at the cause - [Broadcast lifecycle](/quest/m1/broadcast-lifecycle/README.md) - broadcasts are online or offline with no finished state in moq-net, finished-broadcast retention is a fixed moq-mux policy, and each linger timer has one meaning - [A smaller, safer API than release](/quest/m1/api-audit/README.md) - every surface added since release has a consumer or is gone, overlapping operations are collapsed, names mirror across Rust, JS, and the bindings, and every break says how to migrate -- [Studio on the GPU](/quest/m1/studio-gpu/README.md) - moq-video encoders take a GPU surface and its renderer reports zero-copy import, proven through moq-studio on NVENC, VideoToolbox, Media Foundation, DMA-BUF, and Metal - [moq-transport 18 and 22 only](/quest/m1/ietf-drop/README.md) - drafts 14-17 and 19-21 are deleted from Rust, JS, tests, interop lanes, and docs, removing the legacy control-stream adapter and its request limit - [moq-quic's API improves on quinn's](/quest/m1/quic-api/README.md) - moq-quic and both QUIC drivers ship one mirrored API: `send::`/`recv::` namespaces, one error per direction, consuming terminal operations, plain configs, no quinn baggage - [Pure-Rust crypto for moq-relay](/quest/m1/pure-crypto/README.md) - moq-relay's release builds use graviola and RustCrypto and compile no C diff --git a/quest/m1/studio-gpu/README.md b/quest/m1/studio-gpu/README.md deleted file mode 100644 index 374b0587af2..00000000000 --- a/quest/m1/studio-gpu/README.md +++ /dev/null @@ -1,30 +0,0 @@ -# Studio on the GPU - -## Goal - -`moq-studio`'s pictures stay on the GPU from source to encoder on the -backends proven before the cut: sources reach the compositor as textures on -the shared device, and the program reaches the encoder as a GPU surface. A -CPU copy remains only as a counted fallback, visible in the studio's Output -panel. - -## Plan - -Decided 2026-10-25 (maintainer): in m1, so moq-video's GPU encode and import -API changes land before the cut. Each piece lands in moq-video or moq-scene, -never in the app, so other consumers inherit it. - -Scope is the API shape plus the backends provable now. AMD Vulkan Video -encode, Intel VA-API on an external Vulkan image, and Direct3D11 render -import stay in m2 as Related follow-ups; until they land, those backends take -the counted CPU path. Nothing here requires m2 or m3 work, including the -[macOS and Windows runs](/quest/m3/studio/app.md): moq-video and moq-scene -test each platform's path on that platform's CI. - -## Required - -- [Zero-copy source import](/quest/m1/studio-gpu/import.md) - captured and decoded frames reach the compositor as GPU textures, with uploads counted - -## Related - -- [Studio](/quest/m3/studio/README.md) - the app these quests serve diff --git a/quest/m1/studio-gpu/import.md b/quest/m1/studio-gpu/import.md deleted file mode 100644 index 3b0e3e9cb32..00000000000 --- a/quest/m1/studio-gpu/import.md +++ /dev/null @@ -1,37 +0,0 @@ -# [M] Zero-copy source import - -## Goal - -Captured and decoded frames reach `moq-studio`'s compositor as GPU textures -on the shared device, imported rather than uploaded, on Linux (PipeWire and -VA-API DMA-BUFs) and Apple (IOSurface-backed `CVPixelBuffer`s through Metal). -moq-video's renderer reports whether each frame was imported or uploaded, and -the studio counts uploads per source next to its readback stats. - -## Plan - -- moq-scene draws every frame through moq-video's `Renderer`, which already - imports DMA-BUF (`render/dmabuf.rs`) and Metal (`render/metal.rs`) surfaces - without a copy, and decoded broadcast sources already take the decoder's - native surface. What remains: - - The renderer API reports import or upload per frame, so a caller can see - its fallback. This is the moq-video API change the cut should carry. - - moq-scene's `capture` does not enable moq-video's `pipewire` feature - (off by default because it links libpipewire), so Linux display and - camera sources arrive as CPU frames. Decide its build default with - [packaging](/quest/m3/studio/packaging.md) in mind. -- Windows sources keep the counted upload until - [Direct3D11 render import](/quest/m2/render-d3d11.md) lands in m2. -- The still-image `queue.write_texture` in `rs/moq-scene/src/compositor.rs` - runs once per image and is not part of this. -- Keep the import in moq-video or moq-scene, never in the app. -- **Tests:** the compositor's lavapipe tests assert the upload count for a CPU - frame, macOS CI asserts a Metal import, and a PipeWire DMA-BUF import is an - ignored hardware test. - -## Related - -- [Validate PipeWire DMA-BUFs into the Vulkan renderer](/quest/m2/2819-moq-video-carry-pipewire-dma-bufs-safely-into-the-vulkan.md) - the hardware proof of the Linux import path -- [Direct3D11 render import](/quest/m2/render-d3d11.md) - the follow-up that brings Windows sources onto the GPU -- [VAAPI encode and decode](/quest/m2/video-vaapi.md) - decoded DMA-BUF surfaces the renderer imports -- [Hardware validation](/quest/m3/video-hardware.md) - capture and zero-copy paths not yet run on real machines diff --git a/quest/m2/render-d3d11.md b/quest/m2/render-d3d11.md index 0f3c2648d16..34bb81723f5 100644 --- a/quest/m2/render-d3d11.md +++ b/quest/m2/render-d3d11.md @@ -12,12 +12,13 @@ I420 and re-uploaded. The surface is already there and already GPU-resident: `Surface::Texture` carries a D3D11 NV12 texture, decode retains it, and GPU resize works. What is missing is the import at the other end, which the renderer has for Vulkan -DMA-BUF and Metal but not here, so Windows falls through to the CPU path. +DMA-BUF and Metal but not here, so Windows frames take the CPU path. Import the texture as a wgpu texture through its D3D12 or Vulkan backend -rather than copying, keeping the same per-path fallback and disable behavior -the other import paths use: a machine where the import cannot work should -quietly take the I420 route, not fail to present. +rather than copying. Like the other import paths, the renderer has no +fallback: until this lands it refuses `Surface::Texture`, and +`Output::Render` downloads those frames at the decoder or capture instead. +Landing it adds `Surface::Texture` to what `Output::Render` keeps on the GPU. The renderer is the first consumer of `d3d11::Texture`'s contents outside moq-video's frame module (its fields are `pub(crate)`, diff --git a/quest/m3/studio/README.md b/quest/m3/studio/README.md index 8a71d19fcd8..9abe1e6e1ef 100644 --- a/quest/m3/studio/README.md +++ b/quest/m3/studio/README.md @@ -60,14 +60,12 @@ OBS multistream, and the hosted version is product work outside this repository. Added 2026-10-25 (maintainer): the studio's -GPU encode and -[source import](/quest/m1/studio-gpu/import.md) quests are in m1 so +GPU encode and source import quests are in m1 so moq-video's GPU encode and import API changes, mainly encoders taking a GPU surface and zero-copy texture import, land before the cut. ## Required -- [Zero-copy source import](/quest/m1/studio-gpu/import.md) - captured and decoded frames reach the compositor as GPU textures instead of a CPU upload - [Studio on macOS and Windows](/quest/m3/studio/app.md) - the studio runs on macOS and Windows, not only Linux - [Autosave](/quest/m3/studio/autosave.md) - every edit is written to the scene file, with no save step or quit prompt - [Scene editing](/quest/m3/studio/scene-editing.md) - rename and reorder scenes, and resize the canvas while live through a new rendition diff --git a/quest/m3/studio/packaging.md b/quest/m3/studio/packaging.md index 27aeab839c9..bcf58cf3e9a 100644 --- a/quest/m3/studio/packaging.md +++ b/quest/m3/studio/packaging.md @@ -15,7 +15,9 @@ and a `release` feature turns on every backend its platform can load without one (`capture`, `openh264`, `vaapi`, `nvidia`, and the platform frameworks). Every artifact builds with `release`. `pipewire` stays out of `release` because it links libpipewire at load time, but the Flatpak bundles -libpipewire, so it adds `pipewire` itself. +libpipewire, so it adds `pipewire` itself: inside the sandbox the camera +portal is the only way to a camera, and its PipeWire DMA-BUFs are what the +compositor imports without a copy. ## Required diff --git a/quest/m3/video-embedded.md b/quest/m3/video-embedded.md index d97688dd0a7..19c3a2b0628 100644 --- a/quest/m3/video-embedded.md +++ b/quest/m3/video-embedded.md @@ -12,14 +12,14 @@ records what breaks. Decided 2026-10-08: verify first. The Pi 4 and 5 ship Mesa's v3dv Vulkan driver, so the premise that embedded devices have no usable Vulkan driver is doubtful. Run the renderer's DMA-BUF Vulkan import against V4L2 M2M decoder -output, and the I420 CPU fallback, and record the device, OS image, Mesa +output, and the I420 CPU path, and record the device, OS image, Mesa version, and which path presented. If the Vulkan import cannot take the decoder's buffers (a missing extension or modifier) on a device a consumer needs, plan EGL/GLES import as its own -quest: same shape as the Vulkan path (alias the buffer, keep the per-path -fallback and three-strike disable, fall back to I420 when import fails), and -the usual dlopen-and-degrade rule, so a build without the device present +quest: same shape as the Vulkan path (alias the buffer, refuse what it cannot +import rather than download it, and teach `Output::Render` which buffers it +takes), and the usual dlopen-and-degrade rule, so a build without the device present degrades rather than fails to start. Two items from the same wave are deliberately not here. X11 MIT-SHM capture is diff --git a/quest/m3/video-hardware.md b/quest/m3/video-hardware.md index 7237514ac2e..94ce1c3b0ce 100644 --- a/quest/m3/video-hardware.md +++ b/quest/m3/video-hardware.md @@ -68,7 +68,7 @@ A separate-plane producer belongs to Pi produces nothing, write it down and stop. `doc/lib/rs/moq-video.md` says both paths are reachable; correct that sentence if one cannot capture. -### Windows and Apple GPU probes +### Windows GPU probes Ignored moq-video tests that only a GPU on that platform can run, and that no runner hosts (found by the m0 coverage audit): @@ -83,8 +83,6 @@ runner hosts (found by the m0 coverage audit): capture tests belong to [Windows.Graphics.Capture](/quest/m2/capture-wgc.md). - Media Foundation encode: `mediafoundation_cpu_rgba` and `mediafoundation_camera_texture`. -- Metal import on an Apple Silicon Mac: `the_metal_import_matches_the_cpu_path` - and `imports_survive_decoder_pool_recycling`. ### Fixes diff --git a/rs/justfile b/rs/justfile index e44e76217d1..5a0ec1e914f 100644 --- a/rs/justfile +++ b/rs/justfile @@ -143,6 +143,9 @@ macos-test: just rs test -p moq-sock --features fast-apple-datapath just rs test -p moq-video --no-default-features -E 'test(encode::backend::videotoolbox::tests)' just rs test -p moq-video --no-default-features --features render -E 'test(/^render::export::/)' + # The runner's paravirtual Metal device proves the IOSurface import, ignored elsewhere. + just rs test -p moq-video --no-default-features --features render --lib --run-ignored only \ + -E 'test(/^render::renderer::tests::(the_metal_import_matches_the_cpu_path|imports_survive_decoder_pool_recycling)$/)' # Compile and lint the capture and playback features, which the default check skips. [positional-arguments] @@ -369,7 +372,7 @@ render-sw: icd=("$mesa"/share/vulkan/icd.d/lvp_icd.*.json) LD_LIBRARY_PATH="$loader/lib" VK_DRIVER_FILES="${icd[0]}" \ just rs test -p moq-video --features render --lib --run-ignored only \ - -E 'test(/^render::renderer::tests::(the_declared_color_space_beats_the_size_guess|cpu_frames_survive_the_round_trip|config_size_overrides_the_frame_size)$/)' + -E 'test(/^render::renderer::tests::(the_declared_color_space_beats_the_size_guess|cpu_frames_survive_the_round_trip|config_size_overrides_the_frame_size|an_unimportable_gpu_frame_is_refused|the_output_follows_the_device)$/)' # Mesa's software driver exports no memory, so this covers the readback route. LD_LIBRARY_PATH="$loader/lib" VK_DRIVER_FILES="${icd[0]}" \ just rs test -p moq-video --features render,openh264 --lib --run-ignored only -E 'test(/^render::export::(nv12::)?tests::/)' diff --git a/rs/moq-cli/src/play/fake.rs b/rs/moq-cli/src/play/fake.rs index 453824102f5..c4526cceee7 100644 --- a/rs/moq-cli/src/play/fake.rs +++ b/rs/moq-cli/src/play/fake.rs @@ -85,6 +85,10 @@ impl Output for Recorder { fn send(&self, event: Event) { self.state.lock().unwrap().events.push(event); } + + async fn render(&self) -> moq_video::Output { + moq_video::Output::Cpu + } } impl Speaker for Recorder { diff --git a/rs/moq-cli/src/play/headless.rs b/rs/moq-cli/src/play/headless.rs index 89629f7a9a9..4f70b2c6546 100644 --- a/rs/moq-cli/src/play/headless.rs +++ b/rs/moq-cli/src/play/headless.rs @@ -37,6 +37,11 @@ impl Output for Headless { fn send(&self, event: Event) { let _ = self.events.send(event); } + + /// Never asked: headless playback decodes no video. + async fn render(&self) -> moq_video::Output { + moq_video::Output::Cpu + } } pub(crate) async fn run( diff --git a/rs/moq-cli/src/play/media.rs b/rs/moq-cli/src/play/media.rs index b5eed18d985..8bb79ff0b2e 100644 --- a/rs/moq-cli/src/play/media.rs +++ b/rs/moq-cli/src/play/media.rs @@ -258,7 +258,12 @@ impl Media { // doubles as the staleness budget on the wire. let max_delay = self.args.video_max_delay(&snapshot.audio); let opened = async { - let decoder = moq_video::decode::Sink::open(config, &Default::default()).await?; + // The window's renderer refuses a GPU surface it cannot import, so + // the decoder downloads those instead, all of them on a device + // that imports none. + let mut decode = moq_video::decode::Config::new(); + decode.output = self.output.render().await; + let decoder = moq_video::decode::Sink::open(config, &decode).await?; let track = rendition.track(hang::catalog::Reference::track(config, name))?; let mut subscriber = track .subscribe( diff --git a/rs/moq-cli/src/play/output.rs b/rs/moq-cli/src/play/output.rs index fc533026e0f..8ea206cc92e 100644 --- a/rs/moq-cli/src/play/output.rs +++ b/rs/moq-cli/src/play/output.rs @@ -18,6 +18,9 @@ pub(super) trait Output: Clone + Send + Sync + 'static { /// Tell the window something changed. Dropped if the window is gone. fn send(&self, event: Event); + + /// What to decode video as for the window's renderer, once its device is open. + fn render(&self) -> impl Future + Send; } /// An open speaker, closed once it and every sink from it are dropped. diff --git a/rs/moq-cli/src/play/window.rs b/rs/moq-cli/src/play/window.rs index 35b4691f4c2..81a533b0857 100644 --- a/rs/moq-cli/src/play/window.rs +++ b/rs/moq-cli/src/play/window.rs @@ -65,6 +65,10 @@ pub fn run( // Signals the decoder that the presenter took a frame, so it can hand over // the next one instead of dropping it. let drained = Arc::new(tokio::sync::Notify::new()); + // Set once the window's device opens. The media task starts now anyway, so + // the broadcast resolves while the window does, and waits for it only to + // open a decoder. + let render = Arc::new(tokio::sync::SetOnce::new()); let media = tokio::spawn( Media { @@ -74,7 +78,10 @@ pub fn run( video: video.clone(), presentation: presentation.clone(), drained: drained.clone(), - output: proxy.clone(), + output: Remote { + proxy: proxy.clone(), + render: render.clone(), + }, } .run(), ); @@ -93,7 +100,7 @@ pub fn run( } else { format!("moq play: {broadcast}") }; - let mut app = App::new(title, video, presentation, drained); + let mut app = App::new(title, video, presentation, drained, render); let result = event_loop.run_app(&mut app).context("playback event loop failed"); media.abort(); network.abort(); @@ -106,7 +113,15 @@ pub fn run( } } -impl Output for EventLoopProxy { +/// The window as the media task reaches it. +#[derive(Clone)] +struct Remote { + proxy: EventLoopProxy, + /// What the window's device draws, once it is open. + render: Arc>, +} + +impl Output for Remote { type Speaker = moq_audio::playback::Engine; async fn speaker(&self) -> anyhow::Result { @@ -114,7 +129,11 @@ impl Output for EventLoopProxy { } fn send(&self, event: Event) { - let _ = self.send_event(event); + let _ = self.proxy.send_event(event); + } + + async fn render(&self) -> moq_video::Output { + *self.render.wait().await } } @@ -136,6 +155,7 @@ struct App { video: Arc>>, presentation: Arc>, drained: Arc, + render: Arc>, display: Option, next_redraw: Option, /// The media tasks are done. Keep presenting whatever they left queued, then @@ -152,12 +172,14 @@ impl App { video: Arc>>, presentation: Arc>, drained: Arc, + render: Arc>, ) -> Self { Self { title, video, presentation, drained, + render, display: None, next_redraw: None, ending: false, @@ -242,6 +264,9 @@ impl ApplicationHandler for App { // still resolving, rather than showing whatever was in the surface. Ok(display) => { display.window.request_redraw(); + self.render + .set(moq_video::render::output(&display.device)) + .expect("the display opens once"); self.display = Some(display); } Err(err) => self.fail(event_loop, err), @@ -355,7 +380,7 @@ impl Display { compatible_surface: Some(&surface), apply_limit_buckets: false, }))?; - let (device, queue) = pollster::block_on(adapter.request_device(&wgpu::DeviceDescriptor::default()))?; + let (device, queue) = pollster::block_on(moq_video::render::request_device(&adapter, &Default::default()))?; let mut config = surface .get_default_config(&adapter, size.width.max(1), size.height.max(1)) .context("the graphics adapter cannot present to this window")?; diff --git a/rs/moq-scene/Cargo.toml b/rs/moq-scene/Cargo.toml index 2f5d764476b..1fc6845e0ba 100644 --- a/rs/moq-scene/Cargo.toml +++ b/rs/moq-scene/Cargo.toml @@ -20,6 +20,11 @@ categories = ["multimedia", "multimedia::video", "rendering"] # libasound at build time, and a workspace build would otherwise unify it on for # every crate. Without it, those sources report that the build cannot capture. capture = ["moq-video/capture", "moq-audio/capture"] +# PipeWire cameras on Linux, whose DMA-BUFs the compositor imports rather than +# uploads, and audio devices through the sound server. Off by default like +# moq-video's `pipewire`: it links libpipewire-0.3 at build time, so a Linux +# package turns it on where the build host has it. +pipewire = ["capture", "moq-video/pipewire", "moq-audio/pipewire"] [dependencies] bytemuck = { version = "1", features = ["derive"] } diff --git a/rs/moq-scene/src/compositor.rs b/rs/moq-scene/src/compositor.rs index be96f4b25b0..10c6fcdd656 100644 --- a/rs/moq-scene/src/compositor.rs +++ b/rs/moq-scene/src/compositor.rs @@ -709,7 +709,8 @@ mod tests { async fn new(collection: Collection) -> Self { let (device, queue) = gpu().await; let dir = tempfile::tempdir().unwrap(); - let (context, origin) = context(dir.path()); + let (mut context, origin) = context(dir.path()); + context.output = moq_video::render::output(&device); let mut sources = Sources::new(context); let drivers = sources.sync(&collection); start(&sources, drivers).await; diff --git a/rs/moq-scene/src/source.rs b/rs/moq-scene/src/source.rs index a37ad2dc038..77a5b3155c8 100644 --- a/rs/moq-scene/src/source.rs +++ b/rs/moq-scene/src/source.rs @@ -50,7 +50,6 @@ pub enum Status { /// What every source shares. #[derive(Clone)] -#[non_exhaustive] pub struct Context { /// The program audio that microphones, system audio, and broadcasts feed. pub bus: Bus, @@ -58,14 +57,9 @@ pub struct Context { pub origin: moq_net::origin::Consumer, /// What a relative image path is relative to: the scene file's directory. pub dir: PathBuf, -} - -impl Context { - /// Sources feeding `bus`, subscribing through `origin`, with images - /// relative to `dir`. - pub fn new(bus: Bus, origin: moq_net::origin::Consumer, dir: PathBuf) -> Self { - Self { bus, origin, dir } - } + /// What cameras, screens, and broadcasts deliver video as: + /// `moq_video::render::output` of the compositor's device. + pub output: moq_video::Output, } /// The running sources of a collection. @@ -234,6 +228,7 @@ impl Driver { let mut config = moq_video::capture::Config::default(); config.source = video; + config.output = self.context.output; let mut stream = moq_video::capture::open(&config).await?; while let Some(frame) = stream.read().await? { self.show(Picture::Frame(frame)); @@ -352,6 +347,7 @@ impl Driver { ) -> Result<(), Error> { let mut options = moq_video::decode::Options::new(); options.start = moq_video::decode::Start::Latest; + options.decoder.output = self.context.output; let mut consumer = moq_video::decode::Consumer::new(broadcast, &config, name, options).await?; while let Some(read) = consumer.read().await? { if let moq_video::decode::Read::Frame(frame) = read { @@ -466,13 +462,18 @@ fn image(path: &Path) -> Result { pub(crate) mod test { use super::*; - /// A context with no relay behind it, so broadcasts stay offline while the - /// returned origin lives. Call from a [`tokio::task::LocalSet`]. + /// A context with no relay or device behind it, so broadcasts stay offline + /// while the returned origin lives. Call from a [`tokio::task::LocalSet`]. pub(crate) fn context(dir: &Path) -> (Context, moq_net::origin::Producer) { let (origin, driver) = moq_net::origin::Producer::new(Default::default()); tokio::task::spawn_local(moq_net::time::run(driver)); - let bus = Bus::new(Default::default()).unwrap(); - (Context::new(bus, origin.consume(), dir.to_path_buf()), origin) + let context = Context { + bus: Bus::new(Default::default()).unwrap(), + origin: origin.consume(), + dir: dir.to_path_buf(), + output: moq_video::Output::Cpu, + }; + (context, origin) } /// Run `drivers` on the local task set, and wait until every source in diff --git a/rs/moq-studio/Cargo.toml b/rs/moq-studio/Cargo.toml index 824ebc7d333..348f39d50f2 100644 --- a/rs/moq-studio/Cargo.toml +++ b/rs/moq-studio/Cargo.toml @@ -30,9 +30,10 @@ nvidia = ["moq-video/nvidia"] openh264 = ["moq-video/openh264"] # Intel/AMD VAAPI hardware codecs (Linux); its bindgen needs libclang. vaapi = ["moq-video/vaapi"] -# PipeWire cameras and audio. Wayland screens and windows are not reachable yet: -# display and window sources capture X11 only. -pipewire = ["capture", "moq-video/pipewire", "moq-audio/pipewire"] +# PipeWire cameras, drawn without a copy, and audio through the sound server. +# Wayland screens and windows are not reachable yet: display and window sources +# capture X11 only. +pipewire = ["capture", "moq-scene/pipewire"] [dependencies] anyhow = { workspace = true, features = ["backtrace"] } diff --git a/rs/moq-studio/src/engine.rs b/rs/moq-studio/src/engine.rs index 56eb84372ec..d2665a04a00 100644 --- a/rs/moq-studio/src/engine.rs +++ b/rs/moq-studio/src/engine.rs @@ -204,7 +204,12 @@ impl Engine { let bus = moq_audio::bus::Bus::new(Default::default())?; let subscribe = moq_tokio::origin::spawn(); let dir = path.parent().map(Path::to_path_buf).unwrap_or_default(); - let sources = Sources::new(source::Context::new(bus.clone(), subscribe.consume(), dir)); + let sources = Sources::new(source::Context { + bus: bus.clone(), + origin: subscribe.consume(), + dir, + output: moq_video::render::output(&gpu.device), + }); let mut compositor = Compositor::new(&gpu.device, &gpu.queue, collection.canvas)?; compositor.set_preview(collection.scenes.first().map(|scene| scene.name.clone())); compositor.cut(); diff --git a/rs/moq-video/MIGRATE.md b/rs/moq-video/MIGRATE.md index 32dc9dc1359..e9d0f43619d 100644 --- a/rs/moq-video/MIGRATE.md +++ b/rs/moq-video/MIGRATE.md @@ -4,6 +4,17 @@ How to update your code for each breaking change, newest first. ## Unreleased +### The renderer refuses GPU frames it cannot import (#6806) + +`render::Renderer::render` no longer downloads a GPU surface it cannot import; it returns an error naming the cause and the fix. +`render::Config::zero_copy` is gone, along with the retry-and-retire fallback behind it. +CPU frames (`Surface::I420`) upload as before. +Decode and capture with `render::output(&device)` (`decode::Config::output`, `capture::Config::output`) to receive only frames the renderer on that device draws. +It picks `Output::Render`, which keeps the surfaces this platform imports on the GPU and downloads the rest, for a device that can import, and `Output::Cpu` for one that cannot (wgpu's GL backend, or a Linux device without DMA-BUF import). +Either errors on a surface with no download, such as a non-NV12 pixel buffer. +On Linux, open the device with `wgpu::Features::VULKAN_EXTERNAL_MEMORY_DMA_BUF` where the adapter has it, or `render::output` picks the CPU. +To force the CPU path that `zero_copy: false` gave, decode or capture with `Output::Cpu`. + ### `Rate` is `FrameRate` (#6653) `Rate` and `RateError` are `FrameRate` and `FrameRateError`, the name the bindings already use. diff --git a/rs/moq-video/README.md b/rs/moq-video/README.md index cd3e3de050d..a5cff96caac 100644 --- a/rs/moq-video/README.md +++ b/rs/moq-video/README.md @@ -224,9 +224,14 @@ refused rather than narrowed to 8-bit 4:2:0. Any other rendition yields `decode::Config::output` says where decoded pictures live: `Output::Native` (the default) hands back whatever the backend decoded into, a GPU surface or -CPU pixels, and `Output::Cpu` delivers every picture as `Surface::I420`, +CPU pixels, `Output::Cpu` delivers every picture as `Surface::I420`, decoded straight to system memory where the backend can and downloaded where it -cannot. `decode::Config::scale_hint` asks a decoder with a hardware scaler +cannot, and `Output::Render` keeps only the surfaces `render::Renderer` imports +and downloads the rest. The renderer refuses any other GPU surface, so a player +decodes with `render::output(&device)`, which picks `Output::Render` for a +device that can import and `Output::Cpu` for one that cannot (wgpu's GL +backend, or Linux without DMA-BUF import). `capture::Config::output` makes the same choice +for captured frames. `decode::Config::scale_hint` asks a decoder with a hardware scaler (NVDEC) to emit that size; it is a hint, so check `Frame::size` and use `Frame::resize` for the exact size. `decode::Consumer` takes `decode::Options`, which pairs that config with the subscription's `start` and `max_delay`. diff --git a/rs/moq-video/src/capture/mod.rs b/rs/moq-video/src/capture/mod.rs index 5b46d95982f..5f017b2ad1b 100644 --- a/rs/moq-video/src/capture/mod.rs +++ b/rs/moq-video/src/capture/mod.rs @@ -13,7 +13,7 @@ use std::sync::Arc; -use crate::{Error, Frame, FrameRate}; +use crate::{Error, Frame, FrameRate, Output}; mod channel; use channel::FrameChannel; @@ -267,6 +267,9 @@ pub struct Config { /// Draw the mouse cursor into captured frames. Screen/window/app sources /// only; ignored by cameras. Defaults to `true`. pub cursor: bool, + /// Where captured pixels live. Defaults to [`Output::Native`]: whatever the + /// platform captures into, a GPU surface or CPU pixels. + pub output: Output, } impl Default for Config { @@ -277,6 +280,7 @@ impl Default for Config { height: None, framerate: None, cursor: true, + output: Output::Native, } } } @@ -296,6 +300,8 @@ pub struct Stream { framerate: Option, color: Option, label: String, + /// [`Config::output`], applied to each frame as it is read. + output: Output, /// First frame captured during [`open`] (some backends learn their geometry /// only from a frame); returned by the first [`read`](Self::read). pending: Option, @@ -323,6 +329,7 @@ impl Stream { framerate, color, label, + output: Output::Native, pending, _backend: backend, } @@ -338,10 +345,15 @@ impl Stream { /// Dropping this future cancels only the pending read. Dropping the stream /// releases the capture source. pub async fn read(&mut self) -> Result, Error> { - if let Some(frame) = self.pending.take() { - return Ok(Some(frame)); - } - self.chan.recv().await + let frame = match self.pending.take() { + Some(frame) => frame, + None => match self.chan.recv().await? { + Some(frame) => frame, + None => return Ok(None), + }, + }; + let surface = self.output.convert(frame.surface)?; + Ok(Some(Frame::new(surface, frame.timestamp))) } /// Width of the captured frames in pixels. @@ -417,6 +429,12 @@ impl Synthetic { /// Open the capture source described by `config`. pub async fn open(config: &Config) -> Result { + let mut stream = open_source(config).await?; + stream.output = config.output; + Ok(stream) +} + +async fn open_source(config: &Config) -> Result { match &config.source { Source::Camera(device) => { let _ = device; diff --git a/rs/moq-video/src/decode/backend/vaapi.rs b/rs/moq-video/src/decode/backend/vaapi.rs index 3ffed8f982f..58e4b05fbc1 100644 --- a/rs/moq-video/src/decode/backend/vaapi.rs +++ b/rs/moq-video/src/decode/backend/vaapi.rs @@ -87,7 +87,13 @@ impl Vaapi { }; let decoder = Decoder::new(vaapi).map_err(|e| Error::Codec(anyhow::anyhow!("VAAPI decoder init: {e:?}")))?; - let exporting = config.output == Output::Native; + // The renderer imports the NV12 this exports, so `Render` keeps it too + // in a build that has the renderer; without one it would only download. + let exporting = match config.output { + Output::Native => true, + Output::Cpu => false, + Output::Render => cfg!(feature = "render"), + }; tracing::info!(decoder = NAME, exporting, "opened H.264 decoder"); Ok(Box::new(Self { decoder, diff --git a/rs/moq-video/src/decode/decoder.rs b/rs/moq-video/src/decode/decoder.rs index 98c7b4350b7..7010abe83be 100644 --- a/rs/moq-video/src/decode/decoder.rs +++ b/rs/moq-video/src/decode/decoder.rs @@ -30,7 +30,7 @@ use moq_mux::codec::{annexb, h264, h265}; use moq_net::Timestamp; use super::backend::{self, Backend, Codec}; -use crate::{Color, Error, Frame, Output, Size, Surface}; +use crate::{Color, Error, Frame, Output, Size}; /// Which decoder implementation to use. `#[non_exhaustive]` so new selection /// strategies can be added without breaking external `match`es. @@ -72,7 +72,10 @@ pub struct Config { /// decode straight to system memory does, the rest download each picture. /// Ask for it when the pixels are headed for the CPU anyway, since a GPU /// surface handed out and downloaded later costs an allocation the backend - /// could have skipped. + /// could have skipped. [`Output::Render`] keeps the surfaces + /// `render::Renderer` imports and downloads the rest; a player that draws + /// asks `render::output` of its device, which picks it where the device + /// imports. pub output: Output, /// Ask the decoder to scale its output to this size (both dimensions even) /// instead of the stream's native one. @@ -303,10 +306,7 @@ impl Decoder { self.color = color; self.switches.pop_front(); } - let mut surface = match self.output { - Output::Native => frame.surface, - Output::Cpu => Surface::I420(frame.surface.into_i420()?), - }; + let mut surface = self.output.convert(frame.surface)?; if let Some(color) = self.color { surface.label(color); } diff --git a/rs/moq-video/src/frame.rs b/rs/moq-video/src/frame.rs index cd057bf1c48..a14adc693df 100644 --- a/rs/moq-video/src/frame.rs +++ b/rs/moq-video/src/frame.rs @@ -551,8 +551,9 @@ impl Surface { /// labelled with that: scaling across 576 lines would otherwise change the /// guess without changing the samples. pub fn resize(&self, size: Size, config: &crate::resize::Config) -> Result { + let output = config.output.resolve(self); // Counts as a use on builds where every GPU arm is compiled out. - let _ = config; + let _ = output; size.validate("resize to")?; let implied = self @@ -561,9 +562,7 @@ impl Surface { let mut resized = match self { Surface::I420(i420) => Surface::I420(i420.resize(size)?), #[cfg(apple)] - Surface::PixelBuffer(pixels) if config.output == crate::Output::Cpu => { - Surface::I420(pixels.download_i420()?.resize(size)?) - } + Surface::PixelBuffer(pixels) if output == crate::Output::Cpu => Surface::I420(pixels.download_i420()?.resize(size)?), #[cfg(apple)] Surface::PixelBuffer(pixels) => match pixels.resize(size.width, size.height) { Ok(scaled) => Surface::PixelBuffer(scaled), @@ -576,7 +575,7 @@ impl Surface { } }, #[cfg(all(target_os = "linux", feature = "nvidia"))] - Surface::Cuda(cuda) if config.output == crate::Output::Cpu => Surface::I420(cuda.download_i420()?.resize(size)?), + Surface::Cuda(cuda) if output == crate::Output::Cpu => Surface::I420(cuda.download_i420()?.resize(size)?), #[cfg(all(target_os = "linux", feature = "nvidia"))] Surface::Cuda(cuda) => match cuda.resize(size) { Ok(scaled) => Surface::Cuda(scaled), @@ -595,9 +594,7 @@ impl Surface { )); } #[cfg(target_os = "windows")] - Surface::Texture(texture) if config.output == crate::Output::Cpu => { - Surface::I420(texture.download_i420()?.resize(size)?) - } + Surface::Texture(texture) if output == crate::Output::Cpu => Surface::I420(texture.download_i420()?.resize(size)?), #[cfg(target_os = "windows")] Surface::Texture(texture) => match texture.resize(size.width, size.height) { Ok(scaled) => Surface::Texture(scaled), @@ -615,7 +612,7 @@ impl Surface { // encoder to import. If VA-API is missing or refuses the buffer, // download and resize on the CPU. #[cfg(all(target_os = "linux", feature = "vaapi"))] - Surface::DmaBuf(buffer) if config.output == crate::Output::Native => match vaapi::resize(buffer, size) { + Surface::DmaBuf(buffer) if output == crate::Output::Native => match vaapi::resize(buffer, size) { Ok(scaled) => Surface::DmaBuf(scaled), Err(err) => { static WARN_ONCE: std::sync::Once = std::sync::Once::new(); diff --git a/rs/moq-video/src/output.rs b/rs/moq-video/src/output.rs index 42f50ddadf6..8179c58b5f9 100644 --- a/rs/moq-video/src/output.rs +++ b/rs/moq-video/src/output.rs @@ -1,6 +1,8 @@ //! Where a frame's pixels live once an operation hands it back. -/// The representation a decoder or scaler hands frames back in. +use crate::{Error, Surface}; + +/// The representation a decoder, capture, or scaler hands frames back in. /// /// A choice, not a promise of GPU residency. [`Native`](Self::Native) lets a /// hardware backend keep its picture where it made it, which on a software @@ -19,6 +21,82 @@ pub enum Output { Native, /// CPU-resident [`I420`](crate::I420), whatever produced it. A hardware /// decode pays a download per picture; a backend that can decode straight - /// to system memory does that instead. + /// to system memory does that instead. A surface with no download, such as + /// a non-NV12 pixel buffer, is an error (see + /// [`Surface::into_i420`](crate::Surface::into_i420)). Cpu, + /// What `render::Renderer` draws: a GPU surface it imports on this + /// platform stays one, and anything else is converted as [`Cpu`](Self::Cpu) + /// converts it, an error included. Without the `render` feature that is + /// every frame. + /// + /// The renderer refuses the GPU surfaces it cannot import, so a source + /// feeding one asks for this instead of guessing which backend it will get. + /// Says nothing about the device, which may be unable to import at all: + /// `render::output` picks this only for a device that can, and + /// [`Cpu`](Self::Cpu) otherwise. + Render, +} + +impl Output { + /// What this asks of `surface`: [`Native`](Self::Native) or [`Cpu`](Self::Cpu). + pub(crate) fn resolve(self, surface: &Surface) -> Self { + match self { + Self::Render if renders(surface) => Self::Native, + Self::Render => Self::Cpu, + other => other, + } + } + + /// `surface` in this representation, downloaded if it has to be. + pub(crate) fn convert(self, surface: Surface) -> Result { + Ok(match self.resolve(&surface) { + Self::Cpu => Surface::I420(surface.into_i420()?), + _ => surface, + }) + } +} + +fn renders(surface: &Surface) -> bool { + #[cfg(feature = "render")] + { + crate::render::imports(surface) + } + #[cfg(not(feature = "render"))] + { + let _ = surface; + false + } +} + +#[cfg(all(test, target_os = "linux", feature = "render"))] +mod tests { + use super::*; + use crate::Size; + use crate::frame::{DmaBuf, DmaBufLayout, DmaBufPlane, DrmFormat}; + + fn dmabuf(format: DrmFormat, planes: usize) -> Surface { + let planes = (0..planes) + .map(|index| DmaBufPlane::new(index as u32 * 64, 8)) + .collect(); + let layout = DmaBufLayout::new(format, Size::new(8, 8), planes); + let fd = std::fs::File::open("/dev/zero").expect("open /dev/zero").into(); + Surface::DmaBuf(DmaBuf::new(fd, layout, ()).expect("a well-formed DMA-BUF")) + } + + /// `Render` keeps exactly the DMA-BUFs the renderer imports: a format it + /// has no shape for would be refused there, so it must arrive on the CPU. + #[test] + fn render_keeps_only_importable_dma_bufs() { + for (format, planes) in [(DrmFormat::NV12, 2), (DrmFormat::YUV420, 3), (DrmFormat::XRGB8888, 1)] { + assert_eq!( + Output::Render.resolve(&dmabuf(format, planes)), + Output::Native, + "{format:?}" + ); + } + let yuyv = dmabuf(DrmFormat::from_bytes(*b"YUYV"), 1); + assert_eq!(Output::Render.resolve(&yuyv), Output::Cpu); + assert_eq!(Output::Native.resolve(&yuyv), Output::Native, "Native never downloads"); + } } diff --git a/rs/moq-video/src/render/dmabuf.rs b/rs/moq-video/src/render/dmabuf.rs index 9268b91de9f..0a8bda56303 100644 --- a/rs/moq-video/src/render/dmabuf.rs +++ b/rs/moq-video/src/render/dmabuf.rs @@ -15,10 +15,10 @@ //! reformatted; only the descriptor is duplicated, once per plane. //! //! What that leaves is the modifier itself. A buffer whose modifier the driver -//! lists for neither format has nowhere to go at all, and the import fails so -//! the renderer downloads the frame instead. Nothing here re-tiles such a -//! buffer into a layout Vulkan would take: on the hardware this was measured on, -//! a VA-API decode target already lands on an importable modifier. +//! lists for neither format has nowhere to go at all, and the import fails. +//! Nothing here re-tiles such a buffer into a layout Vulkan would take: on the +//! hardware this was measured on, a VA-API decode target already lands on an +//! importable modifier. use std::os::fd::{AsFd, BorrowedFd}; @@ -32,21 +32,11 @@ fn err(message: impl std::fmt::Display) -> Error { } /// Alias one DMA-BUF allocation as the sampled Vulkan textures of a frame. -pub(super) fn import(device: &wgpu::Device, buffer: &DmaBuf) -> Result, Error> { - if !device - .features() - .contains(wgpu::Features::VULKAN_EXTERNAL_MEMORY_DMA_BUF) - { - return Ok(None); - } - - // A device carrying the feature above is Vulkan-backed, so this is not a - // route anything takes. It is here for what it returns rather than for what - // it catches: `Ok(None)` sends the frame to the CPU path without the strike - // an `Err` from the import itself would cost. - // SAFETY: the guard is only asked whether it exists, and drops here. - if unsafe { device.as_hal::() }.is_none() { - return Ok(None); +pub(super) fn import(device: &wgpu::Device, buffer: &DmaBuf) -> Result { + if !supports(device) { + return Err(err( + "the wgpu device cannot import DMA-BUFs: open it on Vulkan with Features::VULKAN_EXTERNAL_MEMORY_DMA_BUF, or decode or capture with render::output(&device)", + )); } let size = Size::new(buffer.width(), buffer.height()); @@ -82,7 +72,21 @@ pub(super) fn import(device: &wgpu::Device, buffer: &DmaBuf) -> Result bool { + // The feature is only offered on Vulkan, so it also proves the backend. + device + .features() + .contains(wgpu::Features::VULKAN_EXTERNAL_MEMORY_DMA_BUF) +} + +/// Whether [`import`] takes a buffer of `format`. +pub(super) fn imports(format: DrmFormat) -> bool { + // Any even size will do: only the format decides whether there is a shape. + Shape::of(format, Size::new(2, 2)).is_ok() } /// How a DRM format is sampled: the shader layout it feeds, and the Vulkan @@ -136,7 +140,12 @@ impl Shape { plane2: Some((wgpu::TextureFormat::R8Unorm, half)), } } - format => return Err(err(format!("cannot import DMA-BUF format {:#x}", format.as_raw()))), + format => { + return Err(err(format!( + "cannot import DMA-BUF format {:#x}; decode or capture with render::output(&device), which delivers it as CPU frames", + format.as_raw() + ))); + } }) } diff --git a/rs/moq-video/src/render/metal.rs b/rs/moq-video/src/render/metal.rs index c673669746a..060a8bf7bc0 100644 --- a/rs/moq-video/src/render/metal.rs +++ b/rs/moq-video/src/render/metal.rs @@ -10,9 +10,7 @@ //! without it (a plain `CVPixelBufferCreate`, as //! [`Surface::into_pixel_buffer`](crate::Surface::into_pixel_buffer) does when //! uploading a CPU frame) has no handle Metal can address, and the import fails. -//! Capture and decoder pools always set it, so frames off the fast path arrive -//! importable; anything else lands on the CPU fallback, which is the right -//! answer for pixels that were on the CPU to begin with. +//! Capture and decoder pools always set it, so their frames arrive importable. use std::ptr::NonNull; @@ -58,6 +56,32 @@ pub(super) struct Keepalive( unsafe impl Send for Keepalive {} unsafe impl Sync for Keepalive {} +/// The layout and range (video swing vs full swing) a pixel format imports as, +/// or `None` when it does not. +fn layout_of(format: u32) -> Option<(Layout, Range)> { + match format { + f if f == kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange => Some((Layout::Nv12, Range::Limited)), + f if f == kCVPixelFormatType_420YpCbCr8BiPlanarFullRange => Some((Layout::Nv12, Range::Full)), + // Rare: a pool configured for planar output rather than the NV12 every + // hardware decoder produces. Cheap to support and it keeps the import + // from being NV12-only by construction. + f if f == kCVPixelFormatType_420YpCbCr8Planar => Some((Layout::I420, Range::Limited)), + _ => None, + } +} + +/// Whether [`Import::import`] takes `buffer`'s pixel format. +pub(super) fn imports(buffer: &PixelBuffer) -> bool { + layout_of(CVPixelBufferGetPixelFormatType(buffer.buffer())).is_some() +} + +/// Whether [`Import::new`] takes `device`: only a Metal one has the `MTLDevice` +/// the texture cache binds to. +pub(super) fn supports(device: &wgpu::Device) -> bool { + // SAFETY: the handle is only checked for presence and dropped at once. + unsafe { device.as_hal::() }.is_some() +} + /// A `CVMetalTextureCache` on `device`'s `MTLDevice`, making every texture /// with `textures` as its attributes. /// @@ -110,8 +134,8 @@ unsafe impl Send for Import {} impl Import { /// Build a texture cache on the same `MTLDevice` wgpu is rendering with. /// - /// Errors when the device is not a Metal one, which is how a caller running - /// wgpu's Vulkan or GL backend on macOS lands on the CPU path. + /// Errors when the device is not a Metal one, for which + /// [`output`](super::output) picks `Output::Cpu`. pub fn new(device: &wgpu::Device) -> Result { Ok(Self { cache: texture_cache(device, None)?, @@ -122,17 +146,10 @@ impl Import { pub fn import(&mut self, device: &wgpu::Device, buffer: &PixelBuffer) -> Result { let size = Size::new(buffer.width(), buffer.height()); let format = CVPixelBufferGetPixelFormatType(buffer.buffer()); - - // The pixel format names the layout and the range (video swing vs full - // swing). - let (layout, range) = match format { - f if f == kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange => (Layout::Nv12, Range::Limited), - f if f == kCVPixelFormatType_420YpCbCr8BiPlanarFullRange => (Layout::Nv12, Range::Full), - // Rare: a pool configured for planar output rather than the NV12 - // every hardware decoder produces. Cheap to support and it keeps the - // import from being NV12-only by construction. - f if f == kCVPixelFormatType_420YpCbCr8Planar => (Layout::I420, Range::Limited), - f => return Err(err(format!("cannot import pixel format {f:#x}"))), + let Some((layout, range)) = layout_of(format) else { + return Err(err(format!( + "cannot import pixel format {format:#x}; decode or capture with render::output(&device), which delivers it as CPU frames" + ))); }; // The matrix comes off the buffer's own attachment, which VideoToolbox diff --git a/rs/moq-video/src/render/mod.rs b/rs/moq-video/src/render/mod.rs index 6b8caa3c852..900ad7279b1 100644 --- a/rs/moq-video/src/render/mod.rs +++ b/rs/moq-video/src/render/mod.rs @@ -35,11 +35,11 @@ // link to it fails the `-D warnings` rustdoc build on every other platform. //! //! Linux callers open the device with [`request_device`], which enables -//! [`wgpu::Features::VULKAN_EXTERNAL_MEMORY_DMA_BUF`] for that import. Every other frame, and any import that fails, goes through -//! [`Surface::into_i420`](crate::Surface::into_i420) and a plane upload. That -//! path is always available, so which route a frame takes is a question of cost. -//! An import path that keeps failing (a driver that cannot do it at all) retires -//! itself after a few frames rather than paying for the attempt forever. +//! [`wgpu::Features::VULKAN_EXTERNAL_MEMORY_DMA_BUF`] for that import. A CPU +//! frame is uploaded as planes. Any other GPU surface, and any import that +//! fails, is an error rather than a download, so a frame's path is settled +//! where it is produced: decode or capture with [`output`] of the device to +//! receive only frames that draw here. //! //! ## Export //! @@ -55,6 +55,9 @@ pub mod export; mod renderer; mod source; +pub(crate) use source::imports; +pub use source::output; + #[cfg(all(target_os = "linux", feature = "dmabuf"))] mod dmabuf; diff --git a/rs/moq-video/src/render/renderer.rs b/rs/moq-video/src/render/renderer.rs index 4ece126e31c..818f46612e9 100644 --- a/rs/moq-video/src/render/renderer.rs +++ b/rs/moq-video/src/render/renderer.rs @@ -2,27 +2,7 @@ use super::color::uniform; use super::source::{self, Layout, Source}; -use crate::{Color, Error, Frame, Size}; - -/// How many consecutive zero-copy import failures retire the fast path. -/// -/// One failure is usually transient (a pool ran dry, a surface arrived in an -/// unexpected format). A driver that cannot do the import at all fails every -/// time, and retrying it per frame forever costs an allocation and a log line at -/// frame rate, so the path is retired and the CPU fallback takes over. -const ZERO_COPY_STRIKES: u32 = 3; - -#[cfg(all(target_os = "linux", feature = "dmabuf"))] -fn dma_buf_import_timed_out(error: &Error) -> bool { - let Error::Render(error) = error else { - return false; - }; - error.chain().any(|cause| { - cause - .downcast_ref::() - .is_some_and(|error| error.kind() == std::io::ErrorKind::TimedOut) - }) -} +use crate::{Color, Error, Frame, Size, Surface}; /// Renderer configuration. /// @@ -65,14 +45,6 @@ pub struct Config { /// whenever it came off the wire: the authoritative answer is in the /// bitstream's VUI and does not survive decoding. pub color: Option, - - /// Whether to import GPU frames zero-copy (aliasing the decoder's surface as - /// a texture) instead of downloading them to the CPU first. On by default. - /// - /// A failing import path retires itself after a few strikes, so this is for - /// forcing the CPU path deliberately: comparing output, or working around a - /// driver without rebuilding. - pub zero_copy: bool, } impl Default for Config { @@ -82,14 +54,13 @@ impl Default for Config { format: wgpu::TextureFormat::Rgba8Unorm, usage: wgpu::TextureUsages::empty(), color: None, - zero_copy: true, } } } impl Config { /// A default config: output at each frame's own size, `Rgba8Unorm`, inferred - /// color space, zero-copy on. + /// color space. pub fn new() -> Self { Self::default() } @@ -101,10 +72,11 @@ impl Config { /// [`render`](Self::render) hands back a plain [`wgpu::Texture`], so what draws /// it (a swapchain blit, an egui image, a bevy material) stays entirely yours. /// -/// A GPU frame is imported without a round trip through the CPU where the -/// platform allows it, and any frame the fast paths do not recognize falls back -/// to uploading [`Surface::into_i420`](crate::Surface::into_i420). So which path -/// a frame takes is a question of cost, not of whether it draws at all. +/// A CPU frame ([`Surface::I420`](crate::Surface::I420)) is uploaded. A GPU +/// frame is imported without a round trip through the CPU, and one this +/// platform has no importer for is an error rather than a hidden download: ask +/// the decoder or capture for [`output`](super::output) of the device and it +/// only hands over frames that draw here. /// /// One renderer draws one video. It caches the pipeline, the plane textures and /// the output texture, so keep it alive across frames rather than rebuilding it @@ -123,14 +95,6 @@ pub struct Renderer { source: source::Cache, output: Option, - - /// Consecutive zero-copy import failures, up to [`ZERO_COPY_STRIKES`]. - strikes: u32, - /// Set once the fast path is retired for the life of this renderer. - retired: bool, - /// Set once a frame has been imported rather than uploaded, so the line - /// saying so is logged once rather than per frame. - imported: bool, } /// The compiled pipeline, one variant per plane layout, and everything they share. @@ -211,9 +175,9 @@ impl Renderer { /// /// The device and queue are the application's: the renderer draws into /// textures that application already owns, so it never creates a device of - /// its own. Both handles are cheap to clone and are kept. On Linux, request - /// [`wgpu::Features::VULKAN_EXTERNAL_MEMORY_DMA_BUF`] on the device to import - /// PipeWire DMA-BUFs instead of downloading them. + /// its own. Both handles are cheap to clone and are kept. On Linux, a device + /// that will be handed DMA-BUFs needs + /// [`wgpu::Features::VULKAN_EXTERNAL_MEMORY_DMA_BUF`]. pub fn new(device: &wgpu::Device, queue: &wgpu::Queue, config: Config) -> Result { if let Some(size) = config.size { size.validate_nonzero("render output")?; @@ -238,9 +202,6 @@ impl Renderer { color: None, source: source::Cache::default(), output: None, - strikes: 0, - retired: false, - imported: false, }) } @@ -249,6 +210,11 @@ impl Renderer { /// The returned handle aliases a texture the renderer reuses, so the next /// call overwrites what you are holding. Present or copy it before rendering /// again. + /// + /// Errors on a GPU frame this renderer cannot import: a surface kind with no + /// importer on this platform, a pixel format its importer cannot alias, a + /// device opened without the feature the import needs, or an import the + /// driver refuses. The message names the fix. pub fn render(&mut self, frame: &Frame) -> Result { let mut source = self.source(frame)?; if let Some(source_color) = source.color { @@ -334,44 +300,13 @@ impl Renderer { Ok(output) } - /// Turn a frame into plane textures, preferring a zero-copy import and - /// falling back to a CPU upload. + /// Turn a frame into plane textures: an upload for a CPU frame, an import + /// for a GPU one. fn source(&mut self, frame: &Frame) -> Result { - if self.config.zero_copy && !self.retired { - match self.source.import(&self.device, &frame.surface) { - Ok(Some(source)) => { - self.strikes = 0; - // The one observable difference between a picture that - // reached the GPU untouched and one that went through system - // memory. Worth a line the first time it happens, because - // nothing else about a working renderer says which it was. - if !self.imported { - self.imported = true; - tracing::debug!( - layout = ?source.layout, - "drawing frames zero-copy; the picture reaches the GPU without a download" - ); - } - return Ok(source); - } - // No import path for this surface on this platform. Not a - // failure, so it costs no strike: the CPU path is the answer - // for this surface and always will be. - Ok(None) => {} - #[cfg(all(target_os = "linux", feature = "dmabuf"))] - Err(err) if dma_buf_import_timed_out(&err) => return Err(err), - Err(err) => { - self.strikes += 1; - self.retired = self.strikes >= ZERO_COPY_STRIKES; - match self.retired { - true => tracing::warn!(%err, "zero-copy import failed repeatedly; using the CPU path"), - false => tracing::debug!(%err, "zero-copy import failed; falling back to the CPU"), - } - } - } + match &frame.surface { + Surface::I420(i420) => self.source.upload(&self.device, &self.queue, i420), + surface => self.source.import(&self.device, surface), } - - self.source.upload(&self.device, &self.queue, frame) } /// The output texture, recreated when the size it should have changes. @@ -528,18 +463,71 @@ mod tests { use moq_net::Timestamp; use super::*; - use crate::Surface; + /// A GPU frame the renderer cannot import is an error that names the fix, + /// never a download. A DMA-BUF on a device opened without the import feature + /// is the case every Linux device can stage. + #[cfg(all(target_os = "linux", feature = "dmabuf"))] + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn an_unimportable_gpu_frame_is_refused() { + use crate::frame::{DmaBuf, DmaBufLayout, DmaBufPlane}; + + let (device, queue) = gpu().await; + assert!( + !device + .features() + .contains(wgpu::Features::VULKAN_EXTERNAL_MEMORY_DMA_BUF) + ); + let mut renderer = Renderer::new(&device, &queue, Config::new()).expect("a renderer"); + + let layout = DmaBufLayout::new( + crate::DrmFormat::NV12, + Size::new(8, 8), + vec![DmaBufPlane::new(0, 8), DmaBufPlane::new(64, 8)], + ); + let fd = std::fs::File::open("/dev/zero").expect("open /dev/zero").into(); + let buffer = DmaBuf::new(fd, layout, ()).expect("a well-formed DMA-BUF"); + let error = renderer + .render(&Frame::new(Surface::DmaBuf(buffer), Timestamp::ZERO)) + .expect_err("a DMA-BUF this device cannot import"); + assert!( + error.to_string().contains("VULKAN_EXTERNAL_MEMORY_DMA_BUF"), + "the error names the missing feature: {error}" + ); + + // The renderer is not poisoned: the next CPU frame still draws. + renderer + .render(&solid(Size::new(8, 8), [255, 0, 0, 255])) + .expect("a CPU frame after the refusal"); + } + + /// [`output`](super::super::output) keeps GPU frames only for a device that + /// imports them, so a source feeding a device without the import feature + /// decodes to the CPU instead of handing over DMA-BUFs it would refuse. #[cfg(all(target_os = "linux", feature = "dmabuf"))] - #[test] - fn dma_buf_fence_timeout_is_terminal_for_the_frame() { - let timed_out = Error::Render(anyhow::Error::new(std::io::Error::from(std::io::ErrorKind::TimedOut))); - let other = Error::Render(anyhow::Error::new(std::io::Error::from( - std::io::ErrorKind::PermissionDenied, - ))); - - assert!(dma_buf_import_timed_out(&timed_out)); - assert!(!dma_buf_import_timed_out(&other)); + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just rs render-sw"] + async fn the_output_follows_the_device() { + let instance = wgpu::Instance::default(); + let adapter = instance + .request_adapter(&wgpu::RequestAdapterOptions::default()) + .await + .expect("a GPU adapter"); + let open = |features| { + adapter.request_device(&wgpu::DeviceDescriptor { + required_features: features, + ..Default::default() + }) + }; + + let (without, _queue) = open(wgpu::Features::empty()).await.expect("a GPU device"); + assert_eq!(super::super::output(&without), crate::Output::Cpu); + + let import = wgpu::Features::VULKAN_EXTERNAL_MEMORY_DMA_BUF; + assert!(adapter.features().contains(import), "the adapter imports DMA-BUFs"); + let (with, _queue) = open(import).await.expect("a DMA-BUF-capable GPU device"); + assert_eq!(super::super::output(&with), crate::Output::Render); } /// Capture packed PipeWire DMA-BUFs, import them through Vulkan, and turn @@ -609,14 +597,11 @@ mod tests { let imported = renderer .source .import(&device, &frame.surface) - .expect("Vulkan DMA-BUF import") - .expect("a DMA-BUF import path"); + .expect("Vulkan DMA-BUF import"); assert_eq!(imported.layout, Layout::Rgba); drop(imported); let texture = renderer.render(&frame).expect("a zero-copy rendered frame"); - assert_eq!(renderer.strikes, 0, "frame {index} fell back to the CPU"); - assert!(!renderer.retired, "frame {index} retired the zero-copy path"); assert_eq!((texture.width(), texture.height()), (stream.width(), stream.height())); } @@ -972,7 +957,7 @@ mod tests { /// /// - The import returns a source at all and its layout is `expected`. Only /// the DMA-BUF importer produces those layouts, so this says which branch - /// ran. Without it the CPU fallback would quietly satisfy everything below. + /// ran. /// - Every pixel matches the CPU upload of the same samples. That is the one /// that catches a swapped chroma pair, a plane at the wrong offset, an /// ignored row pitch, and a mismatched matrix, all at once. @@ -1031,15 +1016,12 @@ mod tests { let imported = renderer .source .import(device, &frame.surface) - .expect("import the DMA-BUF") - .expect("a DMA-BUF import path"); + .expect("import the DMA-BUF"); assert_eq!(imported.layout, expected, "the frame imported as the wrong layout"); assert_eq!(imported.color, Some(color)); drop(imported); let texture = renderer.render(&frame).expect("a rendered frame"); - assert_eq!(renderer.strikes, 0, "the zero-copy import should not have failed"); - assert!(!renderer.retired); let zero_copy = readback(device, queue, &texture).await; // Every pixel, not a sample of them: a shear from an ignored row pitch @@ -1252,13 +1234,11 @@ mod tests { let source = importing .source .import(&device, &gpu.surface) - .expect("import the decoded surface") - .expect("a DMA-BUF import path"); + .expect("import the decoded surface"); assert_eq!(source.layout, Layout::Nv12, "picture {index} did not import as NV12"); drop(source); let texture = importing.render(gpu).expect("draw the imported picture"); - assert_eq!(importing.strikes, 0, "picture {index} fell back to the CPU"); let zero_copy = readback(&device, &queue, &texture).await; let texture = uploading.render(cpu).expect("draw the downloaded picture"); @@ -1386,7 +1366,6 @@ mod tests { drop((file, export)); let texture = importing.render(gpu).expect("draw the imported picture"); - assert_eq!(importing.strikes, 0, "picture {index} fell back to the CPU"); let zero_copy = readback(&device, &queue, &texture).await; let texture = uploading.render(cpu).expect("draw the downloaded picture"); let reference = readback(&device, &queue, &texture).await; @@ -1438,7 +1417,7 @@ mod tests { /// GPU. Run with `--ignored`. #[cfg(apple)] #[tokio::test] - #[ignore = "needs a Metal GPU; see quest/m3/video-hardware.md"] + #[ignore = "needs a Metal GPU; run by just rs macos-test"] async fn imports_survive_decoder_pool_recycling() { let (device, queue) = gpu().await; let size = Size::new(256, 256); @@ -1452,23 +1431,24 @@ mod tests { for round in 0..24 { let rgba = colors[round % colors.len()]; // Built and dropped inside the loop, so its buffer returns to the - // pool the moment `render` returns. + // pool the moment `render` returns. A pixel buffer only renders by + // import, so `Ok` means the fast path ran. + let surface = pooled(size, rgba); + assert!(matches!(surface, crate::Surface::PixelBuffer(_))); let texture = renderer - .render(&Frame::new(pooled(size, rgba), Timestamp::ZERO)) - .expect("a rendered frame"); - assert_eq!(renderer.strikes, 0, "round {round} fell back to the CPU"); + .render(&Frame::new(surface, Timestamp::ZERO)) + .unwrap_or_else(|err| panic!("round {round}: {err}")); let pixels = readback(&device, &queue, &texture).await; assert_close(pixels[128 * 256 + 128], rgba); } } - /// The zero-copy import has to produce the same pixels as the upload, or the - /// fallback would silently change what the user sees. Ignored: needs a GPU. - /// Run with `--ignored`. + /// The zero-copy import has to produce the same pixels as the upload of the + /// same picture. Ignored: needs a GPU. Run with `--ignored`. #[cfg(apple)] #[tokio::test] - #[ignore = "needs a Metal GPU; see quest/m3/video-hardware.md"] + #[ignore = "needs a Metal GPU; run by just rs macos-test"] async fn the_metal_import_matches_the_cpu_path() { let (device, queue) = gpu().await; let size = Size::new(64, 64); @@ -1485,16 +1465,14 @@ mod tests { }; // The surface a hardware decode hands back: NV12 and IOSurface-backed. + // A pixel buffer only renders by import, so `Ok` means the fast path ran. let surface = pooled(size, rgba); + assert!(matches!(surface, crate::Surface::PixelBuffer(_))); let mut renderer = Renderer::new(&device, &queue, config).expect("a renderer"); let texture = renderer .render(&Frame::new(surface, Timestamp::ZERO)) - .expect("a rendered frame"); - // The point of the test: the fast path ran. Without this the CPU - // fallback would quietly satisfy every assertion below. - assert_eq!(renderer.strikes, 0, "the zero-copy import should not have failed"); - assert!(!renderer.retired); + .expect("the Metal import"); let imported = readback(&device, &queue, &texture).await; assert_eq!(uploaded.len(), imported.len()); diff --git a/rs/moq-video/src/render/source.rs b/rs/moq-video/src/render/source.rs index 6c9a2b926d6..86c1b352d00 100644 --- a/rs/moq-video/src/render/source.rs +++ b/rs/moq-video/src/render/source.rs @@ -1,7 +1,7 @@ //! Getting a frame's pixels into textures the shader can sample: a zero-copy -//! import where the platform offers one, a CPU upload everywhere else. +//! import for a GPU frame, an upload for a CPU one. -use crate::{Color, Error, Frame, Size, Surface}; +use crate::{Color, Error, I420, Output, Size, Surface}; /// How the planes of a [`Source`] are arranged, which picks the fragment shader. #[derive(Clone, Copy, Debug, PartialEq, Eq)] @@ -57,13 +57,11 @@ struct Planes { } impl Cache { - /// Try to alias a GPU frame's memory as textures, no download. + /// Alias a GPU frame's memory as textures, no download. /// - /// `Ok(None)` means this surface has no import path here (it is a CPU frame, - /// or the platform's importer is not built), which is a routing decision - /// rather than a failure. `Err` means an import that should have worked did - /// not, which is what the caller counts strikes against. - pub fn import(&mut self, device: &wgpu::Device, surface: &Surface) -> Result, Error> { + /// Errors for a surface this platform has no importer for, naming the + /// [`output`] that keeps such frames away from the renderer. + pub fn import(&mut self, device: &wgpu::Device, surface: &Surface) -> Result { match surface { #[cfg(all(target_os = "linux", feature = "dmabuf"))] Surface::DmaBuf(buffer) => super::dmabuf::import(device, buffer), @@ -73,22 +71,24 @@ impl Cache { Some(metal) => metal, None => self.metal.insert(super::metal::Import::new(device)?), }; - metal.import(device, buffer).map(Some) + metal.import(device, buffer) } - _ => { + #[allow( + unreachable_patterns, + reason = "the other surfaces exist only on some platforms and features" + )] + other => { let _ = device; - Ok(None) + Err(Error::Unsupported(format!( + "the renderer has no importer for {} surfaces; decode or capture with render::output(&device), which delivers them as CPU frames", + kind(other) + ))) } } } - /// Download the frame if it is not already on the CPU, then upload its - /// planes. The path every surface can take, and the one a failed or absent - /// import falls back to. It still fails for a GPU surface holding a pixel - /// format the crate cannot download, which is the honest answer: those - /// pixels are not reachable at all rather than merely expensive. - pub fn upload(&mut self, device: &wgpu::Device, queue: &wgpu::Queue, frame: &Frame) -> Result { - let i420 = frame.surface.to_i420()?; + /// Upload a CPU frame's planes. + pub fn upload(&mut self, device: &wgpu::Device, queue: &wgpu::Queue, i420: &I420) -> Result { let size = Size::new(i420.width(), i420.height()); size.validate("render source")?; @@ -117,6 +117,60 @@ impl Cache { } } +/// The [`Output`] to decode and capture with for a [`Renderer`](super::Renderer) on `device`. +/// +/// [`Output::Render`] when `device` can import this platform's GPU frames, and +/// [`Output::Cpu`] when it cannot, such as wgpu's GL backend, a Linux device +/// opened without [`wgpu::Features::VULKAN_EXTERNAL_MEMORY_DMA_BUF`], or a +/// platform with no importer. +pub fn output(device: &wgpu::Device) -> Output { + #[cfg(all(target_os = "linux", feature = "dmabuf"))] + let imports = super::dmabuf::supports(device); + #[cfg(apple)] + let imports = super::metal::supports(device); + #[cfg(not(any(apple, all(target_os = "linux", feature = "dmabuf"))))] + let imports = { + let _ = device; + false + }; + if imports { Output::Render } else { Output::Cpu } +} + +/// Whether [`Cache::import`] takes `surface` on a device [`output`] picks +/// [`Output::Render`] for: what that output keeps on the GPU. +pub(crate) fn imports(surface: &Surface) -> bool { + match surface { + #[cfg(all(target_os = "linux", feature = "dmabuf"))] + Surface::DmaBuf(buffer) => super::dmabuf::imports(buffer.format()), + #[cfg(apple)] + Surface::PixelBuffer(buffer) => super::metal::imports(buffer), + #[allow( + unreachable_patterns, + reason = "the other surfaces exist only on some platforms and features" + )] + _ => false, + } +} + +/// A GPU surface's kind, as an error names it. +fn kind(surface: &Surface) -> &'static str { + match surface { + #[cfg(target_os = "windows")] + Surface::Texture(_) => "Direct3D11", + #[cfg(all(target_os = "linux", feature = "nvidia"))] + Surface::Cuda(_) => "CUDA", + #[cfg(target_os = "linux")] + Surface::Vulkan(_) => "external Vulkan", + #[cfg(all(target_os = "android", feature = "mediacodec"))] + Surface::HardwareBuffer(_) => "Android hardware buffer", + #[allow( + unreachable_patterns, + reason = "the other surfaces exist only on some platforms and features" + )] + _ => "these", + } +} + impl Planes { fn new(device: &wgpu::Device, size: Size) -> Self { let plane = |label: &str, size: Size| { diff --git a/rs/moq-video/src/resize.rs b/rs/moq-video/src/resize.rs index 24ac01a0cea..853a5cd976c 100644 --- a/rs/moq-video/src/resize.rs +++ b/rs/moq-video/src/resize.rs @@ -13,6 +13,7 @@ pub struct Config { /// /// [`Output::Native`] scales a GPU surface on its own device, downloading /// and scaling on the CPU only when the device refuses. [`Output::Cpu`] - /// downloads first and always scales on the CPU. + /// downloads first and always scales on the CPU. [`Output::Render`] is + /// `Native` for a surface `render::Renderer` imports and `Cpu` otherwise. pub output: Output, }