Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
be99035
chore(quest): claim quest/m3/studio/app.md
kixelated Oct 25, 2026
9cbe6f0
feat(moq-scene): scene collection, sources, and a wgpu compositor
kixelated Oct 25, 2026
c5aeb63
feat(moq-studio): a docked egui studio that publishes its program
kixelated Oct 25, 2026
ed7df65
docs(moq-studio): document the studio and rewrite its quest to what r…
kixelated Oct 25, 2026
ee2db4a
fix(moq-audio): unlink capture-only items from default-feature docs
kixelated Oct 25, 2026
153c2b9
fix(moq-studio): mark a pause in the video and cut a keyframe on resume
kixelated Oct 25, 2026
b98700b
Merge remote-tracking branch 'origin/main' into quest/m3/studio/app
kixelated Oct 25, 2026
867ed0b
test(moq-studio): wait for the composed picture rather than the first…
kixelated Oct 25, 2026
d5b8420
Merge remote-tracking branch 'origin/main' into studio-app-iterate
kixelated Oct 25, 2026
afb2472
fix(moq-audio): panic on an invalid strip gain instead of ignoring it
kixelated Oct 25, 2026
0971a07
quest(studio): split the GPU encode and import into an m2 epic
kixelated Oct 25, 2026
c9a4ffd
Merge remote-tracking branch 'origin/main' into studio-app-iterate
kixelated Oct 25, 2026
2892284
quest(studio): move the GPU encode and import quests into m1
kixelated Oct 25, 2026
056aec2
Merge remote-tracking branch 'origin/main' into studio-app-iterate
kixelated Oct 25, 2026
b299e60
fix(studio): address review findings on sources, output, and startup
kixelated Oct 25, 2026
428a756
Merge remote-tracking branch 'origin/main' into studio-app-iterate
kixelated Oct 25, 2026
b358ac8
chore(quest): claim quest/m1/studio-gpu/encode.md
kixelated Oct 25, 2026
f0f7a75
feat(moq-video): export wgpu textures to an encoder as frames
kixelated Oct 25, 2026
551bef6
fix(moq-scene): restart a broadcast rendition only when its name changes
kixelated Oct 25, 2026
0a07f9c
feat(moq-studio)!: encode the program through moq_video::render::export
kixelated Oct 25, 2026
9318c43
feat(moq-video): export to VideoToolbox and Media Foundation on the GPU
kixelated Oct 25, 2026
ea9eb5a
chore(quest): finish studio-gpu/encode
kixelated Oct 25, 2026
80ec540
Merge remote-tracking branch 'origin/quest/m3/studio/app' into quest/…
kixelated Oct 25, 2026
d18d9d7
Merge remote-tracking branch 'origin/main' into iterate-6797
kixelated Oct 25, 2026
afd8ddd
feat(moq-video)!: settle the export route once, from the encoder, wit…
kixelated Oct 25, 2026
b206186
fix(studio): follow a rendition's track and fail loud instead of degr…
kixelated Oct 25, 2026
35bb133
fix(moq-relay): expect the clock lint only on the test that reads the…
kixelated Oct 25, 2026
c4862f6
fix(moq-video): compile I420::from_bgra only for PipeWire capture
kixelated Oct 25, 2026
3078440
test(moq-video): refuse the Direct3D12 export on WARP, and say why
kixelated Oct 25, 2026
d7710fe
Merge remote-tracking branch 'origin/quest/m3/studio/app' into worktr…
kixelated Oct 25, 2026
810f586
fix(moq-video): settle the export on an encoder that reads this device
kixelated Oct 25, 2026
9233df0
Merge remote-tracking branch 'origin/main' into HEAD
kixelated Oct 26, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 8 additions & 3 deletions doc/bin/studio.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,11 @@ with Opus stereo audio, under the `--broadcast` path. Timestamps come from the
studio's own clock, so a source that stalls repeats its last picture instead of
holding up the stream.

Video is only encoded while someone watches it. Each frame is read back from
the GPU to the CPU for the encoder, which the Output panel counts; frames the
encoder cannot keep up with are dropped there rather than delaying the stream.
Video is only encoded while someone watches it. A hardware encoder that reads
the GPU (NVENC on Linux, VideoToolbox on macOS, Media Foundation on Windows)
takes each frame there when it can read the GPU the studio draws on. When it
cannot, such as NVENC while the studio draws on a laptop's integrated GPU, the
studio passes over it for an encoder that reads each frame back to the CPU
(the opt-in `vaapi` or `openh264` features), which the Output panel counts.
Frames the encoder cannot keep up with are dropped rather than delaying the
stream.
2 changes: 1 addition & 1 deletion quest/m1/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ cut ships a relay that compiles no C. It follows moq-quic's crypto reshape,
whose surface its QUIC adapter plugs into.

Added 2026-10-25 (maintainer): the studio's
[GPU encode](/quest/m1/studio-gpu/encode.md) and
GPU encode and
[source import](/quest/m1/studio-gpu/import.md) quests are in m1 so
moq-video's GPU encode and import API changes, mainly encoders taking a GPU
surface and zero-copy texture import, land before the cut.
Expand Down
1 change: 0 additions & 1 deletion quest/m1/studio-gpu/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@ test each platform's path on that platform's CI.

## Required

- [Zero-copy program encode](/quest/m1/studio-gpu/encode.md) - the program reaches the encoder as a GPU surface instead of a CPU readback
- [Zero-copy source import](/quest/m1/studio-gpu/import.md) - captured and decoded frames reach the compositor as GPU textures, with uploads counted

## Related
Expand Down
36 changes: 0 additions & 36 deletions quest/m1/studio-gpu/encode.md

This file was deleted.

1 change: 0 additions & 1 deletion quest/m1/studio-gpu/import.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,6 @@ the studio counts uploads per source next to its readback stats.

## Related

- [Zero-copy program encode](/quest/m1/studio-gpu/encode.md) - the other half of keeping the studio's pictures on the GPU
- [Validate PipeWire DMA-BUFs into the Vulkan renderer](/quest/m2/2819-moq-video-carry-pipewire-dma-bufs-safely-into-the-vulkan.md) - the hardware proof of the Linux import path
- [Direct3D11 render import](/quest/m2/render-d3d11.md) - the follow-up that brings Windows sources onto the GPU
- [VAAPI encode and decode](/quest/m2/video-vaapi.md) - decoded DMA-BUF surfaces the renderer imports
Expand Down
3 changes: 1 addition & 2 deletions quest/m3/studio/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,14 +60,13 @@ OBS multistream, and the hosted version is product work outside this
repository.

Added 2026-10-25 (maintainer): the studio's
[GPU encode](/quest/m1/studio-gpu/encode.md) and
GPU encode and
[source import](/quest/m1/studio-gpu/import.md) quests are in m1 so
moq-video's GPU encode and import API changes, mainly encoders taking a GPU
surface and zero-copy texture import, land before the cut.

## Required

- [Zero-copy program encode](/quest/m1/studio-gpu/encode.md) - the program reaches the encoder as a GPU surface instead of a CPU readback
- [Zero-copy source import](/quest/m1/studio-gpu/import.md) - captured and decoded frames reach the compositor as GPU textures instead of a CPU upload
- [Studio on macOS and Windows](/quest/m3/studio/app.md) - the studio runs on macOS and Windows, not only Linux
- [Destinations](/quest/m3/studio/destinations.md) - RTMP, RTMPS, and WHIP push with per-destination codecs and shared encoders, on a UI-free egress library a hosted service can reuse
Expand Down
1 change: 0 additions & 1 deletion quest/m3/studio/app.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,5 +19,4 @@ landed.

## Related

- [Zero-copy program encode](/quest/m1/studio-gpu/encode.md) - the GPU encode path, which lands without this run
- [Hardware validation](/quest/m3/video-hardware.md) - capture paths not yet run on real machines
7 changes: 7 additions & 0 deletions quest/m3/video-hardware.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,13 @@ by [VA-API external images](/quest/m2/vaapi-vulkan-import.md).
- **Windows Media Foundation capture**: on-demand open and close, so the
camera LED is off when nobody is watching, and NV12 delivery from MJPEG and
YUY2 cameras.
- **The studio's program on a hardware encoder**: run the ignored
`render::export::dx12::tests::exports_an_nv12_texture` (the Platform runner
has only WARP, which has no Direct3D11 video device) and
`render::export::dx12::tests::media_foundation_encodes_the_export_in_place`
on a Windows GPU with a Media Foundation H.264 encoder, and `moq-studio`
publishing on that machine and a Mac, confirming the Output panel reports
no readback.
- **A live camera run per platform**: capture needs device permission that a
headless or agent process cannot grant itself.
- **OBS follows its grant on a shaped uplink**: stream from a real OBS build
Expand Down
5 changes: 5 additions & 0 deletions rs/justfile
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,7 @@ windows $BASE="":
windows-test:
just rs test --manifest-path rs/moq-auth/Cargo.toml
just rs test -p hang -p moq-tokio -p moq-sock -p moq-relay
just rs test -p moq-video --no-default-features --features render -E 'test(/^render::export::/)'

# Run io_uring tests when the branch affects the relay or its dependencies.
[positional-arguments]
Expand Down Expand Up @@ -141,6 +142,7 @@ macos-test:
just rs test -p hang -p moq-tokio -p moq-sock
just rs test -p moq-sock --features fast-apple-datapath
just rs test -p moq-video --no-default-features -E 'test(encode::backend::videotoolbox::tests)'
just rs test -p moq-video --no-default-features --features render -E 'test(/^render::export::/)'

# Compile and lint the capture and playback features, which the default check skips.
[positional-arguments]
Expand Down Expand Up @@ -368,6 +370,9 @@ render-sw:
LD_LIBRARY_PATH="$loader/lib" VK_DRIVER_FILES="${icd[0]}" \
just rs test -p moq-video --features render --lib --run-ignored only \
-E 'test(/^render::renderer::tests::(the_declared_color_space_beats_the_size_guess|cpu_frames_survive_the_round_trip|config_size_overrides_the_frame_size)$/)'
# Mesa's software driver exports no memory, so this covers the readback route.
LD_LIBRARY_PATH="$loader/lib" VK_DRIVER_FILES="${icd[0]}" \
just rs test -p moq-video --features render,openh264 --lib --run-ignored only -E 'test(/^render::export::(nv12::)?tests::/)'
LD_LIBRARY_PATH="$loader/lib" VK_DRIVER_FILES="${icd[0]}" \
just rs test -p moq-scene --lib --run-ignored only -E 'test(/^compositor::tests::/)'
LD_LIBRARY_PATH="$loader/lib" VK_DRIVER_FILES="${icd[0]}" \
Expand Down
69 changes: 39 additions & 30 deletions rs/moq-scene/src/compositor.rs
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ use crate::collection::{Canvas, Collection, Rect, Source};
use crate::source::{Picture, Sources};

/// The format of every canvas texture: gamma-encoded RGBA, what
/// `moq_video::render` produces and what an encoder reads back.
/// `moq_video::render` produces and what its exporter hands an encoder.
pub const FORMAT: wgpu::TextureFormat = wgpu::TextureFormat::Rgba8Unorm;

/// One draw's uniform: where on the canvas, in clip space, and how opaque.
Expand Down Expand Up @@ -277,7 +277,8 @@ impl Compositor {
}

/// The program output, as of the last [`render`](Self::render). Created
/// with `COPY_SRC` so an encoder can read it back.
/// with `COPY_SRC | TEXTURE_BINDING`, what
/// `moq_video::render::export::Exporter::copy` takes to hand it to an encoder.
pub fn program(&self) -> &wgpu::Texture {
&self.program.texture
}
Expand Down Expand Up @@ -632,11 +633,8 @@ impl Uniforms {

#[cfg(test)]
mod tests {
use moq_net::Timestamp;

use super::*;
use crate::collection::{Item, Rgba, Scene, Source};
use crate::readback::Readback;
use crate::source::test::{context, local, start};

const RED: [u8; 4] = [255, 0, 0, 255];
Expand Down Expand Up @@ -701,7 +699,8 @@ mod tests {
collection: Collection,
sources: Sources,
compositor: Compositor,
readback: Readback,
device: wgpu::Device,
queue: wgpu::Queue,
_origin: moq_net::origin::Producer,
_dir: tempfile::TempDir,
}
Expand All @@ -716,7 +715,8 @@ mod tests {
start(&sources, drivers).await;
Self {
compositor: Compositor::new(&device, &queue, collection.canvas).unwrap(),
readback: Readback::new(&device, &queue, collection.canvas),
device,
queue,
collection,
sources,
_origin: origin,
Expand All @@ -728,8 +728,39 @@ mod tests {
self.compositor.render(&self.collection, &self.sources, now).unwrap();
}

/// The canvas texture's exact RGBA rows: no 4:2:0 round trip, so a test
/// compares the stored values.
fn read(&mut self, texture: &wgpu::Texture) -> Vec<u8> {
self.readback.copy(texture, Timestamp::ZERO).unwrap().rgba().unwrap()
let canvas = self.compositor.canvas();
let row = (canvas.width * 4).next_multiple_of(wgpu::COPY_BYTES_PER_ROW_ALIGNMENT);
let buffer = self.device.create_buffer(&wgpu::BufferDescriptor {
label: None,
size: u64::from(row * canvas.height),
usage: wgpu::BufferUsages::COPY_DST | wgpu::BufferUsages::MAP_READ,
mapped_at_creation: false,
});
let mut encoder = self.device.create_command_encoder(&Default::default());
encoder.copy_texture_to_buffer(
texture.as_image_copy(),
wgpu::TexelCopyBufferInfo {
buffer: &buffer,
layout: wgpu::TexelCopyBufferLayout {
offset: 0,
bytes_per_row: Some(row),
rows_per_image: None,
},
},
texture.size(),
);
self.queue.submit([encoder.finish()]);
buffer.map_async(wgpu::MapMode::Read, .., |result| result.unwrap());
self.device.poll(wgpu::PollType::wait_indefinitely()).unwrap();
let width = canvas.width as usize * 4;
let mapped = buffer.slice(..).get_mapped_range().unwrap();
mapped
.chunks_exact(row as usize)
.flat_map(|row| row[..width].to_vec())
.collect()
}

fn program(&mut self) -> Vec<u8> {
Expand Down Expand Up @@ -911,26 +942,4 @@ mod tests {
})
.await;
}

/// A slow encoder runs out of buffers and drops pictures rather than
/// queuing them, and a finished copy frees its buffer.
#[tokio::test]
#[ignore = "needs a Vulkan device; run by just rs render-sw"]
async fn readback_drops_past_its_depth() {
local(async {
let mut studio = Studio::new(collection()).await;
studio.render(Instant::now());
let program = studio.compositor.program().clone();
let mut held: Vec<_> = (0..Readback::DEPTH)
.map(|_| studio.readback.copy(&program, Timestamp::ZERO).unwrap())
.collect();
assert!(studio.readback.copy(&program, Timestamp::ZERO).is_none());

let frame = held.pop().unwrap().wait().unwrap();
assert_eq!(frame.size(), moq_video::Size::new(8, 8));
held.push(studio.readback.copy(&program, Timestamp::ZERO).unwrap());
assert!(studio.readback.copy(&program, Timestamp::ZERO).is_none());
})
.await;
}
}
10 changes: 2 additions & 8 deletions rs/moq-scene/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@
//! compositor and sound goes to a `moq_audio::bus::Bus`, the program audio.
//! - [`Compositor`] draws the scene in preview and the scene on program into
//! textures on the caller's `wgpu` device, cutting or fading between them.
//! - [`Readback`] copies the program back to the CPU for an encoder that cannot
//! take a GPU surface.
//! `moq_video::render::export` hands the program to an encoder, on the GPU
//! where the platform allows.
//!
//! Every picture is drawn on the program clock the caller ticks
//! [`Compositor::render`] with, from whatever each source delivered last: a
Expand All @@ -22,12 +22,10 @@

mod collection;
mod compositor;
mod readback;
pub mod source;

pub use collection::{Audio, Canvas, Collection, Item, Rect, Rgba, Scene, Source, Transition};
pub use compositor::{Compositor, FORMAT};
pub use readback::{Pending, Readback};

/// The `wgpu` the compositor draws with: `moq_video`'s, so its textures and
/// the decoder's are the same type.
Expand Down Expand Up @@ -63,10 +61,6 @@ pub enum Error {
#[error("unsupported source: {0}")]
Unsupported(String),

/// The GPU failed while reading a picture back.
#[error("gpu: {0}")]
Gpu(String),

/// Capturing, decoding, or drawing video failed.
#[error(transparent)]
Video(#[from] moq_video::Error),
Expand Down
Loading
Loading