Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
be99035
chore(quest): claim quest/m3/studio/app.md
kixelated Oct 25, 2026
9cbe6f0
feat(moq-scene): scene collection, sources, and a wgpu compositor
kixelated Oct 25, 2026
c5aeb63
feat(moq-studio): a docked egui studio that publishes its program
kixelated Oct 25, 2026
ed7df65
docs(moq-studio): document the studio and rewrite its quest to what r…
kixelated Oct 25, 2026
ee2db4a
fix(moq-audio): unlink capture-only items from default-feature docs
kixelated Oct 25, 2026
153c2b9
fix(moq-studio): mark a pause in the video and cut a keyframe on resume
kixelated Oct 25, 2026
b98700b
Merge remote-tracking branch 'origin/main' into quest/m3/studio/app
kixelated Oct 25, 2026
867ed0b
test(moq-studio): wait for the composed picture rather than the first…
kixelated Oct 25, 2026
d5b8420
Merge remote-tracking branch 'origin/main' into studio-app-iterate
kixelated Oct 25, 2026
afb2472
fix(moq-audio): panic on an invalid strip gain instead of ignoring it
kixelated Oct 25, 2026
0971a07
quest(studio): split the GPU encode and import into an m2 epic
kixelated Oct 25, 2026
c9a4ffd
Merge remote-tracking branch 'origin/main' into studio-app-iterate
kixelated Oct 25, 2026
2892284
quest(studio): move the GPU encode and import quests into m1
kixelated Oct 25, 2026
056aec2
Merge remote-tracking branch 'origin/main' into studio-app-iterate
kixelated Oct 25, 2026
b299e60
fix(studio): address review findings on sources, output, and startup
kixelated Oct 25, 2026
428a756
Merge remote-tracking branch 'origin/main' into studio-app-iterate
kixelated Oct 25, 2026
fdc6438
chore(quest): claim quest/m1/studio-gpu/import
kixelated Oct 25, 2026
427ee72
feat(video)!: report GPU import vs CPU upload per rendered frame
kixelated Oct 25, 2026
551bef6
fix(moq-scene): restart a broadcast rendition only when its name changes
kixelated Oct 25, 2026
65eb8ef
Merge remote-tracking branch 'origin/quest/m3/studio/app' into iterat…
kixelated Oct 25, 2026
d18d9d7
Merge remote-tracking branch 'origin/main' into iterate-6797
kixelated Oct 25, 2026
d4e8ac2
feat(video)!: refuse GPU frames the renderer cannot import
kixelated Oct 25, 2026
b206186
fix(studio): follow a rendition's track and fail loud instead of degr…
kixelated Oct 25, 2026
f7a227e
quest(studio-gpu): drop the counted-readback wording the fail-loud re…
kixelated Oct 25, 2026
e064036
Merge remote-tracking branch 'origin/quest/m3/studio/app' into ready-…
kixelated Oct 25, 2026
3b598fe
Merge remote-tracking branch 'origin/main' into HEAD
kixelated Oct 25, 2026
39277bb
docs(studio): pipewire adds cameras and audio, not Wayland screen cap…
kixelated Oct 25, 2026
4b96152
Merge origin/main into quest/m3/studio/app
kixelated Oct 25, 2026
98fb1be
fix(studio): drop an Items pick whose source is gone
kixelated Oct 26, 2026
d95970f
Merge commit '98fb1be6f82e7ebed3afb471fe2d0a7f01d7facd' into ready-6806
kixelated Oct 26, 2026
34e3a08
Merge remote-tracking branch 'origin/main' into ready-6806
kixelated Oct 26, 2026
fc4e7aa
docs(moq-video): Output::Cpu and Render error on a surface with no do…
kixelated Oct 26, 2026
9c9e6c7
fix(moq-video): VA-API exports for Output::Render only with the renderer
kixelated Oct 26, 2026
1818468
feat(moq-video): pick the render output from the device
kixelated Oct 26, 2026
ed0840a
Merge origin/main into quest/m1/studio-gpu/import
kixelated Oct 26, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions doc/bin/studio.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,3 +108,10 @@ studio passes over it for an encoder that reads each frame back to the CPU
(the opt-in `vaapi` or `openh264` features), which the Output panel counts.
Frames the encoder cannot keep up with are dropped rather than delaying the
stream.

Sources make the opposite trip. A frame already on the GPU is drawn where it
lies: PipeWire camera and VA-API decode DMA-BUFs on Linux, given a Vulkan
driver that imports them, and NV12 capture and VideoToolbox decode surfaces on
macOS. Every other source, every source on Windows or on a GPU that cannot
import, and every still image delivers its frames from the CPU, which the
compositor uploads.
4 changes: 1 addition & 3 deletions quest/m1/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,8 +105,7 @@ cut ships a relay that compiles no C. It follows moq-quic's crypto reshape,
whose surface its QUIC adapter plugs into.

Added 2026-10-25 (maintainer): the studio's
GPU encode and
[source import](/quest/m1/studio-gpu/import.md) quests are in m1 so
GPU encode and source import quests are in m1 so
moq-video's GPU encode and import API changes, mainly encoders taking a GPU
surface and zero-copy texture import, land before the cut.

Expand All @@ -132,7 +131,6 @@ surface and zero-copy texture import, land before the cut.
- [More tests under load](/quest/m1/test-flakes-2/README.md) - the second round of load-only failures, one quest per flake, fixed at the cause
- [Broadcast lifecycle](/quest/m1/broadcast-lifecycle/README.md) - broadcasts are online or offline with no finished state in moq-net, finished-broadcast retention is a fixed moq-mux policy, and each linger timer has one meaning
- [A smaller, safer API than release](/quest/m1/api-audit/README.md) - every surface added since release has a consumer or is gone, overlapping operations are collapsed, names mirror across Rust, JS, and the bindings, and every break says how to migrate
- [Studio on the GPU](/quest/m1/studio-gpu/README.md) - moq-video encoders take a GPU surface and its renderer reports zero-copy import, proven through moq-studio on NVENC, VideoToolbox, Media Foundation, DMA-BUF, and Metal
- [moq-transport 18 and 22 only](/quest/m1/ietf-drop/README.md) - drafts 14-17 and 19-21 are deleted from Rust, JS, tests, interop lanes, and docs, removing the legacy control-stream adapter and its request limit
- [moq-quic's API improves on quinn's](/quest/m1/quic-api/README.md) - moq-quic and both QUIC drivers ship one mirrored API: `send::`/`recv::` namespaces, one error per direction, consuming terminal operations, plain configs, no quinn baggage
- [Pure-Rust crypto for moq-relay](/quest/m1/pure-crypto/README.md) - moq-relay's release builds use graviola and RustCrypto and compile no C
Expand Down
30 changes: 0 additions & 30 deletions quest/m1/studio-gpu/README.md

This file was deleted.

37 changes: 0 additions & 37 deletions quest/m1/studio-gpu/import.md

This file was deleted.

9 changes: 5 additions & 4 deletions quest/m2/render-d3d11.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,13 @@ I420 and re-uploaded.
The surface is already there and already GPU-resident: `Surface::Texture`
carries a D3D11 NV12 texture, decode retains it, and GPU resize works. What is
missing is the import at the other end, which the renderer has for Vulkan
DMA-BUF and Metal but not here, so Windows falls through to the CPU path.
DMA-BUF and Metal but not here, so Windows frames take the CPU path.

Import the texture as a wgpu texture through its D3D12 or Vulkan backend
rather than copying, keeping the same per-path fallback and disable behavior
the other import paths use: a machine where the import cannot work should
quietly take the I420 route, not fail to present.
rather than copying. Like the other import paths, the renderer has no
fallback: until this lands it refuses `Surface::Texture`, and
`Output::Render` downloads those frames at the decoder or capture instead.
Landing it adds `Surface::Texture` to what `Output::Render` keeps on the GPU.

The renderer is the first consumer of `d3d11::Texture`'s contents outside
moq-video's frame module (its fields are `pub(crate)`,
Expand Down
4 changes: 1 addition & 3 deletions quest/m3/studio/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,14 +60,12 @@ OBS multistream, and the hosted version is product work outside this
repository.

Added 2026-10-25 (maintainer): the studio's
GPU encode and
[source import](/quest/m1/studio-gpu/import.md) quests are in m1 so
GPU encode and source import quests are in m1 so
moq-video's GPU encode and import API changes, mainly encoders taking a GPU
surface and zero-copy texture import, land before the cut.

## Required

- [Zero-copy source import](/quest/m1/studio-gpu/import.md) - captured and decoded frames reach the compositor as GPU textures instead of a CPU upload
- [Studio on macOS and Windows](/quest/m3/studio/app.md) - the studio runs on macOS and Windows, not only Linux
- [Autosave](/quest/m3/studio/autosave.md) - every edit is written to the scene file, with no save step or quit prompt
- [Scene editing](/quest/m3/studio/scene-editing.md) - rename and reorder scenes, and resize the canvas while live through a new rendition
Expand Down
4 changes: 3 additions & 1 deletion quest/m3/studio/packaging.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,9 @@ and a `release` feature turns on every backend its platform can load without
one (`capture`, `openh264`, `vaapi`, `nvidia`, and the platform frameworks).
Every artifact builds with `release`. `pipewire` stays out of `release`
because it links libpipewire at load time, but the Flatpak bundles
libpipewire, so it adds `pipewire` itself.
libpipewire, so it adds `pipewire` itself: inside the sandbox the camera
portal is the only way to a camera, and its PipeWire DMA-BUFs are what the
compositor imports without a copy.

## Required

Expand Down
8 changes: 4 additions & 4 deletions quest/m3/video-embedded.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,14 +12,14 @@ records what breaks.
Decided 2026-10-08: verify first. The Pi 4 and 5 ship Mesa's v3dv Vulkan
driver, so the premise that embedded devices have no usable Vulkan driver is
doubtful. Run the renderer's DMA-BUF Vulkan import against V4L2 M2M decoder
output, and the I420 CPU fallback, and record the device, OS image, Mesa
output, and the I420 CPU path, and record the device, OS image, Mesa
version, and which path presented.

If the Vulkan import cannot take the decoder's buffers (a missing extension
or modifier) on a device a consumer needs, plan EGL/GLES import as its own
quest: same shape as the Vulkan path (alias the buffer, keep the per-path
fallback and three-strike disable, fall back to I420 when import fails), and
the usual dlopen-and-degrade rule, so a build without the device present
quest: same shape as the Vulkan path (alias the buffer, refuse what it cannot
import rather than download it, and teach `Output::Render` which buffers it
takes), and the usual dlopen-and-degrade rule, so a build without the device present
degrades rather than fails to start.

Two items from the same wave are deliberately not here. X11 MIT-SHM capture is
Expand Down
4 changes: 1 addition & 3 deletions quest/m3/video-hardware.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ A separate-plane producer belongs to
Pi produces nothing, write it down and stop. `doc/lib/rs/moq-video.md` says
both paths are reachable; correct that sentence if one cannot capture.

### Windows and Apple GPU probes
### Windows GPU probes

Ignored moq-video tests that only a GPU on that platform can run, and that no
runner hosts (found by the m0 coverage audit):
Expand All @@ -83,8 +83,6 @@ runner hosts (found by the m0 coverage audit):
capture tests belong to [Windows.Graphics.Capture](/quest/m2/capture-wgc.md).
- Media Foundation encode: `mediafoundation_cpu_rgba` and
`mediafoundation_camera_texture`.
- Metal import on an Apple Silicon Mac: `the_metal_import_matches_the_cpu_path`
and `imports_survive_decoder_pool_recycling`.

### Fixes

Expand Down
5 changes: 4 additions & 1 deletion rs/justfile
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,9 @@ macos-test:
just rs test -p moq-sock --features fast-apple-datapath
just rs test -p moq-video --no-default-features -E 'test(encode::backend::videotoolbox::tests)'
just rs test -p moq-video --no-default-features --features render -E 'test(/^render::export::/)'
# The runner's paravirtual Metal device proves the IOSurface import, ignored elsewhere.
just rs test -p moq-video --no-default-features --features render --lib --run-ignored only \
-E 'test(/^render::renderer::tests::(the_metal_import_matches_the_cpu_path|imports_survive_decoder_pool_recycling)$/)'

# Compile and lint the capture and playback features, which the default check skips.
[positional-arguments]
Expand Down Expand Up @@ -369,7 +372,7 @@ render-sw:
icd=("$mesa"/share/vulkan/icd.d/lvp_icd.*.json)
LD_LIBRARY_PATH="$loader/lib" VK_DRIVER_FILES="${icd[0]}" \
just rs test -p moq-video --features render --lib --run-ignored only \
-E 'test(/^render::renderer::tests::(the_declared_color_space_beats_the_size_guess|cpu_frames_survive_the_round_trip|config_size_overrides_the_frame_size)$/)'
-E 'test(/^render::renderer::tests::(the_declared_color_space_beats_the_size_guess|cpu_frames_survive_the_round_trip|config_size_overrides_the_frame_size|an_unimportable_gpu_frame_is_refused|the_output_follows_the_device)$/)'
# Mesa's software driver exports no memory, so this covers the readback route.
LD_LIBRARY_PATH="$loader/lib" VK_DRIVER_FILES="${icd[0]}" \
just rs test -p moq-video --features render,openh264 --lib --run-ignored only -E 'test(/^render::export::(nv12::)?tests::/)'
Expand Down
4 changes: 4 additions & 0 deletions rs/moq-cli/src/play/fake.rs
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,10 @@ impl Output for Recorder {
fn send(&self, event: Event) {
self.state.lock().unwrap().events.push(event);
}

async fn render(&self) -> moq_video::Output {
moq_video::Output::Cpu
}
}

impl Speaker for Recorder {
Expand Down
5 changes: 5 additions & 0 deletions rs/moq-cli/src/play/headless.rs
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,11 @@ impl Output for Headless {
fn send(&self, event: Event) {
let _ = self.events.send(event);
}

/// Never asked: headless playback decodes no video.
async fn render(&self) -> moq_video::Output {
moq_video::Output::Cpu
}
}

pub(crate) async fn run(
Expand Down
7 changes: 6 additions & 1 deletion rs/moq-cli/src/play/media.rs
Original file line number Diff line number Diff line change
Expand Up @@ -258,7 +258,12 @@ impl<O: Output> Media<O> {
// doubles as the staleness budget on the wire.
let max_delay = self.args.video_max_delay(&snapshot.audio);
let opened = async {
let decoder = moq_video::decode::Sink::open(config, &Default::default()).await?;
// The window's renderer refuses a GPU surface it cannot import, so
// the decoder downloads those instead, all of them on a device
// that imports none.
let mut decode = moq_video::decode::Config::new();
decode.output = self.output.render().await;
let decoder = moq_video::decode::Sink::open(config, &decode).await?;
let track = rendition.track(hang::catalog::Reference::track(config, name))?;
let mut subscriber = track
.subscribe(
Expand Down
3 changes: 3 additions & 0 deletions rs/moq-cli/src/play/output.rs
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,9 @@ pub(super) trait Output: Clone + Send + Sync + 'static {

/// Tell the window something changed. Dropped if the window is gone.
fn send(&self, event: Event);

/// What to decode video as for the window's renderer, once its device is open.
fn render(&self) -> impl Future<Output = moq_video::Output> + Send;
}

/// An open speaker, closed once it and every sink from it are dropped.
Expand Down
35 changes: 30 additions & 5 deletions rs/moq-cli/src/play/window.rs
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,10 @@ pub fn run(
// Signals the decoder that the presenter took a frame, so it can hand over
// the next one instead of dropping it.
let drained = Arc::new(tokio::sync::Notify::new());
// Set once the window's device opens. The media task starts now anyway, so
// the broadcast resolves while the window does, and waits for it only to
// open a decoder.
let render = Arc::new(tokio::sync::SetOnce::new());

let media = tokio::spawn(
Media {
Expand All @@ -74,7 +78,10 @@ pub fn run(
video: video.clone(),
presentation: presentation.clone(),
drained: drained.clone(),
output: proxy.clone(),
output: Remote {
proxy: proxy.clone(),
render: render.clone(),
},
}
.run(),
);
Expand All @@ -93,7 +100,7 @@ pub fn run(
} else {
format!("moq play: {broadcast}")
};
let mut app = App::new(title, video, presentation, drained);
let mut app = App::new(title, video, presentation, drained, render);
let result = event_loop.run_app(&mut app).context("playback event loop failed");
media.abort();
network.abort();
Expand All @@ -106,15 +113,27 @@ pub fn run(
}
}

impl Output for EventLoopProxy<Event> {
/// The window as the media task reaches it.
#[derive(Clone)]
struct Remote {
proxy: EventLoopProxy<Event>,
/// What the window's device draws, once it is open.
render: Arc<tokio::sync::SetOnce<moq_video::Output>>,
}

impl Output for Remote {
type Speaker = moq_audio::playback::Engine;

async fn speaker(&self) -> anyhow::Result<Self::Speaker> {
Ok(moq_audio::playback::Engine::open(Default::default()).await?)
}

fn send(&self, event: Event) {
let _ = self.send_event(event);
let _ = self.proxy.send_event(event);
}

async fn render(&self) -> moq_video::Output {
*self.render.wait().await
}
}

Expand All @@ -136,6 +155,7 @@ struct App {
video: Arc<Mutex<VecDeque<moq_video::Frame>>>,
presentation: Arc<Mutex<Presentation>>,
drained: Arc<tokio::sync::Notify>,
render: Arc<tokio::sync::SetOnce<moq_video::Output>>,
display: Option<Display>,
next_redraw: Option<Instant>,
/// The media tasks are done. Keep presenting whatever they left queued, then
Expand All @@ -152,12 +172,14 @@ impl App {
video: Arc<Mutex<VecDeque<moq_video::Frame>>>,
presentation: Arc<Mutex<Presentation>>,
drained: Arc<tokio::sync::Notify>,
render: Arc<tokio::sync::SetOnce<moq_video::Output>>,
) -> Self {
Self {
title,
video,
presentation,
drained,
render,
display: None,
next_redraw: None,
ending: false,
Expand Down Expand Up @@ -242,6 +264,9 @@ impl ApplicationHandler<Event> for App {
// still resolving, rather than showing whatever was in the surface.
Ok(display) => {
display.window.request_redraw();
self.render
.set(moq_video::render::output(&display.device))
.expect("the display opens once");
self.display = Some(display);
}
Err(err) => self.fail(event_loop, err),
Expand Down Expand Up @@ -355,7 +380,7 @@ impl Display {
compatible_surface: Some(&surface),
apply_limit_buckets: false,
}))?;
let (device, queue) = pollster::block_on(adapter.request_device(&wgpu::DeviceDescriptor::default()))?;
let (device, queue) = pollster::block_on(moq_video::render::request_device(&adapter, &Default::default()))?;
let mut config = surface
.get_default_config(&adapter, size.width.max(1), size.height.max(1))
.context("the graphics adapter cannot present to this window")?;
Expand Down
5 changes: 5 additions & 0 deletions rs/moq-scene/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,11 @@ categories = ["multimedia", "multimedia::video", "rendering"]
# libasound at build time, and a workspace build would otherwise unify it on for
# every crate. Without it, those sources report that the build cannot capture.
capture = ["moq-video/capture", "moq-audio/capture"]
# PipeWire cameras on Linux, whose DMA-BUFs the compositor imports rather than
# uploads, and audio devices through the sound server. Off by default like
# moq-video's `pipewire`: it links libpipewire-0.3 at build time, so a Linux
# package turns it on where the build host has it.
pipewire = ["capture", "moq-video/pipewire", "moq-audio/pipewire"]

[dependencies]
bytemuck = { version = "1", features = ["derive"] }
Expand Down
3 changes: 2 additions & 1 deletion rs/moq-scene/src/compositor.rs
Original file line number Diff line number Diff line change
Expand Up @@ -709,7 +709,8 @@ mod tests {
async fn new(collection: Collection) -> Self {
let (device, queue) = gpu().await;
let dir = tempfile::tempdir().unwrap();
let (context, origin) = context(dir.path());
let (mut context, origin) = context(dir.path());
context.output = moq_video::render::output(&device);
let mut sources = Sources::new(context);
let drivers = sources.sync(&collection);
start(&sources, drivers).await;
Expand Down
Loading
Loading