Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions doc/bin/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -126,8 +126,8 @@ audio is Opus. The camera is opened only while someone is watching, and
`--bitrate` is the opening ceiling. Backends with live bitrate control lower it
to fit the connection's bandwidth estimate. `moq devices` prints every source
id. Requires the `capture` feature; on Linux that needs the ALSA headers for
the microphone, and `--display` also needs the `pipewire` feature (links
libpipewire).
the microphone, and `--display` and `pipewire:` cameras also need the
`pipewire` feature (links libpipewire).

## Transcode

Expand Down
16 changes: 13 additions & 3 deletions doc/lib/rs/moq-video.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,11 +39,21 @@ and an empty rate list means no discrete intervals were reported. Device errors
are returned rather than treated as an empty list. Other platforms return
`Error::Unsupported`.

With `pipewire` enabled, `capture::cameras` also lists PipeWire camera nodes as
`pipewire:<node name>` after the V4L2 devices, and `pipewire` alone opens the
camera with the highest session priority, the session manager's default. That
reaches cameras V4L2 cannot: a Raspberry Pi CSI camera behind libcamera, and any
camera from inside a Flatpak or Snap sandbox, where the default camera comes
through the xdg-desktop-portal Camera interface. The mode is chosen from the
node's own format list by the same rules as V4L2 (below) and offered exactly,
across YUY2, NV12, RGB, and MJPEG, and `camera_modes` lists the same modes.

`capture::Config::framerate` is an `Option<Rate>` request in the same exact
type; the stream reports the rate the device accepted, or `None` when the
driver reported none.
V4L2 chooses the closest geometry, then the format whose accepted rate is
nearest the request, then the cheaper conversion when both match equally well.
V4L2 and PipeWire cameras choose the closest geometry, then the format whose
accepted rate is nearest the request, then the cheaper conversion when both
match equally well.

```rust
let mut video = moq_video::decode::Consumer::new(&broadcast, &rendition, "video", Default::default()).await?;
Expand All @@ -59,7 +69,7 @@ cargo add moq-video --features capture # camera + screen capture, no system bu
cargo add moq-video --features render # wgpu rendering
cargo add moq-video --features v4l2 # Linux V4L2 M2M codecs, no system build deps
cargo add moq-video --features vaapi # Linux VAAPI codecs (bindgen needs libclang)
cargo add moq-video --features pipewire # Wayland screen capture (links libpipewire)
cargo add moq-video --features pipewire # Wayland screen + PipeWire cameras (links libpipewire)
cargo add moq-video --no-default-features --features openh264 # software H.264 only
cargo add moq-video --no-default-features --features nvidia # Linux NVIDIA only, no C++ or wgpu
```
Expand Down
4 changes: 2 additions & 2 deletions rs/moq-cli/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -74,8 +74,8 @@ vaapi = ["moq-video?/vaapi", "moq-transcode?/vaapi"]
# checks its bindings in. Only applies when `capture`, `transcode`, or `play`
# pulls in moq-video.
v4l2 = ["moq-video?/v4l2", "moq-transcode?/v4l2"]
# Screen capture for `capture --display` on Linux (xdg-desktop-portal + PipeWire),
# opt-in like moq-video's own `pipewire` because it links libpipewire-0.3 at
# Screen capture for `capture --display` on Linux (xdg-desktop-portal + PipeWire)
# and PipeWire cameras for `capture --camera pipewire:<node>`, opt-in like moq-video's own `pipewire` because it links libpipewire-0.3 at
# build time. Only bites when `capture` pulls in moq-video and moq-audio.
# Also reaches audio, so `devices`, microphone capture and playback go through
# the sound server rather than its ALSA plugin on a machine running one.
Expand Down
3 changes: 2 additions & 1 deletion rs/moq-cli/src/publish.rs
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,8 @@ impl From<VideoCodec> for moq_video::encode::Codec {
#[usage(group("audio-source"))]
pub struct CaptureArgs {
/// Capture a camera, by the id `moq devices` reports (an AVFoundation
/// `uniqueID`, `/dev/videoN` path, or Media Foundation symbolic link).
/// `uniqueID`, `/dev/videoN` path, `pipewire:` node, or Media Foundation
/// symbolic link).
/// Bare `--camera`, or no source flag at all, opens the default camera.
#[usage(long, group = "video-source")]
pub camera: Option<Option<String>>,
Expand Down
23 changes: 14 additions & 9 deletions rs/moq-video/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -86,10 +86,12 @@ render = [
"objc2-core-video/CVMetalTextureCache",
"objc2-core-video/objc2-metal",
]
# Screen capture on Linux via xdg-desktop-portal + PipeWire (Wayland and X11).
# Off by default because the `pipewire` crate links libpipewire-0.3 via pkg-config
# at build time (and its bindgen needs libclang), so it only belongs in builds
# that actually capture displays. Camera capture (V4L2) needs only what
# Screen capture on Linux via xdg-desktop-portal + PipeWire (Wayland and X11),
# and camera capture from PipeWire camera nodes: libcamera sensors such as a
# Raspberry Pi CSI camera, and any camera from inside a Flatpak or Snap sandbox
# through the camera portal. Off by default because the `pipewire` crate links
# libpipewire-0.3 via pkg-config at build time (and its bindgen needs libclang),
# so it only belongs in builds that use it. V4L2 camera capture needs only what
# `capture` already does.
pipewire = ["capture", "dmabuf", "dep:pipewire", "dep:ashpd"]

Expand Down Expand Up @@ -136,10 +138,12 @@ yuv = "0.8.14"
ndk = { version = "0.9", optional = true, default-features = false, features = ["media", "api-level-26"] }

[target.'cfg(target_os="linux")'.dependencies]
# xdg-desktop-portal ScreenCast negotiation for the `pipewire` feature (pure Rust,
# zbus over DBus). The portal owns source selection: the user picks a monitor in a
# compositor dialog, and we get back a PipeWire fd + node id to stream from.
ashpd = { version = "0.13", optional = true, default-features = false, features = ["tokio", "screencast"] }
# xdg-desktop-portal ScreenCast and Camera negotiation for the `pipewire` feature
# (pure Rust, zbus over DBus). The ScreenCast portal owns source selection: the
# user picks a monitor in a compositor dialog, and we get back a PipeWire fd +
# node id to stream from. The Camera portal grants a sandboxed app a PipeWire
# remote that exposes the cameras.
ashpd = { version = "0.13", optional = true, default-features = false, features = ["tokio", "screencast", "camera"] }
# The NVIDIA codecs, behind the default-on `nvidia` feature. Everything is
# dlopen'd at runtime: cudarc loads CUDA, and moq-nvenc (an in-tree fork of
# nvidia-video-codec-sdk, see rs/moq-nvenc) loads libnvidia-encode. So the binary
Expand Down Expand Up @@ -176,7 +180,8 @@ moq-v4l = { workspace = true, optional = true }
# then falls back to the next candidate instead of failing at load, matching the
# NVENC and NVDEC backends.
moq-vaapi = { workspace = true, optional = true }
# PipeWire video stream for the portal's node, behind the `pipewire` feature.
# PipeWire video stream for a portal's node or a camera node, behind the
# `pipewire` feature.
# Links libpipewire-0.3 via pkg-config at build time; bindgen needs libclang.
pipewire = { version = "0.10", optional = true }
# X11 display/window enumeration and capture. Pure Rust and used when no
Expand Down
136 changes: 126 additions & 10 deletions rs/moq-video/src/capture/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
//! per-source:
//! - macOS camera -> AVFoundation, screen -> ScreenCaptureKit, both yielding
//! zero-copy `CVPixelBuffer` surfaces straight to VideoToolbox.
//! - Linux camera -> native V4L2 (YUYV / MJPEG -> CPU I420), X11 display and
//! window -> X11, Wayland display -> xdg-desktop-portal + PipeWire
//! (`pipewire` feature).
//! - Linux camera -> native V4L2 (YUYV / MJPEG -> CPU I420), or a PipeWire
//! camera node (`pipewire` feature), X11 display and window -> X11, Wayland
//! display -> xdg-desktop-portal + PipeWire (`pipewire` feature).
//! - Windows camera -> native Media Foundation (`IMFSourceReader`), screen ->
//! DXGI Desktop Duplication, window -> GDI (BGRA -> CPU I420).
//!
Expand Down Expand Up @@ -39,11 +39,14 @@ mod surface;
// Native V4L2 camera capture on Linux.
#[cfg(target_os = "linux")]
mod v4l2;
// The mode choice V4L2 and PipeWire cameras share.
#[cfg(target_os = "linux")]
mod mode;
// Native X11 display and window capture, including the portal fallback.
#[cfg(target_os = "linux")]
mod x11;

// Portal + PipeWire screen capture on Linux.
// Portal + PipeWire screen capture and PipeWire camera capture on Linux.
#[cfg(all(target_os = "linux", feature = "pipewire"))]
mod pipewire;

Expand Down Expand Up @@ -82,6 +85,14 @@ pub enum Source {
/// The identifiers from [`cameras`] are an AVFoundation `uniqueID` on macOS,
/// a `/dev/videoN` path on Linux, and a Media Foundation symbolic link on
/// Windows. Bare numeric indices remain accepted on Linux and Windows.
///
/// With the `pipewire` feature, Linux also lists PipeWire camera nodes as
/// `pipewire:<node name>`, and `pipewire` alone opens the camera with the
/// highest session priority, which the session manager treats as default.
/// The id picks the backend: a `/dev/videoN` path always opens V4L2, and a
/// `pipewire` id always opens PipeWire. `None` opens `/dev/video0`, except
/// inside a sandbox (Flatpak or Snap), where it opens PipeWire's default
/// camera through the camera portal.
Camera(Option<String>),

/// A whole display. `None` opens the main display.
Expand Down Expand Up @@ -364,7 +375,15 @@ pub async fn open(config: &Config) -> Result<Stream, Error> {
}
#[cfg(target_os = "linux")]
{
v4l2::open(config, device.as_deref()).await
match LinuxCamera::select(device.as_deref(), sandboxed()) {
LinuxCamera::V4l2(device) => v4l2::open(config, device).await,
#[cfg(feature = "pipewire")]
LinuxCamera::PipeWire(node) => pipewire::camera::open(config, node).await,
#[cfg(not(feature = "pipewire"))]
LinuxCamera::PipeWire(_) => Err(Error::Unsupported(
"PipeWire camera capture without the `pipewire` feature".to_string(),
)),
}
}
#[cfg(target_os = "windows")]
{
Expand Down Expand Up @@ -436,14 +455,23 @@ pub async fn open(config: &Config) -> Result<Stream, Error> {
}

/// List the available cameras and the identifiers [`Source::Camera`] accepts.
///
/// On Linux with the `pipewire` feature, V4L2 devices come first and PipeWire
/// camera nodes follow, so a webcam that PipeWire also serves appears once per
/// backend. A session without PipeWire lists V4L2 alone. Inside a sandbox the
/// PipeWire nodes come through the camera portal, which may ask the user for
/// camera access.
pub async fn cameras() -> Result<Vec<Camera>, Error> {
#[cfg(target_os = "macos")]
{
avfoundation::cameras()
}
#[cfg(target_os = "linux")]
{
blocking(v4l2::cameras).await
let cameras = blocking(v4l2::cameras).await?;
#[cfg(feature = "pipewire")]
let cameras = [cameras, pipewire::camera::cameras().await?].concat();
Ok(cameras)
}
#[cfg(target_os = "windows")]
{
Expand All @@ -462,9 +490,10 @@ pub async fn cameras() -> Result<Vec<Camera>, Error> {
/// are ones [`open`] could negotiate rather than everything the driver
/// advertises.
///
/// Linux only; other backends return [`Error::Unsupported`].
/// Linux only (V4L2, and PipeWire cameras with the `pipewire` feature); other
/// platforms return [`Error::Unsupported`].
/// Rates are exact device reports and [`Config::framerate`] accepts the same type.
/// V4L2 prefers the closest geometry, then the accepted rate
/// Both Linux backends prefer the closest geometry, then the accepted rate
/// nearest that request, then the cheaper conversion format.
///
/// An empty list is not a failure: it means the driver enumerated nothing this
Expand All @@ -473,8 +502,18 @@ pub async fn camera_modes(camera: Option<&str>) -> Result<Vec<Mode>, Error> {
let _ = camera;
#[cfg(target_os = "linux")]
{
let camera = camera.map(str::to_string);
blocking(move || v4l2::modes(camera.as_deref())).await
match LinuxCamera::select(camera, sandboxed()) {
LinuxCamera::V4l2(camera) => {
let camera = camera.map(str::to_string);
blocking(move || v4l2::modes(camera.as_deref())).await
}
#[cfg(feature = "pipewire")]
LinuxCamera::PipeWire(node) => pipewire::camera::modes(node).await,
#[cfg(not(feature = "pipewire"))]
LinuxCamera::PipeWire(_) => Err(Error::Unsupported(
"PipeWire camera modes without the `pipewire` feature".to_string(),
)),
}
}
#[cfg(not(target_os = "linux"))]
{
Expand Down Expand Up @@ -538,6 +577,52 @@ pub async fn apps() -> Result<Vec<App>, Error> {
}
}

/// The id prefix of a PipeWire camera, and on its own the default one.
#[cfg(any(target_os = "linux", test))]
const PIPEWIRE: &str = "pipewire";

/// The Linux backend a [`Source::Camera`] selector names.
#[cfg(any(target_os = "linux", test))]
#[derive(Debug, PartialEq, Eq)]
enum LinuxCamera<'a> {
/// A V4L2 device, by path or index. `None` opens `/dev/video0`.
V4l2(Option<&'a str>),
/// A PipeWire camera, by `node.name`. `None` picks the camera with the
/// highest session priority, the one the session manager links by default.
PipeWire(Option<&'a str>),
}

#[cfg(any(target_os = "linux", test))]
impl<'a> LinuxCamera<'a> {
/// Parse a camera selector. A sandbox has no V4L2 devices to open by
/// default, so its default camera is PipeWire's, reached through the portal.
fn select(selector: Option<&'a str>, sandboxed: bool) -> Self {
match selector {
None if sandboxed => Self::PipeWire(None),
None => Self::V4l2(None),
Some(PIPEWIRE) => Self::PipeWire(None),
Some(selector) => match selector.strip_prefix(PIPEWIRE).and_then(|rest| rest.strip_prefix(':')) {
Some(name) => Self::PipeWire(Some(name)),
None => Self::V4l2(Some(selector)),
},
}
}
}

/// Whether this process runs in a Flatpak or Snap sandbox. Without the
/// `pipewire` feature there is no portal to use, so it never matters.
#[cfg(target_os = "linux")]
fn sandboxed() -> bool {
#[cfg(feature = "pipewire")]
{
ashpd::is_sandboxed()
}
#[cfg(not(feature = "pipewire"))]
{
false
}
}

/// Run synchronous platform enumeration off the async runtime's worker threads.
#[cfg(any(target_os = "linux", target_os = "windows"))]
async fn blocking<T, F>(f: F) -> Result<T, Error>
Expand All @@ -549,3 +634,34 @@ where
.await
.map_err(|err| Error::Codec(anyhow::anyhow!("capture enumeration thread failed: {err}")))?
}

#[cfg(test)]
mod tests {
use super::*;

#[test]
fn camera_selectors_name_their_backend() {
assert_eq!(LinuxCamera::select(None, false), LinuxCamera::V4l2(None));
assert_eq!(LinuxCamera::select(None, true), LinuxCamera::PipeWire(None));
for sandboxed in [false, true] {
assert_eq!(
LinuxCamera::select(Some("/dev/video2"), sandboxed),
LinuxCamera::V4l2(Some("/dev/video2"))
);
assert_eq!(LinuxCamera::select(Some("1"), sandboxed), LinuxCamera::V4l2(Some("1")));
assert_eq!(
LinuxCamera::select(Some("pipewire"), sandboxed),
LinuxCamera::PipeWire(None)
);
assert_eq!(
LinuxCamera::select(Some("pipewire:v4l2_input.pci-0000_00_14.0-usb-0_4_1.0"), sandboxed),
LinuxCamera::PipeWire(Some("v4l2_input.pci-0000_00_14.0-usb-0_4_1.0"))
);
}
// Only the whole prefix selects PipeWire; anything else is a device path.
assert_eq!(
LinuxCamera::select(Some("pipewired"), false),
LinuxCamera::V4l2(Some("pipewired"))
);
}
}
Loading
Loading