Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -179,7 +179,7 @@ moq-uring = { version = "0.0.2", path = "rs/moq-uring", default-features = false
moq-v4l = { version = "0.0.1", path = "rs/moq-v4l" }
# Standalone crate (moq-dev/vaapi); vendored from cros-libva + cros-codecs.
# dlopen's libva at runtime (no libva-dev at build, no NEEDED libva in the binary).
moq-vaapi = "0.0.4"
moq-vaapi = "0.1.0"
# `default-features = false` is here for `capture` (V4L2, libclang), which the
# cross-compiled Python, Swift, Kotlin, and Go builds have no use for: adding
# `features = ["capture"]` to a consumer that ships in those bindings pulls the
Expand Down
20 changes: 11 additions & 9 deletions quest/m4/video-vaapi.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,21 +9,23 @@ release first.

## Plan

Four gaps, one external dependency.
Three gaps, one external dependency.

**Decode.** The H.264 decoder landed (moq-vaapi 0.0.4, `decode/backend/vaapi.rs`),
with the default `decode::Config::output` of `Output::Native` handing out
DMA-BUF surfaces the renderer imports without a download. H.265 decode is still missing, so a Linux box
without NVDEC has no hardware path for it.

**The encoder.** Ours is a 111-line CPU-only adapter whose own header says it
is unvalidated on hardware. iroh-live's imports a DMA-BUF directly and does
scale and convert through VPP, validated on Intel Meteor Lake. Take theirs and
reshape it to our surface rather than growing ours toward it.
**The encoder.** H.264 is done. `moq-vaapi` imports DMA-BUFs and scales and
converts them through VPP (`dmabuf`, `vpp`, `Encoder::encode_dmabuf`,
`Encoder::set_bitrate`); `encode/backend/vaapi.rs` encodes a `Surface::DmaBuf`
without a download, and `Surface::resize` scales one through VPP. Validated on
Intel Meteor Lake. What is left is H.265, below.

**H.265.** The VAAPI backend advertises H.264 only. `moq-vaapi` 0.0.2 vendors
the HEVC buffer types (`src/buffer/hevc.rs`) but its `Encoder` is hardcoded to
`VAProfileH264Main` / `VAEntrypointEncSlice`, so exposing an HEVC encoder is a
`VAProfileH264Main` (with `VAEntrypointEncSlice`, or the low-power entrypoint
where that is all a device has), so exposing an HEVC encoder is a
change to that crate, not a flag here.

**Build cost.** `moq-vaapi` 0.0.3 dlopens libva (no `DT_NEEDED`), so a
Expand All @@ -43,6 +45,6 @@ already falls back cleanly, since `Encoder::new` returns `Err` and

## Required

- A `moq-dev/vaapi` release exposing an HEVC encoder, the decode half and a
VPP wrapper, and shipping pre-generated bindings instead of a bindgen build
script
- A `moq-dev/vaapi` release exposing an HEVC encoder (H.264 decode is in
0.0.4; DMA-BUF encode and VPP shipped in 0.1.0) and pre-generated bindings
instead of a bindgen build script
4 changes: 2 additions & 2 deletions rs/moq-video/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,8 @@ openh264 = ["dep:openh264", "dep:openh264-sys2"]
# default because moq-vaapi's build script runs bindgen over its vendored libva
# headers, so the build host needs libclang. At runtime libva is dlopen'd: the
# binary carries no NEEDED libva and still starts on a libva-less host, where
# automatic backend selection falls back to the next candidate. The decoder is
# hardware-validated, while the encoder is not yet.
# automatic backend selection falls back to the next candidate. Both are
# validated on Intel Meteor Lake.
vaapi = ["dmabuf", "dep:moq-vaapi"]
# The V4L2 stateful memory-to-memory hardware codecs on Linux: the H.264 encoder
# and decoder most ARM SoCs expose (a Raspberry Pi's VideoCore, and the
Expand Down
43 changes: 21 additions & 22 deletions rs/moq-video/DESIGN-native-codecs.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@
Status: **phases 2-6 implemented** (openh264, VideoToolbox, NVENC, VAAPI, capture
swap + ffmpeg removal). Capture is now native on all three platforms (AVFoundation
/ ScreenCaptureKit on macOS, V4L2 on Linux, Media Foundation on Windows), so
**nokhwa is fully removed**. VAAPI is back via discord/cros-codecs with an NV12
surface-upload input path; the zero-copy dmabuf capture is a follow-up. See "As
built" at the bottom for where the implementation diverged from this plan.
**nokhwa is fully removed**. VAAPI is back via moq-vaapi, which encodes DMA-BUFs
on the GPU and uploads CPU frames as NV12. See "As built" at the bottom for where
the implementation diverged from this plan.

## Goal

Expand Down Expand Up @@ -412,25 +412,24 @@ software (see `backend::open`).
> libva and vendors the libva headers its bindgen reads, so VAAPI costs the build
> nothing: no libva-dev, no `NEEDED libva.so.2`, no entry in the nix devShell.

**Input is an NV12 surface upload, not zero-copy dmabuf.** The encoder wants an
NV12 VA surface, but UVC webcams deliver YUYV/MJPEG (decoded to CPU I420); they
rarely expose NV12 to import zero-copy. So `backend/vaapi.rs` drives
`new_native_vaapi` with a `VaSurfacePool`: each frame uploads I420 into a pooled
surface as NV12 (`libva::Image`, honoring plane pitches) and encodes the surface.
This works with the existing CPU V4L2 capture, no new capture code.

Follow-up (not in this PR): the **zero-copy dmabuf path** for the rare NV12-capable
V4L2 source. Re-add `Frame::DmaBuf`, a V4L2 `VIDIOC_EXPBUF` capture (the `v4l`
crate exposes the raw ioctl but no dmabuf stream), and a `requires_dmabuf` capture
coupling, then import the dmabuf into a VA surface (`MemoryType::DrmPrime2`).

**NOT YET VALIDATED ON HARDWARE.** Compiles on Linux with libva headers; written
against discord/cros-codecs `discord-0.0.5` with type/field names checked against
source. Needs a Linux + Intel/AMD GPU to confirm: (1) the `low_power` entrypoint
(recent Intel iHD often requires the low-power encode entrypoint, AMD the full
one; we request full and let `Kind::Auto` fall back); (2) the NV12 upload
pitch/offset handling round-trips; (3) `cargo deny` accepts the new transitive
licenses (drm, drm-fourcc, etc.) once the vaapi graph resolves.
**Input: DMA-BUFs stay on the GPU, CPU frames are uploaded.** A
`Surface::DmaBuf` is encoded without a download. An NV12 buffer at the encoder's
size (a VA-API decode, or one `Surface::resize` already scaled through VPP) is
imported and encoded in place when that size is a whole number of macroblocks;
anything else the driver imports, packed RGB from a PipeWire screen capture in
particular, goes through VPP into the encoder's own surface. A layout the driver
refuses falls back to the CPU path and is not tried again. Every other surface
is converted to I420, interleaved to NV12, and uploaded into the encoder's input
surface, which is the path a UVC webcam's YUYV or MJPEG frames take. A V4L2 capture that exports DMA-BUFs (`VIDIOC_EXPBUF`) would let an
NV12-capable camera skip the upload too; it is not built.

**Validated on Intel Meteor Lake** (iHD 26.1.5) by the tests in
`encode/backend/vaapi.rs`: they encode CPU frames, VA-API decodes handed over as
DMA-BUFs, and packed RGB buffers at and above the encoder's size, decode each
stream with openh264, and compare pixels. They skip without a VA-API device.
moq-vaapi opens the full encode entrypoint and falls back to the low-power one
(`VAEntrypointEncSliceLP`) that some Intel parts expose alone; no part that
needs the fallback has run it yet.

### NVENC ships via dlopen (no driver dependency at build or load)

Expand Down
120 changes: 4 additions & 116 deletions rs/moq-video/src/decode/backend/vaapi.rs
Original file line number Diff line number Diff line change
Expand Up @@ -47,15 +47,12 @@
//! back through `vaDeriveImage` rather than trying to read a tiled buffer as
//! rows.

use std::os::fd::{AsFd, OwnedFd};
use std::sync::{Arc, Mutex};

use bytes::Bytes;
use moq_net::Timestamp;
use moq_vaapi::decode::{Config as VaapiConfig, Decoder, ExportedFrame};

use super::{Backend, Codec, Config};
use crate::frame::{DmaBuf, DmaBufFrame, DmaBufPlane, DrmFormat, I420, Surface};
use crate::frame::{I420, Surface, vaapi};
use crate::{Error, Frame, Output};

pub(crate) const NAME: &str = "vaapi";
Expand Down Expand Up @@ -204,122 +201,13 @@ fn share(exported: Vec<ExportedFrame>) -> anyhow::Result<Vec<Frame>> {
.into_iter()
.map(|frame| {
let timestamp = Timestamp::from_micros(frame.timestamp).unwrap_or(Timestamp::ZERO);
Ok(Frame::new(Surface::DmaBuf(adopt(frame)?), timestamp))
// A decode target names no color space, so the renderer infers one
// from the frame size exactly as it does for a downloaded picture.
Ok(Frame::new(Surface::DmaBuf(vaapi::adopt(frame, None)?), timestamp))
})
.collect()
}

/// Describe an exported picture as a [`DmaBuf`]: the driver's format modifier,
/// and the offset and pitch of each of its memory planes.
///
/// The width and height are the visible frame rather than the exported extent,
/// which is the driver's padded allocation. Neither the pitches nor the offsets
/// follow from the visible size, which is exactly why they are read off the
/// export rather than computed from it.
///
/// # Errors
///
/// When the export is not the one shape a [`DmaBuf`] can describe: a single NV12
/// layer whose planes all live in a single object. The Intel and AMD drivers
/// export exactly that, and the alternatives are refused rather than guessed at,
/// because every one of them draws as a plausible-looking picture made of the
/// wrong bytes.
fn adopt(frame: ExportedFrame) -> anyhow::Result<DmaBuf> {
let (width, height) = (frame.width, frame.height);
// One object, because a consumer imports every plane from the one descriptor
// `Exported::export` hands out, and one layer, because the planes are read
// off it as a group. Both are what `VA_EXPORT_SURFACE_COMPOSED_LAYERS` asks
// for; neither is what it guarantees.
let [object] = frame.descriptor.objects.as_slice() else {
anyhow::bail!(
"VA-API exported {} objects, expected one holding every plane",
frame.descriptor.objects.len()
);
};
let [layer] = frame.descriptor.layers.as_slice() else {
anyhow::bail!(
"VA-API exported {} layers, expected one composed layer",
frame.descriptor.layers.len()
);
};
if layer.drm_format != DrmFormat::NV12.as_raw() {
anyhow::bail!("VA-API exported DRM format {:#x}, expected NV12", layer.drm_format);
}

// `num_planes` and the arrays it indexes both come from the driver, and only
// the arrays are bounded, so indexing on the count would panic rather than
// fail.
let count = layer.num_planes as usize;
anyhow::ensure!(
count <= layer.offset.len(),
"VA-API exported {count} planes, more than a PRIME descriptor holds"
);
let planes = (0..count)
.map(|plane| DmaBufPlane::new(layer.offset[plane], layer.pitch[plane]))
.collect();
let modifier = object.drm_format_modifier;

// A decode target names no color space, so the renderer infers one from the
// frame size exactly as it does for a downloaded picture.
DmaBuf::new(
DrmFormat::NV12,
modifier,
width,
height,
planes,
None,
Arc::new(Exported::new(frame)),
)
.map_err(|e| anyhow::anyhow!("{e}"))
}

/// A decoded picture the consumer holds as a DMA-BUF.
///
/// Both of the things a consumer can do with one: hand a descriptor to a
/// graphics API, or give up on drawing it and read the pixels back. Dropping the
/// last clone destroys the surface, which is what returns its allocation to the
/// driver.
struct Exported {
/// Locked because [`DmaBufFrame`] hands out `&self` while what is behind it
/// is a single libva surface: `download_i420` maps that surface, and two
/// threads doing so at once is more than libva promises to serialize. The
/// frame is [`Send`] on its own, so a lock is enough and no `unsafe impl` is
/// involved.
frame: Mutex<ExportedFrame>,
}

impl Exported {
fn new(frame: ExportedFrame) -> Self {
Self {
frame: Mutex::new(frame),
}
}
}

impl DmaBufFrame for Exported {
/// Vulkan takes ownership of an imported descriptor on success and closes it
/// on failure, so every import needs one of its own and the original stays
/// with the picture.
fn export(&self) -> std::io::Result<OwnedFd> {
let frame = self.frame.lock().expect("poisoned");
let object = frame.descriptor.objects.first().ok_or_else(|| {
std::io::Error::new(std::io::ErrorKind::InvalidData, "the VA-API export carries no object")
})?;
object.fd.as_fd().try_clone_to_owned()
}

/// Read the picture back through the retained surface rather than the
/// descriptor: a decode target is tiled, so mapping the file descriptor as
/// rows would be wrong.
fn download_i420(&self) -> Result<I420, Error> {
let frame = self.frame.lock().expect("poisoned");
let nv12 = frame
.download()
.map_err(|e| Error::Codec(anyhow::anyhow!("read a VA-API decode surface back: {e:?}")))?;
I420::from_nv12(&nv12.data, crate::Size::new(nv12.width, nv12.height))
}
}

#[cfg(test)]
mod tests {
use super::*;
Expand Down
2 changes: 1 addition & 1 deletion rs/moq-video/src/encode/backend/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -357,7 +357,7 @@ pub(crate) mod test_util {
/// BT.601 goes out as SMPTE 170M primaries and matrix (code point 6) with the
/// BT.709 transfer curve (1). The two curves are defined identically, and
/// CoreVideo's SMPTE 170M transfer constant is deprecated while Media
/// Foundation has none at all, so 1 is the only value all four backends can
/// Foundation has none at all, so 1 is the only value all five backends can
/// actually emit.
pub(crate) const BT601_DESCRIBED: Described = Described {
primaries: 6,
Expand Down
Loading
Loading