Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 3 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -131,7 +131,9 @@ moq-token-cli = { version = "0.5.46", path = "rs/moq-token-cli" }
moq-transcode = { version = "0.0.15", path = "rs/moq-transcode", default-features = false }
# Standalone crate (moq-dev/vaapi); vendored from cros-libva + cros-codecs.
# dlopen's libva at runtime (no libva-dev at build, no NEEDED libva in the binary).
moq-vaapi = "0.0.3"
# The decoder this crate's VAAPI backend calls is not in 0.0.3 yet; see
# moq-dev/vaapi#2. Point this back at a release once one carries `decode`.
moq-vaapi = { git = "https://github.com/Frando/vaapi", branch = "pr/decode" }

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Keep the VAAPI dependency publishable

When the Rust release workflow packages moq-video, Cargo rejects this inherited git dependency because it has no version requirement, even though it is optional in moq-video (cargo package reports that all dependencies must specify a version). Adding the old 0.0.3 version would not suffice because packaged manifests discard the git source and that release lacks the new decode API. This therefore blocks release-plz from publishing the crate until moq-vaapi has a crates.io release containing decode and the workspace dependency points to it.

Useful? React with 👍 / 👎.

# `default-features = false` is here for `capture` (V4L2, libclang), which the
# cross-compiled Python, Swift, Kotlin, and Go builds have no use for: adding
# `features = ["capture"]` to a consumer that ships in those bindings pulls the
Expand Down
21 changes: 11 additions & 10 deletions rs/moq-video/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -35,11 +35,12 @@ nvidia = ["dep:cudarc", "dep:moq-nvenc", "dep:libloading"]
# feature tables name only `nvidia`.
nvenc = ["nvidia"]
nvdec = ["nvidia"]
# Intel/AMD VAAPI hardware encoder (Linux). Off by default because the backend has
# never been validated on real hardware, not because of anything it costs the
# build: moq-vaapi dlopen's libva, so an enabled-vaapi build needs no libva at
# build time and still starts on a libva-less host, where automatic backend
# selection falls back to the next encoder like the NVENC backend does.
# Intel/AMD VAAPI hardware codecs (Linux): the H.264 encoder and decoder. Off by
# default because the encoder has never been validated on real hardware (the
# decoder has), not because of anything it costs the build: moq-vaapi dlopen's
# libva, so an enabled-vaapi build needs no libva at build time and still starts
# on a libva-less host, where automatic backend selection falls back to the next
# candidate like the NVENC and NVDEC backends do.
vaapi = ["dmabuf", "dep:moq-vaapi"]
# Linux DMA-BUF surface vocabulary and CPU fallback support. Enabled by every
# backend that can produce or consume one. It pulls no graphics API by itself.
Expand Down Expand Up @@ -125,11 +126,11 @@ libloading = { workspace = true, optional = true }
# Architecture-correct DMA_BUF_IOCTL_SYNC request values for CPU access.
linux-raw-sys = { workspace = true, optional = true }
moq-nvenc = { workspace = true, optional = true }
# Intel/AMD VAAPI hardware encoder, behind the opt-in `vaapi` feature. As of
# moq-vaapi 0.0.3 libva is dlopen'd at runtime, so the binary carries no NEEDED
# libva: the build needs no libva-dev, and a libva-less host still loads. Automatic
# backend selection then falls back to the next encoder instead of failing at load,
# matching the NVENC backend.
# Intel/AMD VAAPI hardware H.264 encoder and decoder, behind the opt-in `vaapi`
# feature. As of moq-vaapi 0.0.3 libva is dlopen'd at runtime, so the binary carries
# no NEEDED libva: the build needs no libva-dev, and a libva-less host still loads.
# Automatic backend selection then falls back to the next candidate instead of
# failing at load, matching the NVENC and NVDEC backends.
moq-vaapi = { workspace = true, optional = true }
# PipeWire video stream for the portal's node, behind the `pipewire` feature.
# Links libpipewire-0.3 via pkg-config at build time; bindgen needs libclang.
Expand Down
33 changes: 31 additions & 2 deletions rs/moq-video/src/decode/backend/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@
//!
//! [`open`] picks the best backend for a [`Codec`] and [`Config`], trying
//! hardware candidates (platform-gated: VideoToolbox on macOS, Media Foundation
//! / DXVA on Windows, NVDEC on Linux) before the openh264 software fallback,
//! exactly like the encode side. Only backends that support the requested codec
//! / DXVA on Windows, NVDEC then VAAPI on Linux) before the openh264 software
//! fallback, exactly like the encode side. Only backends that support the requested codec
//! are considered: there is no software H.265 or AV1 decoder, so those tracks
//! have no fallback below the hardware path.

Expand All @@ -33,6 +33,11 @@ mod mediafoundation;
#[cfg(all(target_os = "linux", feature = "nvidia"))]
mod nvdec;

// Crate-visible so `decode::Consumer`'s end-of-track test can name the one
// backend that holds pictures back; nothing outside the tests reaches for it.
#[cfg(all(target_os = "linux", feature = "vaapi"))]
pub(crate) mod vaapi;

/// The video codec a decoder handles. Derived from the catalog, not chosen by the
/// caller.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
Expand Down Expand Up @@ -64,6 +69,24 @@ pub(crate) trait Backend: Send {
/// reordering.
fn decode(&mut self, access_unit: Bytes, timestamp: Timestamp, keyframe: bool) -> Result<Vec<Frame>, Error>;

/// Return the pictures the codec is still holding, in output order, once the
/// stream has ended.
///
/// The last access unit of a stream is not the last picture out of a decoder
/// that buffers: H.264 releases a picture from the DPB only when a later one
/// needs its slot, so the tail sits there until something asks for it. That is
/// several pictures, not one, since the DPB bumps against the sequence's
/// reference and reorder limits rather than the reorder depth actually used.
///
/// Defaults to no frames, which is right for every backend configured for zero
/// delay: openh264, Media Foundation with its reorder buffer disabled, NVDEC
/// with `ulMaxDisplayDelay` at zero, and VideoToolbox decoding without temporal
/// processing all hand each picture back within the call that fed it. Override
/// it in a backend that holds pictures across calls, or its stream ends short.
fn flush(&mut self) -> Result<Vec<Frame>, Error> {
Ok(Vec::new())
}

/// The decoder name in use, e.g. `"videotoolbox"` (for logging).
fn name(&self) -> &str;
}
Expand Down Expand Up @@ -99,6 +122,12 @@ const HARDWARE: &[Candidate] = &[
supports: |c| matches!(c, Codec::H264 | Codec::H265 | Codec::Av1),
open: nvdec::Nvdec::open,
},
#[cfg(all(target_os = "linux", feature = "vaapi"))]
Candidate {
name: vaapi::NAME,
supports: |c| matches!(c, Codec::H264),
open: vaapi::Vaapi::open,
},
];

const SOFTWARE: Candidate = Candidate {
Expand Down
269 changes: 269 additions & 0 deletions rs/moq-video/src/decode/backend/vaapi.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,269 @@
//! Intel/AMD VAAPI hardware H.264 decode via the opt-in `moq-vaapi` crate on Linux.
//!
//! The decode half of [`encode::backend::vaapi`](crate::encode). `moq-vaapi` is a
//! focused VA-API H.264 codec vendored and trimmed from cros-libva +
//! discord/cros-codecs. Its decoder takes one Annex-B access unit with the
//! parameter sets inline and hands back tightly-packed NV12, which this
//! deinterleaves to the CPU I420 the rest of the crate speaks.
//!
//! libva is `dlopen`'d at runtime, so a VAAPI-enabled build needs no libva at
//! build time and the binary carries no `NEEDED libva`. A libva-less host, a
//! missing render node, or a driver with no H.264 decode entrypoint makes
//! `Decoder::new` return an error; under automatic selection
//! [`backend::open`](super::open) then moves on to the next candidate, like the
//! NVDEC backend.
//!
//! Progressive 8-bit 4:2:0 only, which is everything a browser's `VideoEncoder`,
//! WebRTC, or this crate's own encoders emit. The decoder rejects an interlaced or
//! high-bit-depth sequence at its first SPS, which surfaces here as a decode
//! error rather than wrong pixels.
//!
//! Unlike NVDEC, which cuvid lets us pin to zero display delay, output trails the
//! input even without B-frames: H.264's DPB releases a picture only once a later
//! one needs its slot, and the slot count comes from the sequence's reference and
//! reorder limits rather than the reorder depth actually used. A stream coded with
//! three reference frames therefore keeps three pictures in hand, which is why
//! this backend implements [`Backend::flush`] and the layers above call it when
//! the track ends. The decoded frames carry their own timestamps, so the delay
//! itself needs nothing downstream.
//!
//! The output is CPU I420 rather than a zero-copy [`Surface::DmaBuf`]: VA-API
//! decode targets on Intel come back Y-tiled (DRM modifier `0x100000000000002`),
//! a modifier the Vulkan renderer cannot import, so a DMA-BUF here would only move
//! the download somewhere less convenient.

use bytes::Bytes;
use moq_net::Timestamp;
use moq_vaapi::decode::{Config as VaapiConfig, Decoder};

use super::{Backend, Codec, Config};
use crate::frame::{I420, Surface};
use crate::{Error, Frame};

pub(crate) const NAME: &str = "vaapi";

pub(crate) struct Vaapi {
decoder: Decoder,
}

// The decoder is `!Send` (libva uses `Rc` internally) but is created, used, and
// dropped only on the dedicated decode thread (see `decode::sink`); the `Send`
// impl just lets the boxed trait object satisfy `Backend: Send`.
unsafe impl Send for Vaapi {}

impl Vaapi {
/// VA-API H.265 and AV1 decode exist but are not wired up in `moq-vaapi`, so
/// this handles H.264 only. `config` carries no hardware scaler request we can
/// honor: VA-API's scaler is a separate VPP pipeline, so callers scale the
/// frames themselves.
pub(crate) fn open(codec: Codec, _config: &Config) -> Result<Box<dyn Backend>, Error> {
if codec != Codec::H264 {
return Err(Error::Codec(anyhow::anyhow!("VAAPI cannot decode {}", codec.label())));
}

let decoder =
Decoder::new(VaapiConfig::new()).map_err(|e| Error::Codec(anyhow::anyhow!("VAAPI decoder init: {e:?}")))?;

tracing::info!(decoder = NAME, "opened H.264 decoder");
Ok(Box::new(Self { decoder }))
}
}

impl Backend for Vaapi {
fn decode(&mut self, access_unit: Bytes, timestamp: Timestamp, _keyframe: bool) -> Result<Vec<Frame>, Error> {
// The timestamp rides through the decoder with the picture, so it
// survives the DPB reordering a stream with B-frames goes through.
let decoded = self
.decoder
.decode(&access_unit, timestamp.as_micros() as u64)
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
.map_err(|e| Error::Codec(anyhow::anyhow!("VAAPI decode: {e:?}")))?;

convert(decoded)
}

/// The tail of the stream, which is why this backend overrides the trait's
/// no-op: the DPB is holding as many pictures as the sequence's reference and
/// reorder limits allow, and nothing else will ever ask for them.
fn flush(&mut self) -> Result<Vec<Frame>, Error> {
let decoded = self
.decoder
.flush()
.map_err(|e| Error::Codec(anyhow::anyhow!("VAAPI flush: {e:?}")))?;

convert(decoded)
}

fn name(&self) -> &str {
NAME
}
}

/// Deinterleave each decoded picture's NV12 into the CPU I420 the rest of the
/// crate speaks, keeping the timestamp the picture was coded with.
fn convert(decoded: Vec<moq_vaapi::decode::Frame>) -> Result<Vec<Frame>, Error> {
decoded
.into_iter()
.map(|frame| {
let i420 = I420::from_nv12(&frame.data, frame.width, frame.height)?;
let timestamp = Timestamp::from_micros(frame.timestamp).unwrap_or(Timestamp::ZERO);
Ok(Frame::new(Surface::I420(i420), timestamp))
})
.collect()
}

#[cfg(test)]
mod tests {
use super::*;
use crate::decode::{Config as DecodeConfig, Kind as DecodeKind};
use crate::encode::{Config as EncodeConfig, Encoder, Kind as EncodeKind};

/// Real hardware only: skip on a box with no libva or no VA-API H.264 decode
/// entrypoint, so these are no-ops on CI and validate on an Intel or AMD box.
fn hw_available() -> bool {
Decoder::new(VaapiConfig::new()).is_ok()
}

fn decode_config() -> DecodeConfig {
DecodeConfig {
kind: DecodeKind::Named(NAME.into()),
..DecodeConfig::new()
}
}

/// On a host with no usable VA stack, opening must return an `Err` (so
/// `Kind::Auto` falls through to openh264) rather than panicking inside libva.
/// A no-op on a box that does have one.
#[test]
fn missing_driver_errors_instead_of_panicking() {
if hw_available() {
return;
}
assert!(Vaapi::open(Codec::H264, &decode_config()).is_err());
}

/// A static RGBA gradient that varies in both axes, so the chroma planes have
/// spatial structure and a pitch or plane-split bug corrupts the picture.
fn gradient_rgba(width: u32, height: u32) -> Vec<u8> {
let (w, h) = (width as usize, height as usize);
let mut buf = vec![0u8; w * h * 4];
for y in 0..h {
for x in 0..w {
let i = (y * w + x) * 4;
buf[i] = (x * 255 / w) as u8;
buf[i + 1] = (y * 255 / h) as u8;
buf[i + 2] = ((x + y) * 255 / (w + h)) as u8;
buf[i + 3] = 255;
}
}
buf
}

/// Mean absolute error between two equal-length planes.
fn mae(a: &[u8], b: &[u8]) -> u64 {
assert_eq!(a.len(), b.len());
a.iter().zip(b).map(|(x, y)| x.abs_diff(*y) as u64).sum::<u64>() / a.len() as u64
}

/// H.264 through the real hardware: openh264 encodes a gradient (Annex-B with
/// inline SPS/PPS) and VA-API decodes it. Asserts the downloaded picture
/// matches the input, which a plane split or stride bug would shear, and that
/// the timestamp rides through with its picture.
#[test]
fn vaapi_h264_round_trip() {
if !hw_available() {
return;
}
let (w, h) = (320u32, 240u32);
let rgba = gradient_rgba(w, h);
let expected = I420::from_rgba(&rgba, w * 4, w, h).unwrap();

let mut encoder = Encoder::new(&EncodeConfig {
kind: EncodeKind::Software,
..EncodeConfig::new(w, h, 30)
})
.unwrap();
let mut decoder = Vaapi::open(Codec::H264, &decode_config()).expect("VAAPI H.264 decoder");

let mut decoded = Vec::new();
for i in 0..10u64 {
if i == 0 {
encoder.keyframe();
}
let surface = Surface::rgba(&rgba, crate::Size::new(w, h)).unwrap();
let frame = Frame::new(surface, Timestamp::from_micros(i * 33_333).unwrap());
for encoded in encoder.encode(&frame).unwrap() {
decoded.extend(decoder.decode(encoded.payload, encoded.timestamp, i == 0).unwrap());
}
}

assert!(!decoded.is_empty(), "VAAPI produced no frames");
for (i, frame) in decoded.iter().enumerate() {
// openh264 encodes IPPP, so the pictures come back in feed order.
assert_eq!(
frame.timestamp.as_micros(),
i as u128 * 33_333,
"timestamp did not ride the picture"
);
let i420 = frame.surface.to_i420().unwrap();
assert_eq!((i420.width(), i420.height()), (w, h));
assert!(mae(i420.y(), expected.y()) < 8, "Y plane corrupt");
assert!(mae(i420.u(), expected.u()) < 8, "U plane corrupt");
assert!(mae(i420.v(), expected.v()) < 8, "V plane corrupt");
}
}

/// Regression: every picture fed in comes back out, which needs the flush.
///
/// H.264 releases a picture from the DPB only once a later one needs its slot,
/// so a stream simply stopping leaves its tail there. How many depends on the
/// sequence's reference and reorder limits, not on the reorder depth the
/// stream used: openh264 codes one reference frame and loses the last picture,
/// x264's default three loses three. The `decode` half of the assertion is
/// what keeps this honest, since a decoder that held nothing back would pass
/// the rest of it without a flush ever running.
#[test]
fn flushing_returns_the_tail_the_dpb_holds() {
if !hw_available() {
return;
}
const FRAMES: u64 = 5;
let (w, h) = (320u32, 240u32);
let rgba = gradient_rgba(w, h);

let mut encoder = Encoder::new(&EncodeConfig {
kind: EncodeKind::Software,
..EncodeConfig::new(w, h, 30)
})
.unwrap();
let mut decoder = Vaapi::open(Codec::H264, &decode_config()).expect("VAAPI H.264 decoder");

let mut streamed = Vec::new();
for i in 0..FRAMES {
if i == 0 {
encoder.keyframe();
}
let surface = Surface::rgba(&rgba, crate::Size::new(w, h)).unwrap();
let frame = Frame::new(surface, Timestamp::from_micros(i * 33_333).unwrap());
for encoded in encoder.encode(&frame).unwrap() {
streamed.extend(decoder.decode(encoded.payload, encoded.timestamp, i == 0).unwrap());
}
}
assert!(
(streamed.len() as u64) < FRAMES,
"the DPB held nothing back, so this test proves nothing"
);

let flushed = decoder.flush().unwrap();
let timestamps: Vec<u128> = streamed
.iter()
.chain(&flushed)
.map(|frame| frame.timestamp.as_micros())
.collect();
let expected: Vec<u128> = (0..FRAMES as u128).map(|i| i * 33_333).collect();
assert_eq!(timestamps, expected, "the stream lost pictures at its end");

// A second flush has nothing left to hand back, so the drain is idempotent
// and a caller that flushes twice does not see a picture twice.
assert!(decoder.flush().unwrap().is_empty());
}
}
Loading
Loading