Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 8 additions & 6 deletions doc/bin/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -370,10 +370,12 @@ moq --client-connect https://relay.example.com/anon --broadcast screen.hang \
import capture --display --system-audio
```

On Linux the NVENC (NVIDIA) and VAAPI (Intel/AMD) encoders and the PipeWire
On Linux the NVENC (NVIDIA), VAAPI (Intel/AMD), and V4L2 M2M (an ARM SoC's
codec block, such as a Raspberry Pi's VideoCore) encoders and the PipeWire
screen capture are compiled in by default. To build `capture` without any of
them (software openh264 + V4L2 camera capture only), drop the default features. `capture` itself still needs libclang and the V4L2 headers for
the camera, and ALSA for the microphone:
them (software openh264 + V4L2 camera capture only), drop the default features.
`capture` itself still needs libclang and the V4L2 headers for the camera, and
ALSA for the microphone:

```bash
cargo build --release -p moq-cli --no-default-features \
Expand All @@ -398,9 +400,9 @@ The Linux screen backend links libpipewire and is behind the default-on
`pipewire` feature; drop it (like the codecs above) for a build without the
dependency. The codec is chosen with `--codec` (`h264` default, or `h265`). For
H.264 it picks a hardware encoder (VideoToolbox on macOS, NVENC on Linux NVIDIA,
or VAAPI on Linux Intel/AMD when built with the `vaapi` feature) when one is
present, falling back to the built-in software encoder (openh264); force either
with `--hardware` / `--software`. A hardware encoder that was compiled in but
VAAPI on Linux Intel/AMD, or an ARM SoC's V4L2 M2M encoder) when one is present,
falling back to the built-in software encoder (openh264); force either with
`--hardware` / `--software`. A hardware encoder that was compiled in but
can't open, typically because its driver libraries aren't on the loader path,
warns rather than falling back quietly. H.265 is hardware-only (VideoToolbox on macOS,
Media Foundation on Windows). `--camera` takes a bare integer as a device index,
Expand Down
1 change: 1 addition & 0 deletions doc/lib/rs/crate/moq-video.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ cargo add moq-video --features pipewire
| `capture` | yes | Native device capture (`v4l` and `zune-jpeg` on Linux) |
| `nvidia` | yes | NVENC encode and NVDEC decode on Linux (`cudarc`, `moq-nvenc`), dlopen'd at runtime |
| `vaapi` | yes | Intel/AMD encode on Linux (`moq-vaapi`), dlopen'd at runtime; not yet validated on hardware |
| `v4l2` | yes | V4L2 M2M encode and decode on ARM SoCs (Raspberry Pi and friends), validated on a Raspberry Pi 4 |
| `render` | yes | `wgpu`, the GPU renderer, and Linux DMA-BUF support |
| `pipewire` | no | Wayland screen capture via xdg-desktop-portal and DMA-BUF |

Expand Down
3 changes: 2 additions & 1 deletion quest/m3/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ its verdict lands and the follow-on work becomes concrete.

## Quests

- [Embedded video](/quest/m3/video-embedded.md) - V4L2 M2M codecs and EGL import, so moq-video works on a Pi
- [Embedded video](/quest/m3/video-embedded.md) - EGL import in the renderer, so moq-video presents on a Pi
- [Decoder drain](/quest/m3/decode-drain.md) - flush and finish for pipelined decoders, so no picture is lost at a track end or crosses a group boundary
- [#2819](/quest/m3/2819-moq-video-carry-pipewire-dma-bufs-safely-into-the-vulkan.md) - moq-video: carry PipeWire DMA-BUFs safely into the Vulkan renderer
- [#2893](/quest/m3/2893-video-validate-pipewire-dma-buf-capture-on-kde-hardware.md) - video: validate PipeWire DMA-BUF capture on KDE hardware
- [Video hardware validation](/quest/m3/video-hardware.md) - run the encode, capture, and zero-copy paths that were written but never run on real machines
Expand Down
33 changes: 33 additions & 0 deletions quest/m3/decode-drain.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# [M] Drain pipelined decoders at group and track boundaries

## Goal

A decoder that holds pictures back (NVDEC, the V4L2 stateful decoder) hands
every one of them over before a track ends, and a transcode's output groups
begin with the picture that began the input group rather than a straggler
from the one before.

## Plan

`decode::Backend` has `decode` and `name` and nothing else: there is no way to
tell a backend the stream is pausing or over. A one-in-one-out decoder
(openh264, VideoToolbox) never needs one, but NVDEC and V4L2 both pipeline, so
the last few pictures of a finite stream stay inside the driver, and at a
group boundary in `moq-transcode` the encoder is flushed while the decoder is
not, so pictures from the closing group surface after the next group's first
access unit went in and are encoded ahead of its keyframe.

Add `flush` and `finish` to `decode::Backend`, mirroring the encode trait and
with the same rule: no default implementation, so a pipelined backend cannot
inherit a no-op and look like it worked. The V4L2 implementation is the
kernel's drain sequence (`V4L2_DEC_CMD_STOP`, dequeue CAPTURE through
`V4L2_BUF_FLAG_LAST`, `V4L2_DEC_CMD_START`), with a source change that lands
mid-drain handled the way `drain_tail` already handles one. NVDEC's is an
end-of-stream packet through the parser. Call `flush` where `moq-transcode`
flushes the encoder and `finish` before a consumer reports the track over, and
cover it with a delayed-backend test that proves no picture is lost or crosses
a group boundary.

## Related

- [Embedded video](/quest/m3/video-embedded.md) - the V4L2 decoder this drains
11 changes: 6 additions & 5 deletions quest/m3/teleop/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,9 @@ operator arbitration, and the latency instrumentation from scratch.

Two things are genuinely missing rather than merely undocumented: the hang
catalog is video plus audio only (location tracks arrived in moq#401 and were
dropped when the catalog became generic), and `moq-video` has no V4L2-M2M
encoder backend, so the boards that fly have no native hardware-encode path.
dropped when the catalog became generic), and `moq-video`'s V4L2-M2M encoder
backend is compiled into no released `moq-cli`, so the boards that fly have no
native hardware-encode path anyone can install.

### Two delivery classes, one session

Expand Down Expand Up @@ -94,9 +95,9 @@ stating plainly because it is what a builder is comparing against.
- [SITL proof and browser ground station](/quest/m3/teleop/proof.md) - ArduPilot
SITL and a synthetic camera flown from a browser ground station, reproducible
in five minutes
- [V4L2-M2M encoding](/quest/m3/teleop/v4l2-encode.md) - `moq-video` encodes in
hardware on the boards that fly, so `moq-cli` needs no GStreamer detour on a
Raspberry Pi
- [V4L2-M2M encoding](/quest/m3/teleop/v4l2-encode.md) - a released `moq-cli`
reaches `moq-video`'s hardware encoder on the boards that fly, and the boards
worth buying are written down
- [Teleoperation use-case docs](/quest/m3/teleop/docs.md) - `doc/concept/use-case/other.md`
becomes the teleoperation page, with a runnable non-media example beside it
- [ROS 2 bridge](/quest/m3/teleop/ros2.md) - a ROS 2 bridge sibling to the
Expand Down
45 changes: 28 additions & 17 deletions quest/m3/teleop/v4l2-encode.md
Original file line number Diff line number Diff line change
@@ -1,30 +1,41 @@
# [L] V4L2-M2M encoding
# [S] V4L2-M2M encoding

## Goal

`moq-video` encodes in hardware on the boards that fly, so `moq-cli` needs no
GStreamer detour on a Raspberry Pi, and the hardware worth buying is written
down.
A released `moq-cli` encodes in hardware on the boards that fly, so a Raspberry
Pi needs no GStreamer detour, and the hardware worth buying is written down.

## Plan

Add a V4L2-M2M stateful encoder backend next to openh264, NVENC, VAAPI,
VideoToolbox and MediaFoundation, pairing with the V4L2 capture path that
already exists. Mainline kernel API, no vendor SDK. It covers Pi 4, CM4 and
Zero 2 W via `bcm2835-codec`, which is what the drone world actually flies.
The backend exists: `moq-video`'s default-on `v4l2` feature drives the stateful
V4L2 M2M encoder and decoder through `rs/moq-video/src/v4l2.rs`, and both have
run on a Pi 4 (`bcm2835-codec`). What is left is getting it into people's
hands.

Rockchip is deliberately excluded: RK3588 encoding goes through rkmpp in a
vendor kernel rather than V4L2, and the existing `moq-gst` route already ships
Released `moq-cli` binaries are built with default features, so `capture` is
not compiled in and the backend reaches nobody who installs a release. A
hardware encoder nobody can install is not a fix; the feature flip is
[CLI packaging](/quest/m2/cli-packaging.md)'s, and this quest finishes once a
released binary on a Pi 4 publishes from `moq import capture` through the
hardware encoder.

Write the hardware note down, in `doc/bin/cli.md` next to the capture build
instructions. Two landmines make it part of the deliverable: Raspberry Pi 5 has
no video encoder at all, and Jetson Orin Nano ships without NVENC. The boards
that still encode are Pi 4/CM4/Zero 2 W, Orin NX and above, and RK3588.
Rockchip stays excluded from the backend: RK3588 encoding goes through rkmpp in
a vendor kernel rather than V4L2, and the existing `moq-gst` route already ships
aarch64 packages for it. Adding an rkmpp backend is a separate decision.

Two landmines make this worth owning, and make the hardware note part of the
deliverable rather than an afterthought: Raspberry Pi 5 has no video encoder at
all, and Jetson Orin Nano ships without NVENC. The boards that still encode are
Pi 4/CM4/Zero 2 W, Orin NX and above, and RK3588.
Validate what the Pi 4 run did not reach: `set_bitrate` on a running encoder
(congestion control retunes through it), resolutions past 640x360 where 1080p
codes as 1088 rows and the compose rectangle crops it back, and the other
`bcm2835-codec` boards (Zero 2 W, CM4).

Released `moq-cli` binaries are built with default features, so `capture` is
not compiled in. A hardware encoder nobody can install is not a fix; settle how
the capture-enabled build ships as part of this quest.
## Required

- [CLI packaging](/quest/m2/cli-packaging.md) - the released binary has to
compile `capture` before a hardware encoder in it reaches anyone

## Related

Expand Down
26 changes: 10 additions & 16 deletions quest/m3/video-embedded.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,26 +2,20 @@

## Goal

`moq-video` encodes and decodes on Raspberry Pi and similar embedded hardware,
and can present there. Neither is possible today: there is no stateful V4L2
codec backend, and the renderer's only zero-copy import is Vulkan.
`moq-video` presents on Raspberry Pi and similar embedded hardware. Encoding
and decoding there is the V4L2 M2M backends' job already; presenting is not
possible today, because the renderer's only zero-copy import is Vulkan and
those devices have no usable Vulkan driver.

## Plan

**V4L2 M2M encode and decode.** The stateful hardware codec path embedded
devices expose, and the one gap with no alternative: openh264 software encode
on a Pi is not real-time. The seam these need already exists, since timestamps
ride on `Frame` through encode, so a queued device draining frame N-k while
frame N is submitted stamps its output correctly rather than with the
submission clock.
**EGL/GLES import** in the renderer. The devices with a V4L2 codec are the ones
without a usable Vulkan driver, so the existing DMA-BUF import cannot reach
them. Same shape as the Vulkan path: alias the buffer, keep the per-path
fallback and three-strike disable, fall back to I420 when import fails.

**EGL/GLES import** in the renderer. The same devices are the ones without a
usable Vulkan driver, so the existing DMA-BUF import cannot reach them. Same
shape as the Vulkan path: alias the buffer, keep the per-path fallback and
three-strike disable, fall back to I420 when import fails.

Hardware-gated end to end. Both halves need a real device to validate, and the
usual dlopen-and-degrade rule applies: a build without the device present must
Hardware-gated end to end. It needs a real device to validate, and the usual
dlopen-and-degrade rule applies: a build without the device present must
degrade rather than fail to start.

Two items from the same wave are deliberately not here. X11 MIT-SHM capture is
Expand Down
8 changes: 7 additions & 1 deletion rs/moq-cli/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ path = "src/main.rs"
doc = false

[features]
default = ["iroh", "quinn", "websocket", "nvidia", "vaapi", "pipewire"]
default = ["iroh", "quinn", "websocket", "nvidia", "vaapi", "v4l2", "pipewire"]
iroh = ["moq-native/iroh"]
jemalloc = ["moq-native/jemalloc"]
noq = ["moq-native/noq"]
Expand Down Expand Up @@ -64,6 +64,12 @@ nvdec = ["nvidia"]
# trimmable the same way: libva is dlopen'd at runtime, so it costs the build
# nothing. Only bites when `capture` or `transcode` pulls in moq-video.
vaapi = ["moq-video?/vaapi", "moq-transcode?/vaapi"]
# The V4L2 M2M codecs of an ARM SoC (a Raspberry Pi's VideoCore and the like),
# on by default like `nvidia` and trimmable the same way. The `v4l` crate it
# needs is the one `capture` already builds for the camera, so it costs nothing
# a capture build does not pay already. Only bites when `capture`, `transcode`,
# or `play` pulls in moq-video.
v4l2 = ["moq-video?/v4l2", "moq-transcode?/v4l2"]
# Screen capture for `capture --display` on Linux (xdg-desktop-portal + PipeWire),
# on by default like the hardware codecs above and trimmable the same way. Only
# bites when `capture` pulls in moq-video; links libpipewire-0.3 at build time.
Expand Down
5 changes: 4 additions & 1 deletion rs/moq-transcode/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ categories = ["multimedia", "multimedia::video", "multimedia::encoding"]
# the whole point of a GPU transcode fleet, and it is the zero-copy GPU pipeline
# (NVDEC decodes and scales in hardware, NVENC encodes the CUDA frame in place).
# Both dlopen their driver at runtime, so disabling them only slims the compile.
default = ["nvidia", "vaapi"]
default = ["nvidia", "vaapi", "v4l2"]
# NVENC + NVDEC (Linux, dlopen'd at runtime; a no-op elsewhere).
nvidia = ["moq-video/nvidia"]
# Compatibility aliases for the pre-consolidation names, kept so a manifest that
Expand All @@ -27,6 +27,9 @@ nvenc = ["nvidia"]
nvdec = ["nvidia"]
# Intel/AMD VAAPI hardware encoder (Linux; libva dlopen'd at runtime).
vaapi = ["moq-video/vaapi"]
# The V4L2 M2M codecs of an ARM SoC (Linux; the `v4l` crate's bindgen needs
# libclang at build time).
v4l2 = ["moq-video/v4l2"]

[dependencies]
bytes = { workspace = true }
Expand Down
31 changes: 25 additions & 6 deletions rs/moq-video/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ categories = ["multimedia", "multimedia::video", "multimedia::encoding"]
# it links libpipewire at build time. Workspace consumers opt out at the root
# manifest (`default-features = false`) and pick per crate, so the relay and the
# language bindings never build the device or render stacks.
default = ["capture", "nvidia", "vaapi", "render"]
default = ["capture", "nvidia", "vaapi", "v4l2", "render"]
# Native camera and screen capture. Enabled by default for direct consumers,
# but workspace consumers opt in so codec-only bindings avoid device build
# dependencies such as V4L2 and libclang.
Expand All @@ -44,6 +44,21 @@ nvdec = ["nvidia"]
# The backend has not been validated on real hardware yet; `Kind::Named`
# ("openh264") sidesteps it if it misbehaves.
vaapi = ["dmabuf", "dep:moq-vaapi"]
# The V4L2 stateful memory-to-memory hardware codecs on Linux: the H.264 encoder
# and decoder most ARM SoCs expose (a Raspberry Pi's VideoCore, and the
# equivalent block on Rockchip / Amlogic / Allwinner / Exynos). One feature
# because they are one dependency and one code path: both drive the same device
# node through `src/v4l2.rs`, so splitting them would save no build. Both have
# run on a Raspberry Pi 4 (bcm2835-codec, Raspberry Pi OS Bookworm): the encoder
# on `/dev/video11` produced Constrained Baseline 640x360 that plays back
# correctly, and the decoder played a 640x360 stream for a minute without a
# failure and is what automatic selection picks on that board. On by default
# because it costs the build nothing `capture` does not already: the `v4l` crate
# and its bindgen are the same ones camera capture pulls in. A host with no M2M
# codec node fails at open, so automatic backend selection falls through to the
# next encoder or decoder; `Kind::Named("openh264")` sidesteps it on a board
# whose driver misbehaves.
v4l2 = ["dep:v4l", "dep:libc"]
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
# Linux DMA-BUF surface vocabulary and CPU fallback support. Enabled by every
# backend that can produce or consume one. It pulls no graphics API by itself.
dmabuf = ["dep:libc", "dep:linux-raw-sys"]
Expand Down Expand Up @@ -116,9 +131,9 @@ ashpd = { version = "0.13", optional = true, default-features = false, features
# extra dependencies); libnvrtc itself is never loaded since we ship pre-built PTX
# (see frame/nv12_resize.ptx) that the driver JIT-compiles.
cudarc = { workspace = true, features = ["nvrtc"], optional = true }
# Mapping a linear DMA-BUF for the universal CPU I420 fallback. Non-linear
# modifiers stay GPU-only and return an honest error instead of treating tiled
# memory as rows.
# Mapping a linear DMA-BUF for the universal CPU I420 fallback, and the V4L2 M2M
# codec node's poll / mmap flags. Non-linear modifiers stay GPU-only and return
# an honest error instead of treating tiled memory as rows.
libc = { workspace = true, optional = true }
# Probe for the NVIDIA driver libraries before calling cudarc / the NVENC SDK,
# which panic (process-abort under release `panic = "abort"`) if their library is
Expand All @@ -136,8 +151,12 @@ moq-vaapi = { workspace = true, optional = true }
# PipeWire video stream for the portal's node, behind the `pipewire` feature.
# Links libpipewire-0.3 via pkg-config at build time; bindgen needs libclang.
pipewire = { version = "0.10", optional = true }
# Native V4L2 webcam capture (replaces nokhwa). `v4l` streams MMAP buffers;
# MJPEG cameras are decoded with the pure-Rust `zune-jpeg` (no libjpeg/IJG).
# Native V4L2 webcam capture (replaces nokhwa) and, behind the `v4l2` feature,
# the M2M hardware codecs. `v4l` streams MMAP buffers; MJPEG cameras are decoded
# with the pure-Rust `zune-jpeg` (no libjpeg/IJG). The codec backends use its raw
# layer (`v4l::v4l_sys` for the videodev2.h structs, `v4l::v4l2::vidioc` for the
# request codes) rather than a second V4L2 crate, so no ioctl struct is
# hand-rolled and the dependency is one the tree already carries.
v4l = { version = "0.14", optional = true }
# X11 display/window enumeration and capture. Pure Rust and used when no
# Wayland session is active, so Linux screen capture still works without a
Expand Down
16 changes: 14 additions & 2 deletions rs/moq-video/src/decode/backend/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@
//!
//! [`open`] picks the best backend for a [`Codec`] and [`Config`], trying
//! hardware candidates (platform-gated: VideoToolbox on macOS, Media Foundation
//! / DXVA on Windows, NVDEC on Linux) before the openh264 software fallback,
//! exactly like the encode side. Only backends that support the requested codec
//! / DXVA on Windows, NVDEC then V4L2 on Linux) before the openh264 software
//! fallback, exactly like the encode side. Only backends that support the requested codec
//! are considered: there is no software H.265 or AV1 decoder, so those tracks
//! have no fallback below the hardware path.

Expand All @@ -33,6 +33,9 @@ mod mediafoundation;
#[cfg(all(target_os = "linux", feature = "nvidia"))]
mod nvdec;

#[cfg(all(target_os = "linux", feature = "v4l2"))]
mod v4l2;

/// The video codec a decoder handles. Derived from the catalog, not chosen by the
/// caller.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
Expand Down Expand Up @@ -99,6 +102,15 @@ const HARDWARE: &[Candidate] = &[
supports: |c| matches!(c, Codec::H264 | Codec::H265 | Codec::Av1),
open: nvdec::Nvdec::open,
},
// Last of the Linux hardware decoders, for the same reason as its encode
// counterpart: the SoC blocks it drives are the only hardware on a board that
// has no NVIDIA GPU.
#[cfg(all(target_os = "linux", feature = "v4l2"))]
Candidate {
name: v4l2::NAME,
supports: |c| matches!(c, Codec::H264),
open: v4l2::V4l2::open,
},
];

const SOFTWARE: Candidate = Candidate {
Expand Down
Loading
Loading