Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion quest/m1/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,6 @@ moves what remains of them once their claims merge.
- [Push-to-talk's press-to-audible latency fits a budget on the defaults](/quest/m1/push-to-talk-latency.md) - press to audible on the browser defaults, excluding device output, stays under a kixtop-measured budget that `just test push-to-talk` enforces
- [Lifecycles are explicit state](/quest/m1/explicit-state/README.md) - every lifecycle the 2026-11-07 audit found implicit is one enum or union, fixing the bugs in the 11 lifecycles it redesigns, with review and agent rules against new ones
- [Track drop runs are forgotten below the floor](/quest/m1/drop-runs-prune.md) - a track's dropped-group record stays proportional to its window in Rust and JS
- [A macOS camera delivers frames the renderer imports](/quest/m1/camera-nv12-macos.md) - AVFoundation capture asks for NV12, so a camera whose native format is `2vuy` reaches the renderer

## Related

Expand Down
30 changes: 0 additions & 30 deletions quest/m1/camera-nv12-macos.md

This file was deleted.

2 changes: 2 additions & 0 deletions rs/justfile
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,8 @@ macos-test:
run just rs test -p kio -p hang -p moq-tokio -p moq-sock
run just rs test -p moq-sock --features fast-apple-datapath
run just rs test -p moq-video --no-default-features -E 'test(encode::backend::videotoolbox::tests)'
# Runner has no camera; these tests need none.
run just rs test -p moq-video --no-default-features --features capture -E 'test(/^capture::avfoundation::tests::/)'
run just rs test -p moq-video --no-default-features --features render -E 'test(/^render::export::/)'
# The runner's paravirtual Metal device proves the IOSurface import, ignored elsewhere.
run just rs test -p moq-video --no-default-features --features render --lib --run-ignored only \
Expand Down
35 changes: 31 additions & 4 deletions rs/moq-video/src/capture/avfoundation.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,11 @@
//!
//! `AVCaptureVideoDataOutput` delivers IOSurface-backed `CVPixelBuffer`s on a
//! dispatch queue; the delegate wraps each as a [`Surface::PixelBuffer`] and pushes it
//! into the shared [`FrameChannel`], which the encode loop awaits. Frames reach
//! VideoToolbox with no copy and no color conversion.
//! into the shared [`FrameChannel`], which the encode loop awaits. The output
//! requests NV12, the one layout the encoder, the Metal import, and the CPU
//! download all read, so a camera whose native format differs (a MacBook Air's
//! `2vuy`) is converted by AVFoundation before the frame reaches us. Frames then
//! reach VideoToolbox with no copy.
//!
//! Sample delivery is the only callback AVFoundation makes on the happy path, so
//! the same delegate also observes the notifications that say the session died:
Expand All @@ -27,7 +30,10 @@ use objc2_av_foundation::{
AVError, AVFoundationErrorDomain, AVMediaType, AVMediaTypeVideo,
};
use objc2_core_media::CMSampleBuffer;
use objc2_foundation::{NSError, NSNotification, NSNotificationCenter, NSObject, NSObjectProtocol, NSString};
use objc2_core_video::{kCVPixelBufferPixelFormatTypeKey, kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange};
use objc2_foundation::{
NSDictionary, NSError, NSNotification, NSNotificationCenter, NSNumber, NSObject, NSObjectProtocol, NSString,
};

use super::surface::surface_frame;
use super::{Camera, Config, FrameChannel, Stream};
Expand Down Expand Up @@ -95,6 +101,7 @@ pub(super) async fn open(config: &Config, device: Option<&str>) -> Result<Stream

let output = unsafe { AVCaptureVideoDataOutput::new() };
unsafe {
output.setVideoSettings(Some(&nv12_settings()));
// Drop late frames instead of queuing them; we want the newest.
output.setAlwaysDiscardsLateVideoFrames(true);
let proto = ProtocolObject::from_ref(&*delegate);
Expand Down Expand Up @@ -151,6 +158,15 @@ pub(super) async fn open(config: &Config, device: Option<&str>) -> Result<Stream
))
}

/// The `videoSettings` that make the output deliver NV12. Without a pixel
/// format AVFoundation passes the device's native one through.
fn nv12_settings() -> Retained<NSDictionary<NSString, AnyObject>> {
let key: &NSString = unsafe { kCVPixelBufferPixelFormatTypeKey }.as_ref();
let format = NSNumber::new_u32(kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange);
let format: &AnyObject = &format;
NSDictionary::from_slices(&[key], &[format])
}

/// Ensure the process is authorized to use the camera, prompting once if the
/// decision hasn't been made yet.
///
Expand Down Expand Up @@ -436,7 +452,7 @@ impl Delegate {
/// grant, or a test-only hook in the capture path.
#[cfg(test)]
mod tests {
use objc2_foundation::{NSDictionary, NSInteger};
use objc2_foundation::NSInteger;

use super::*;

Expand All @@ -451,6 +467,17 @@ mod tests {
(chan, session, guard)
}

/// CI has no camera, and `videoSettings` reads back nil until a device is
/// connected, so check the settings `open` hands the output.
#[test]
fn the_output_requests_nv12() {
let settings = nv12_settings();
let key: &NSString = unsafe { kCVPixelBufferPixelFormatTypeKey }.as_ref();
let format = settings.objectForKey(key).expect("pixel format requested");
let format = format.downcast::<NSNumber>().expect("pixel format is a number");
assert_eq!(format.as_u32(), kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange);
}

/// Teardown must never call `stopRunning` while configuration is open. This
/// is the setup-failure path when `canAddInput` or `canAddOutput` returns false.
#[test]
Expand Down
Loading