Repository navigation
Conversation
Add a "low_latency" encoder setting, not exposed in the UI, that services can enable through apply_encoder_settings for real-time protocols. On hardware H.264/HEVC encoders it creates the session with EnableLowLatencyRateControl, which selects the low latency (.rtvc) encoder, and sets RealTime. CBR falls back to ABR and Spatial AQ is skipped, as neither is supported in this mode. Other encoders ignore the setting and encode normally.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a
low_latencysetting to the VideoToolbox H.264/HEVC hardware encoders. Not a user-facing feature, the idea is that a service turns it on throughapply_encoder_settingswhen it streams over a real-time protocol.With it set, the session is created with
kVTVideoEncoderSpecification_EnableLowLatencyRateControl. VT only allows that on the dedicated.rtvcencoders (com.apple.videotoolbox.videoencoder.h264.rtvc/hevc.rtvc), so the encoder asks VT for that one and uses it instead of the selected one.RealTimeis turned on too.A few things work differently in this mode. CBR isn't supported (
ConstantBitRatefails withkVTPropertyNotSupportedErr), so it falls back to ABR. Spatial AQ isn't supported either, so it's skipped. And the encoder won't produce B-frames even ifbframesis on.Software and ProRes encoders just ignore the setting. If VT has no low latency encoder for the codec/resolution, it logs that and encodes normally. The mode is picked when the session is created, so later updates (dynamic bitrate, for example) don't change it.
Motivation and Context
We use this from a service that needs low encoder delay. WHIP could potentially use it too.
I measured encode latency outside OBS with the same session properties (1080p30 camera footage, ABR 6 Mbps, M4 chip):
How Has This Been Tested?
MacBook Pro M4, macOS 26.6.2, streaming over RTMP to a local ffmpeg:
low_latencyon in the stream encoder settings: usesh264.rtvc, CBR falls back to ABR, and changing the bitrate while live works.low_latency=trueplus a bitrate change while live: it stays a normal session and keeps CBR.low_latency: logs the fallback and encodes normally.Types of changes
Checklist:
This change is