Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions dart/moq_ffi/lib/src/moq.dart
Original file line number Diff line number Diff line change
Expand Up @@ -432,6 +432,7 @@ class MoqAudio {
final int? bitrate;
final bool enabled;
final MoqContainer container;
final int? jitterUs;
MoqAudio({
this.label = null,
this.broadcast = null,
Expand All @@ -443,6 +444,7 @@ class MoqAudio {
this.bitrate,
this.enabled = true,
required this.container,
this.jitterUs = null,
});
}

Expand Down Expand Up @@ -503,6 +505,11 @@ class FfiConverterMoqAudio {
);
final container = container_lifted.value;
new_offset += container_lifted.bytesRead;
final jitterUs_lifted = FfiConverterOptionalUInt64.read(
Uint8List.view(buf.buffer, new_offset),
);
final jitterUs = jitterUs_lifted.value;
new_offset += jitterUs_lifted.bytesRead;
return LiftRetVal(
MoqAudio(
label: label,
Expand All @@ -515,6 +522,7 @@ class FfiConverterMoqAudio {
bitrate: bitrate,
enabled: enabled,
container: container,
jitterUs: jitterUs,
),
new_offset - buf.offsetInBytes,
);
Expand All @@ -532,6 +540,7 @@ class FfiConverterMoqAudio {
FfiConverterOptionalUInt64.allocationSize(value.bitrate) +
FfiConverterBool.allocationSize(value.enabled) +
FfiConverterMoqContainer.allocationSize(value.container) +
FfiConverterOptionalUInt64.allocationSize(value.jitterUs) +
0;
final buf = Uint8List(total_length);
write(value, buf);
Expand Down Expand Up @@ -580,6 +589,10 @@ class FfiConverterMoqAudio {
value.container,
Uint8List.view(buf.buffer, new_offset),
);
new_offset += FfiConverterOptionalUInt64.write(
value.jitterUs,
Uint8List.view(buf.buffer, new_offset),
);
return new_offset - buf.offsetInBytes;
}

Expand All @@ -594,6 +607,7 @@ class FfiConverterMoqAudio {
FfiConverterOptionalUInt64.allocationSize(value.bitrate) +
FfiConverterBool.allocationSize(value.enabled) +
FfiConverterMoqContainer.allocationSize(value.container) +
FfiConverterOptionalUInt64.allocationSize(value.jitterUs) +
0;
}
}
Expand Down
5 changes: 3 additions & 2 deletions doc/lib/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ how it looks in that language:
- **Connect** to a relay with TLS options and a JWT in the URL, or **serve** sessions yourself and accept or reject each request by path.
- **Reconnect** automatically with backoff when the transport drops.
- **Discover** broadcasts by prefix, wait for a specific one, or request one by path. Advertise an exact path (a created broadcast is invisible until announced), or claim a prefix and serve requests beneath it on demand.
- **Publish and subscribe to media** with the hang catalog filled in from the bitstream, or hand over raw pixels and PCM and let the binding encode and decode (VideoToolbox, Media Foundation, NVENC, openh264, Opus). The built-in video encoder can follow the connection's send estimate.
- **Publish and subscribe to media** with the hang catalog filled in from the bitstream, or hand over raw pixels and PCM and let the binding encode and decode (VideoToolbox, Media Foundation, NVENC, openh264, Opus). The built-in video encoder can follow the connection's send estimate. The audio decoder reports the playout delay to buffer ahead of the playhead, fixed or estimated from arrivals, so read each frame as soon as the last is handled rather than pacing reads to the speaker.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Mirror playout-delay guidance into binding docs

For Python, Go, Swift, Kotlin, and generated C++ users, the only new guide text is this shared sentence; none of the language-specific pages explain the fixed-versus-estimated behavior or the requirement to read frames promptly. Users who follow the normal binding guide can therefore miss the new capability and use the estimator incorrectly, so add the relevant guidance to each affected language page.

AGENTS.md reference: AGENTS.md:L102-L104

Useful? React with 👍 / 👎.

- **Raw tracks** of arbitrary bytes, with per-subscriber priority and max delay, and best-effort datagrams.
- **JSON tracks** as a latest value with merge-patch deltas, or as an append log.
- **Fetch** a single cached group by sequence.
Expand All @@ -47,4 +47,5 @@ how it looks in that language:
- **Connection stats** and **errors** that tell an auth rejection (don't retry) from a shutdown or a transport failure.

Dart is the exception on codecs: its published binaries carry no encoder or
decoder, so it moves already-encoded frames.
decoder, so it moves already-encoded frames. C's audio decoder neither takes
nor reports a playout delay.
21 changes: 17 additions & 4 deletions go/wrapper/audio/audio.go
Original file line number Diff line number Diff line change
Expand Up @@ -61,17 +61,24 @@ func (o EncoderOutput) ffi() (ffi.MoqAudioEncoderOutput, error) {
return ffi.MoqAudioEncoderOutput{Codec: o.Codec, Track: o.Track, SampleRate: o.SampleRate, Channels: o.Channels, Bitrate: o.Bitrate, FrameDurationUs: uint32(frame), Preset: o.Preset}, err
}

// DecoderOutput configures the PCM a Consumer delivers and how stale a group may get before it is skipped.
// DecoderOutput configures the PCM a Consumer delivers, how stale a group may get before it is skipped, and the playout delay.
type DecoderOutput struct {
Format SampleFormat
SampleRate *uint32
Channels *uint32
MaxDelay *time.Duration
// MaxDelay bounds how stale a group may get before it is skipped; it adds no buffering.
MaxDelay *time.Duration
// Delay fixes the jitter buffer to hold ahead of the playhead; nil estimates it from arrival timing.
Delay *time.Duration
}

func (o DecoderOutput) ffi() (ffi.MoqAudioDecoderOutput, error) {
delay, err := bridge.OptionalDuration(o.MaxDelay, "audio max delay")
return ffi.MoqAudioDecoderOutput{Format: o.Format, SampleRate: o.SampleRate, Channels: o.Channels, MaxDelayUs: delay}, err
maxDelay, err := bridge.OptionalDuration(o.MaxDelay, "audio max delay")
if err != nil {
return ffi.MoqAudioDecoderOutput{}, err
}
delay, err := bridge.OptionalDuration(o.Delay, "audio delay")
return ffi.MoqAudioDecoderOutput{Format: o.Format, SampleRate: o.SampleRate, Channels: o.Channels, MaxDelayUs: maxDelay, DelayUs: delay}, err
}

// Frame is interleaved PCM with its presentation timestamp.
Expand Down Expand Up @@ -203,5 +210,11 @@ func (d *Consumer) Next(ctx context.Context) (*Frame, error) {
// All ranges over decoded frames until the track ends or the loop breaks.
func (d *Consumer) All(ctx context.Context) iter.Seq2[*Frame, error] { return bridge.Seq(ctx, d.Next) }

// Delay is the playout delay to hold ahead of the playhead: the fixed one, or the estimate from every frame read so far.
// The estimate measures when Next returns, so call it again as soon as the previous frame is handled.
func (d *Consumer) Delay() time.Duration {
return time.Duration(d.inner.DelayUs()) * time.Microsecond
}

// Cancel stops the stream.
func (d *Consumer) Cancel() { d.inner.Cancel() }
80 changes: 80 additions & 0 deletions go/wrapper/moq_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -213,6 +213,86 @@ func TestEncodeAudioWithOpusObject(t *testing.T) {
}
}

// A fixed playout delay is reported back as is, and an estimated one moves as frames are read.
func TestAudioConsumerDelay(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), testTimeout)
defer cancel()

origin := newOrigin(t)
broadcast, err := origin.CreateBroadcast("audio-delay")
if err != nil {
t.Fatal(err)
}
track := "mic"
catalogProducer, err := moqmedia.NewCatalogProducer(broadcast)
if err != nil {
t.Fatal(err)
}
audio, err := moqaudio.NewProducer(catalogProducer,
moqaudio.EncoderInput{Format: moq.AudioSampleFormatF32, SampleRate: 48000, Channels: 1},
moqaudio.EncoderOutput{Codec: moq.OpusAudioCodec(), Track: &track},
nil,
)
if err != nil {
t.Fatal(err)
}
if err := broadcast.Announce(moq.Route{}); err != nil {
t.Fatal(err)
}

bc, err := origin.Consume().RequestBroadcast(ctx, "audio-delay")
if err != nil {
t.Fatal(err)
}
catalog, err := moqmedia.CatalogSnapshot(ctx, bc)
if err != nil {
t.Fatal(err)
}
rendition, ok := catalog.Audio[track]
if !ok {
t.Fatalf("catalog has no %q rendition: %v", track, catalog.Audio)
}

fixedDelay := 80 * time.Millisecond
fixed, err := moqaudio.Subscribe(ctx, bc, track, rendition, moqaudio.DecoderOutput{Format: moq.AudioSampleFormatF32, Delay: &fixedDelay})
if err != nil {
t.Fatal(err)
}
defer fixed.Cancel()
estimated, err := moqaudio.Subscribe(ctx, bc, track, rendition, moqaudio.DecoderOutput{Format: moq.AudioSampleFormatF32})
if err != nil {
t.Fatal(err)
}
defer estimated.Cancel()
if got := fixed.Delay(); got != fixedDelay {
t.Fatalf("fixed delay = %v, want %v", got, fixedDelay)
}
initial := estimated.Delay()

// Two 20 ms Opus frames of silence at 48 kHz mono.
for i := range 2 {
if err := audio.Write(moqaudio.Frame{Timestamp: time.Duration(i) * 20 * time.Millisecond, Data: make([]byte, 960*4)}); err != nil {
t.Fatal(err)
}
}
for _, consumer := range []*moqaudio.Consumer{fixed, estimated} {
if frame, err := consumer.Next(ctx); err != nil || frame == nil {
t.Fatalf("next = %v, %v", frame, err)
}
}

if got := fixed.Delay(); got != fixedDelay {
t.Fatalf("fixed delay after a read = %v, want %v", got, fixedDelay)
}
if got := estimated.Delay(); got == initial {
t.Fatalf("estimated delay stayed at %v after a read", got)
}

if err := audio.Finish(); err != nil {
t.Fatal(err)
}
}

// FrameDurationUs is microseconds so Opus' 2.5 ms frame is expressible at all,
// and a duration outside the Opus set is refused rather than silently rounded.
func TestEncodeAudioFrameDurations(t *testing.T) {
Expand Down
8 changes: 8 additions & 0 deletions kt/moq/src/jvmAndAndroidMain/kotlin/dev/moq/Durations.kt
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,14 @@ val TrackInfo.maxAge: Duration?
val AudioDecoderOutput.maxDelay: Duration?
get() = maxDelayUs?.toLong()?.microseconds

/** Fixed playout delay to hold ahead of the playhead, or null to estimate it from arrival timing. */
val AudioDecoderOutput.delay: Duration?
get() = delayUs?.toLong()?.microseconds

/** Playout delay to hold ahead of the playhead: the fixed one, or the estimate from every frame read so far. */
val dev.moq.audio.Consumer.delay: Duration
get() = delayUs().toLong().microseconds

/** Upper bound on buffering before a stalled group is skipped, or null for the default. */
val VideoDecoderOutput.maxDelay: Duration?
get() = maxDelayUs?.toLong()?.microseconds
Expand Down
2 changes: 1 addition & 1 deletion kt/moq/src/jvmAndAndroidMain/kotlin/dev/moq/audio/Audio.kt
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ typealias Consumer = uniffi.moq.MoqAudioConsumer
typealias EncoderInput = uniffi.moq.MoqAudioEncoderInput
/** The codec side: codec, track name, rate, channels, bitrate, frame duration (0 takes the preset's or codec's own frame), and preset. */
typealias EncoderOutput = uniffi.moq.MoqAudioEncoderOutput
/** The PCM layout a [Consumer] delivers and how stale a group may get. */
/** The PCM layout a [Consumer] delivers, how stale a group may get, and its playout delay. */
typealias DecoderOutput = uniffi.moq.MoqAudioDecoderOutput
/** Interleaved PCM with its presentation timestamp. */
typealias Frame = uniffi.moq.MoqAudioFrame
Expand Down
41 changes: 41 additions & 0 deletions kt/moq/src/jvmAndAndroidTest/kotlin/dev/moq/SmokeTest.kt
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
package dev.moq

import dev.moq.media.*
import dev.moq.audio.Consumer as AudioConsumer
import dev.moq.audio.Producer as AudioProducer
import dev.moq.video.Consumer as VideoConsumer
import dev.moq.video.Producer as VideoProducer
Expand Down Expand Up @@ -585,6 +586,46 @@ class SmokeTest {
}
}

/** A fixed playout delay is reported back as is, and an estimated one moves as frames are read. */
@Test
fun `audio consumer reports its playout delay`() = runTest {
OriginProducer(OriginConfig()).use { origin ->
origin.createBroadcast("audio-delay").use { broadcast ->
val audio = AudioProducer(
CatalogProducer(broadcast),
AudioEncoderInput(format = AudioSampleFormat.F32, sampleRate = 48_000u, channels = 1u),
AudioEncoderOutput(codec = AudioCodec.opus(), track = "mic"),
)
broadcast.announce(Route())

val consumer = origin.consume().requestBroadcast("audio-delay")
val catalog = CatalogConsumer.subscribe(consumer).next()!!
val rendition = catalog.audio["mic"]!!

val fixedOutput = AudioDecoderOutput(format = AudioSampleFormat.F32, delayUs = 80_000uL)
assertEquals(80.milliseconds, fixedOutput.delay)
val fixed = AudioConsumer.subscribe(consumer, "mic", rendition, fixedOutput)
val estimated = AudioConsumer.subscribe(consumer, "mic", rendition, AudioDecoderOutput(format = AudioSampleFormat.F32))
assertEquals(80.milliseconds, fixed.delay)
val initial = estimated.delay

// Two 20 ms Opus frames of silence at 48 kHz mono.
for (i in 0 until 2) {
audio.write(AudioFrame(timestampUs = i.toULong() * 20_000uL, data = ByteArray(960 * 4)))
}
assertNotNull(fixed.next())
assertNotNull(estimated.next())

assertEquals(80.milliseconds, fixed.delay)
assertTrue(estimated.delay != initial, "the estimate should move as frames are read")

fixed.cancel()
estimated.cancel()
audio.finish()
}
}
}

@Test
fun `decoded video frame owns its picture`() = runTest {
OriginProducer(OriginConfig()).use { origin ->
Expand Down
18 changes: 17 additions & 1 deletion py/moq-rs/moq/audio.py
Original file line number Diff line number Diff line change
Expand Up @@ -78,19 +78,26 @@ def _ffi(self) -> MoqAudioEncoderOutput:

@dataclass(frozen=True)
class DecoderOutput:
"""The PCM layout a :class:`Consumer` delivers and how stale a group may get."""
"""The PCM layout a :class:`Consumer` delivers, how stale a group may get, and its playout delay.

``delay`` fixes the jitter buffer to hold ahead of the playhead; ``None``
estimates it from arrival timing. ``max_delay`` only bounds how stale a
group may get before it is skipped, and adds no buffering.
"""

format: SampleFormat
sample_rate: int | None = None
channels: int | None = None
max_delay: timedelta | None = None
delay: timedelta | None = None

def _ffi(self) -> MoqAudioDecoderOutput:
return MoqAudioDecoderOutput(
format=self.format,
sample_rate=self.sample_rate,
channels=self.channels,
max_delay_us=_opt_us(self.max_delay, "max_delay"),
delay_us=_opt_us(self.delay, "delay"),
)


Expand Down Expand Up @@ -171,6 +178,15 @@ async def __anext__(self) -> Frame:
raise StopAsyncIteration
return Frame._from_ffi(value)

@property
def delay(self) -> timedelta:
"""The playout delay to hold ahead of the playhead: the fixed one, or the estimate so far.

The estimate measures when each frame is read, so read the next frame as
soon as the previous one is handled rather than pacing reads to the speaker.
"""
return _from_us(self._inner.delay_us())

def cancel(self) -> None:
"""Cancel the subscription and stop delivering audio frames."""
self._inner.cancel()
Expand Down
39 changes: 39 additions & 0 deletions py/moq-rs/tests/test_local.py
Original file line number Diff line number Diff line change
Expand Up @@ -1275,6 +1275,7 @@ def test_optional_binding_records_use_none_defaults():
assert decoder.sample_rate is None
assert decoder.channels is None
assert decoder.max_delay is None
assert decoder.delay is None


def test_audio_encoder_returns_namespaced_records():
Expand Down Expand Up @@ -1338,6 +1339,44 @@ def test_encode_audio_with_opus_object():
broadcast.close()


async def test_audio_consumer_reports_its_playout_delay():
"""A fixed playout delay is reported back as is; an estimated one moves as frames are read."""
origin = moq.OriginProducer()
broadcast = create_announced(origin, "audio-delay")
audio = moq.audio.Producer(
moq.media.CatalogProducer(broadcast),
moq.AudioEncoderInput(format=moq.AudioSampleFormat.F32, sample_rate=48_000, channels=1),
moq.audio.EncoderOutput(codec=moq.AudioCodec.opus(), track="mic"),
)

consumer = origin.consume()
broadcast_consumer = await asyncio.wait_for(consumer.request_broadcast("audio-delay"), timeout=5.0)
catalog = await asyncio.wait_for(moq.media.catalog(broadcast_consumer), timeout=5.0)
rendition = catalog.audio["mic"]

def output(delay: timedelta | None) -> moq.audio.DecoderOutput:
return moq.audio.DecoderOutput(format=moq.AudioSampleFormat.F32, delay=delay)

fixed = await moq.audio.Consumer.subscribe(broadcast_consumer, "mic", rendition, output(timedelta(milliseconds=80)))
estimated = await moq.audio.Consumer.subscribe(broadcast_consumer, "mic", rendition, output(None))
assert fixed.delay == timedelta(milliseconds=80)
initial = estimated.delay

# Two 20 ms Opus frames of silence at 48 kHz mono.
for i in range(2):
audio.write(moq.audio.Frame(timestamp=timedelta(milliseconds=20 * i), data=bytes(960 * 4)))
await asyncio.wait_for(anext(fixed), timeout=5.0)
await asyncio.wait_for(anext(estimated), timeout=5.0)

assert fixed.delay == timedelta(milliseconds=80)
assert estimated.delay != initial

fixed.cancel()
estimated.cancel()
audio.finish()
broadcast.close()


async def test_decode_video_frame():
"""A decoded frame owns its picture and converts to either layout, even after its consumer is cancelled."""
origin = moq.OriginProducer()
Expand Down
1 change: 0 additions & 1 deletion quest/m2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,6 @@ ranked just below deterministic TS legs, which it requires.

- [Time test sweep](/quest/m2/time-test-sweep.md) - unrelated test conversion and the workspace lint ratchet

- [Binding audio delay](/quest/m2/binding-surface.md) - moq-ffi and every wrapper configure and observe audio playout delay
- [Pipelined requests](/quest/m2/pipeline-requests/README.md) - SUBSCRIBE and the first FETCH go out with the track-info request at every hop, so first data arrives a round trip sooner per hop
- [OBS native codecs](/quest/m2/obs-moq-video/README.md) - replace FFmpeg video and audio decoding with moq-video and moq-audio, deliver GPU frames, and use native audio/video encoders
- [OBS multitrack](/quest/m2/obs-multitrack.md) - the obs-moq output publishes OBS's native multitrack video encoders as simulcast renditions, configured like OBS multitrack and over obs-websocket
Expand Down
Loading
Loading