Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions doc/lib/go/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,12 @@ one: `FetchGroup`/`FetchMediaGroup`, `Dynamic()` with `Requests(ctx)`,
`AppendDatagram`/`Datagrams(ctx)`, `SetCatalogSection`, `Used`/`Unused`,
`Session().Stats()`. `moq.IsAuthError` and `moq.IsShutdown` classify errors. `moq.ProtocolError(err)` is the structured protocol failure (scope, verbatim code, kind) when the peer sent one.

`DecodeVideo` picks the decoded CPU pixel layout: `VideoDecoderOutput.Format`
is I420 when nil, or `VideoPixelFormatRgba` for four bytes a pixel, and every
`VideoDecodedFrame` repeats the layout it was decoded to. `Resize` is best
effort: only NVDEC has a built-in scaler, so read each frame's own `Width` and
`Height` rather than assuming it took.

- API reference: [pkg.go.dev/github.com/moq-dev/moq-go](https://pkg.go.dev/github.com/moq-dev/moq-go)
- Source: [`go/`](https://github.com/moq-dev/moq/tree/main/go); `just go check` builds and tests locally
- Mirrors the vanity path resolves to: [moq-dev/moq-go](https://github.com/moq-dev/moq-go) (wrapper), [moq-dev/moq-go-ffi](https://github.com/moq-dev/moq-go-ffi) (raw bindings and static libraries)
6 changes: 6 additions & 0 deletions doc/lib/kt/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,12 @@ connection's send estimate; pass it to `encodeVideo` / `encodeAudio` or
(scope, verbatim code, kind) when the peer sent one. Cancelling the collecting coroutine cancels the
native side.

`decodeVideo` picks the decoded CPU pixel layout: `VideoDecoderOutput.format`
is `VideoPixelFormat.I420` when null, or `VideoPixelFormat.RGBA` for four bytes
a pixel, and every frame repeats the layout it was decoded to. `resize` is best
effort: only NVDEC has a built-in scaler, and MediaCodec is not it, so read each
frame's own `width` and `height` rather than assuming it took.

- API reference: [javadoc.io/doc/dev.moq/moq](https://javadoc.io/doc/dev.moq/moq)
- Source: [`kt/`](https://github.com/moq-dev/moq/tree/main/kt); `just kt check` builds and tests locally
- Artifacts: [dev.moq:moq](https://central.sonatype.com/artifact/dev.moq/moq), [dev.moq:moq-ffi](https://central.sonatype.com/artifact/dev.moq/moq-ffi)
6 changes: 6 additions & 0 deletions doc/lib/py/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,12 @@ app-owned track. `moq.is_auth(err)` and `moq.is_shutdown(err)` classify errors.
Each server request reports a `moq.Transport` enum, including QUIC, Iroh,
WebSocket, TCP, and Unix sockets.

`decode_video` picks the decoded CPU pixel layout: `VideoDecoderOutput.format`
is `VideoPixelFormat.I420` when unset, or `VideoPixelFormat.RGBA` for four
bytes a pixel, and every frame repeats the layout it was decoded to. `resize`
is best effort: only NVDEC has a built-in scaler, so read each frame's own
`width` and `height` rather than assuming it took.

- API reference: [moq-rs.readthedocs.io](https://moq-rs.readthedocs.io)
- Source and examples: [`py/moq-rs`](https://github.com/moq-dev/moq/tree/main/py/moq-rs)
- Raw bindings: [`moq-ffi`](https://pypi.org/project/moq-ffi/) on PyPI, for the unwrapped API
6 changes: 6 additions & 0 deletions doc/lib/swift/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,12 @@ divides the connection's send estimate; pass it to `encodeVideo` /
`isShutdown` classify errors. `protocolError` is the structured protocol failure
(scope, verbatim code, kind) when the peer sent one.

`decodeVideo` picks the decoded CPU pixel layout: `VideoDecoderOutput.format`
is `.i420` when unset, or `.rgba` for four bytes a pixel, and every frame
repeats the layout it was decoded to. `resize` is best effort: only NVDEC has a
built-in scaler, and VideoToolbox is not it, so read each frame's own `width`
and `height` rather than assuming it took.

- API reference: [Swift Package Index (DocC)](https://swiftpackageindex.com/moq-dev/moq-swift/documentation/moq)
- Source: [`swift/`](https://github.com/moq-dev/moq/tree/main/swift); `just swift check` builds and tests on a Mac
- Packages SPM resolves: [moq-dev/moq-swift](https://github.com/moq-dev/moq-swift), [moq-dev/moq-swift-ffi](https://github.com/moq-dev/moq-swift-ffi)
115 changes: 115 additions & 0 deletions go/wrapper/moq_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -240,6 +240,121 @@ func TestVideoPropertiesUseDefaultedFields(t *testing.T) {
}
}

// TestDecodeVideoFormat pins the decode side picking its CPU layout: an unset
// Format is I420, and RGBA is four bytes a pixel, with each frame naming the
// layout it was decoded to.
func TestDecodeVideoFormat(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), testTimeout)
defer cancel()

origin := moq.NewOriginProducer()
broadcast, err := origin.CreateBroadcast("video-decode-format")
if err != nil {
t.Fatal(err)
}
track := "camera"
video, err := broadcast.EncodeVideo(
moq.VideoEncoderInput{Format: moq.VideoPixelFormatRgba, Width: 320, Height: 240, Framerate: 30},
// Software both ways so the test is deterministic everywhere.
moq.VideoEncoderOutput{Codec: moq.VideoCodecH264, Track: &track, Kind: moq.SoftwareEncoder()},
nil,
)
if err != nil {
t.Fatal(err)
}
if err := broadcast.Announce(moq.Route{}); err != nil {
t.Fatal(err)
}

// Seed the track so a subscriber joining below lands on encoded media.
rgba := make([]byte, 320*240*4)
for i := range rgba {
rgba[i] = 0x80
}
if err := video.Cut(); err != nil {
t.Fatal(err)
}
for i := range 10 {
if err := video.Write(moq.VideoFrame{TimestampUs: uint64(i) * 33333, Data: rgba}); err != nil {
t.Fatal(err)
}
}

bc, err := origin.Consume().RequestBroadcast(ctx, "video-decode-format")
if err != nil {
t.Fatal(err)
}
catalog, err := bc.Catalog(ctx)
if err != nil {
t.Fatal(err)
}
rendition, ok := catalog.Video[track]
if !ok {
t.Fatalf("catalog has no %q rendition: %v", track, catalog.Video)
}

// Two subscribers over one publication, so the same encoded frames are read
// twice and only the requested layout differs.
i420, err := bc.DecodeVideo(ctx, track, rendition, moq.VideoDecoderOutput{})
if err != nil {
t.Fatal(err)
}
defer i420.Cancel()
format := moq.VideoPixelFormatRgba
packed, err := bc.DecodeVideo(ctx, track, rendition, moq.VideoDecoderOutput{Format: &format})
if err != nil {
t.Fatal(err)
}
defer packed.Cancel()

// Keep the encoder fed so both decoders see frames after they joined.
for i := 10; i < 40; i++ {
if err := video.Write(moq.VideoFrame{TimestampUs: uint64(i) * 33333, Data: rgba}); err != nil {
t.Fatal(err)
}
}

frame, err := i420.Next(ctx)
if err != nil {
t.Fatal(err)
}
if frame == nil {
t.Fatal("expected an I420 frame")
}
if frame.Format != moq.VideoPixelFormatI420 {
t.Fatalf("format = %v, want I420", frame.Format)
}
if want := int(frame.Width) * int(frame.Height) * 3 / 2; len(frame.Data) != want {
t.Fatalf("I420 length = %d, want %d", len(frame.Data), want)
}

frame, err = packed.Next(ctx)
if err != nil {
t.Fatal(err)
}
if frame == nil {
t.Fatal("expected an RGBA frame")
}
if frame.Format != moq.VideoPixelFormatRgba {
t.Fatalf("format = %v, want RGBA", frame.Format)
}
if want := int(frame.Width) * int(frame.Height) * 4; len(frame.Data) != want {
t.Fatalf("RGBA length = %d, want %d", len(frame.Data), want)
}
for i := 3; i < len(frame.Data); i += 4 {
if frame.Data[i] != 0xFF {
t.Fatalf("RGBA alpha at %d = %#x, want 0xff", i, frame.Data[i])
}
}

if err := video.Finish(); err != nil {
t.Fatal(err)
}
if err := broadcast.Finish(); err != nil {
t.Fatal(err)
}
}

func TestFetchGroupAndServeDynamicMiss(t *testing.T) {
ctx, cancel := context.WithTimeout(context.Background(), testTimeout)
defer cancel()
Expand Down
7 changes: 4 additions & 3 deletions go/wrapper/subscribe.go
Original file line number Diff line number Diff line change
Expand Up @@ -116,9 +116,10 @@ func (b *BroadcastConsumer) DecodeAudio(
return &AudioConsumer{inner: inner}, nil
}

// DecodeVideo subscribes to a video track and decodes it inside the bindings,
// yielding packed I420. catalogVideo comes from the catalog. output.Resize is
// best effort, so read each frame's own dimensions.
// DecodeVideo subscribes to a video track and decodes it inside the bindings.
// catalogVideo comes from the catalog. output.Format picks the packed CPU layout
// every frame arrives in, defaulting to I420 when nil; each frame repeats it.
// output.Resize is best effort, so read each frame's own dimensions.
func (b *BroadcastConsumer) DecodeVideo(
ctx context.Context,
name string,
Expand Down
11 changes: 6 additions & 5 deletions go/wrapper/types.go
Original file line number Diff line number Diff line change
Expand Up @@ -52,11 +52,11 @@ type (
// Video describes one catalog rendition, including whether the publisher recommends temporarily avoiding it.
Video = ffi.MoqVideo
// VideoHint supplies catalog fields a video stream can't reveal itself, such as bitrate, filling only the gaps.
// VideoDecodedFrame is one decoded video frame: packed I420, its dimensions, and a timestamp in microseconds.
VideoHint = ffi.MoqVideoHint
// VideoDecodedFrame is one decoded video frame: packed pixels, the layout they are in, their dimensions, and a timestamp in microseconds.
VideoDecodedFrame = ffi.MoqVideoDecodedFrame
// VideoDecoderOutput configures what DecodeVideo delivers: an optional resize plus a max age.
// VideoDecoderOutput configures what DecodeVideo delivers: an optional pixel format and resize, plus a max age.
VideoDecoderOutput = ffi.MoqVideoDecoderOutput
VideoHint = ffi.MoqVideoHint
// AudioFormat is a single audio codec an importer can parse.
AudioFormat = ffi.MoqAudioFormat
// VideoFormat is a single video codec an importer can parse.
Expand All @@ -67,7 +67,7 @@ type (
VideoProperties = ffi.MoqVideoProperties
// VideoCodec identifies a published video track's codec: H.264 or H.265.
VideoCodec = ffi.MoqVideoCodec
// VideoPixelFormat is a raw pixel layout (I420 or RGBA) written to a VideoProducer.
// VideoPixelFormat is a CPU pixel layout (I420 or RGBA): written to a VideoProducer, or delivered by DecodeVideo.
VideoPixelFormat = ffi.MoqVideoPixelFormat
// VideoEncoderInput declares the pixel layout, resolution, and framerate of frames written to a video producer.
VideoEncoderInput = ffi.MoqVideoEncoderInput
Expand Down Expand Up @@ -184,7 +184,8 @@ func OpusAudioCodec() *AudioCodec {
return ffi.MoqAudioCodecOpus()
}

// VideoPixelFormat values: the raw pixel layout fed to the in-process encoder.
// VideoPixelFormat values: the raw pixel layout fed to the in-process encoder,
// and the one the in-process decoder delivers.
const (
// VideoPixelFormatI420 is tightly-packed planar I420: Y, then U, then V.
VideoPixelFormatI420 = ffi.MoqVideoPixelFormatI420
Expand Down
8 changes: 4 additions & 4 deletions kt/moq/src/jvmAndAndroidMain/kotlin/dev/moq/Aliases.kt
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ typealias MediaGroupConsumer = uniffi.moq.MoqMediaGroupConsumer
typealias AudioProducer = uniffi.moq.MoqAudioProducer
/** The read side of a raw-audio track: yields decoded PCM frames. */
typealias AudioConsumer = uniffi.moq.MoqAudioConsumer
/** The read side of a video track decoded inside the bindings: yields packed I420 frames. */
/** The read side of a video track decoded inside the bindings: yields packed frames in the layout asked for. */
typealias VideoConsumer = uniffi.moq.MoqVideoConsumer
/** The write side of a raw-video track; pixels written here are encoded inside the FFI boundary. */
typealias VideoProducer = uniffi.moq.MoqVideoProducer
Expand Down Expand Up @@ -138,9 +138,9 @@ typealias AudioCodec = uniffi.moq.MoqAudioCodec
typealias AudioSampleFormat = uniffi.moq.MoqAudioSampleFormat
/** The PCM layout an [AudioConsumer] should decode to. */
typealias AudioDecoderOutput = uniffi.moq.MoqAudioDecoderOutput
/** What a [VideoConsumer] decodes to: an optional resize plus a latency budget. */
/** What a [VideoConsumer] decodes to: an optional pixel format and resize, plus a latency budget. */
typealias VideoDecoderOutput = uniffi.moq.MoqVideoDecoderOutput
/** One decoded video frame: packed I420, its dimensions, and a timestamp. */
/** One decoded video frame: packed pixels, the layout they are in, its dimensions, and a timestamp. */
typealias VideoDecodedFrame = uniffi.moq.MoqVideoDecodedFrame
/** The PCM layout the caller feeds an [AudioProducer]. */
typealias AudioEncoderInput = uniffi.moq.MoqAudioEncoderInput
Expand All @@ -150,7 +150,7 @@ typealias AudioEncoderOutput = uniffi.moq.MoqAudioEncoderOutput
typealias VideoFrame = uniffi.moq.MoqVideoFrame
/** A video codec identifier (H.264 or H.265). */
typealias VideoCodec = uniffi.moq.MoqVideoCodec
/** A raw pixel layout (I420 or RGBA) fed to a [VideoProducer]. */
/** A CPU pixel layout (I420 or RGBA): fed to a [VideoProducer], or delivered by `decodeVideo`. */
typealias VideoPixelFormat = uniffi.moq.MoqVideoPixelFormat
/** The pixel layout, resolution, and framerate the caller feeds a [VideoProducer]. */
typealias VideoEncoderInput = uniffi.moq.MoqVideoEncoderInput
Expand Down
53 changes: 53 additions & 0 deletions kt/moq/src/jvmAndAndroidTest/kotlin/dev/moq/SmokeTest.kt
Original file line number Diff line number Diff line change
Expand Up @@ -357,6 +357,59 @@ class SmokeTest {
}
}

@Test
fun `decode video picks its pixel format`() = runTest {
OriginProducer(OriginConfig()).use { origin ->
origin.createBroadcast("video-decode-format").use { broadcast ->
val video = broadcast.encodeVideo(
VideoEncoderInput(format = VideoPixelFormat.RGBA, width = 320u, height = 240u, framerate = 30u),
// Software both ways so the test is deterministic everywhere.
VideoEncoderOutput(codec = VideoCodec.H264, track = "camera", kind = softwareEncoder),
null,
)
broadcast.announce(Route())

// Seed the track so a subscriber joining below lands on encoded media.
val rgba = ByteArray(320 * 240 * 4) { 0x80.toByte() }
video.cut()
for (i in 0 until 10) {
video.write(VideoFrame(timestampUs = i.toULong() * 33_333uL, data = rgba))
}

val consumer = origin.consume().requestBroadcast("video-decode-format")
val catalog = consumer.subscribeCatalog().next()!!
val rendition = catalog.video["camera"]!!

// Two subscribers over one publication, so the same encoded frames
// are read twice and only the requested layout differs.
val i420 = consumer.decodeVideo("camera", rendition, VideoDecoderOutput())
val packed = consumer.decodeVideo(
"camera",
rendition,
VideoDecoderOutput(format = VideoPixelFormat.RGBA),
)

// Keep the encoder fed so both decoders see frames after they joined.
for (i in 10 until 40) {
video.write(VideoFrame(timestampUs = i.toULong() * 33_333uL, data = rgba))
}

val planar = i420.next()!!
assertEquals(VideoPixelFormat.I420, planar.format)
assertEquals(planar.width.toInt() * planar.height.toInt() * 3 / 2, planar.data.size)

val frame = packed.next()!!
assertEquals(VideoPixelFormat.RGBA, frame.format)
assertEquals(frame.width.toInt() * frame.height.toInt() * 4, frame.data.size)
assertTrue(frame.data.indices.filter { it % 4 == 3 }.all { frame.data[it] == 0xFF.toByte() })

i420.cancel()
packed.cancel()
video.finish()
}
}
}

@Test
fun `encode audio with opus object`() {
val input = AudioEncoderInput(format = AudioSampleFormat.F32, sampleRate = 48_000u, channels = 1u)
Expand Down
15 changes: 9 additions & 6 deletions py/moq-rs/moq/subscribe.py
Original file line number Diff line number Diff line change
Expand Up @@ -271,9 +271,9 @@ class VideoConsumer:
"""Async iterator of decoded video frames.

Built via :meth:`BroadcastConsumer.decode_video`. Each frame is
tightly-packed I420 and carries its own ``width`` and ``height``:
``output.resize`` is best effort, so read the frame rather than
assuming it took.
tightly packed in its own ``format`` and carries its own ``width``
and ``height``: ``output.resize`` is best effort, so read the frame
rather than assuming it took.
"""

def __init__(self, inner: MoqVideoConsumer) -> None:
Expand Down Expand Up @@ -513,9 +513,12 @@ async def decode_video(

``catalog_video`` comes from the catalog (e.g.
``await broadcast.catalog()`` followed by ``catalog.video[name]``).
Frames arrive as tightly-packed I420. An unrecognized codec raises
here; a recognized one no native backend handles raises when the
decoder opens, both before the first frame.
An unrecognized codec raises here; a recognized one no native backend
handles raises when the decoder opens, both before the first frame.

``output.format`` picks the packed CPU layout frames arrive in and
defaults to :attr:`VideoPixelFormat.I420`, which is what a decoder
produces natively; each frame repeats the layout it was decoded to.

``output.resize`` asks the decoder for a different size and is best
effort, so read each frame's own dimensions.
Expand Down
Loading
Loading