[Bug]: MediaCodecEncoder holds MediaCodec output buffer until fully written downstream, stalling the encoder under network backpressure

[filippoadessi]

Version

3.2.0

Environment that reproduces the issue

Samsung Galaxy S10 (SM-G973F) - Android 12

RTMP/SRT/... Server

SRT to MediaMTX v1.21.0 (self-hosted) over a real mobile/variable-bandwidth link, re-transported as RTSP and read via SRT

Audio configuration

No audio (video-only stream)

Video configuration

• HEVC (H.265)
• profile: Main
• resolution: 1920x1080 (adaptive down to 1280x720 based on measured bitrate)
• fps: 30
• bitrate: 4 Mbps target
• gopDurationInS: 1.0

Is it reproducible in the demos application?

Yes

Reproduction steps

1. Stream over a link with limited/variable bandwidth (e.g. a real mobile network, or an artificially throttled link below the configured bitrate) using StreamPack's SRT (or RTMP) endpoint.
2. On the server side, have it re-transport the incoming stream as RTSP (or SRT read-pull) to a third client, while also serving it via a segmented protocol (e.g. HLS) for comparison.
3. Capture the raw encoded frames being written to the SrtSink (or equivalent endpoint sink) — e.g. by hooking IEndpointInternal.write()/Sink.write() and dumping the packet bytes with timestamps — and look at the time gaps between consecutive video access units.

Expected result

The encoder should keep producing output frames at a steady rate (matching the configured fps) regardless of transient network slowness, decoupling encode throughput from network throughput — as RootEncoder does.

Actual result

Under real network backpressure, the H.265 elementary stream captured at the sink shows bursts of exactly ~4 frames (matching MediaCodec's small output buffer pool, typically ~4 buffers) followed by multi-second gaps of complete silence, then a fresh keyframe. Camera/surface input keeps feeding continuously and unaffected during these gaps (confirmed via app-side capture logs) — only the encoder's output stalls.

Root cause: FrameFactory.createFrame() in MediaCodecEncoder.kt wraps codec.getOutputBuffer(index) by reference (zero-copy) and only calls codec.releaseOutputBuffer(index, false) inside the Frame's onClosed callback, which fires only after the frame is fully consumed by the entire downstream pipeline (channel -> muxer -> endpoint -> network write/send). If that final network write blocks for any real amount of time (e.g. SrtSink.write() -> socket.send() under congestion), MediaCodec exhausts its small output buffer pool and stops calling onOutputBufferAvailable entirely until the network call returns.

Impact: any output expecting a continuous stream (RTSP, SRT read/pull) breaks or times out during these stalls (mediamtx's RTSP/SRT-read sessions on our end opened and closed within ~4s, delivering zero bytes), while segmented outputs (HLS) mask the problem by just starting a new segment/muxer session.

Additional context

Isolated by comparing against RootEncoder (pedroSG94/RootEncoder) under the exact same network conditions, which does not exhibit this stall — it copies encoder output into a queue immediately (BaseSender's queue.take() producer/consumer pattern), decoupling encode speed from network speed.

We patched this locally in FrameFactory.createFrame() by copying the buffer content into a new ByteBuffer immediately and releasing the MediaCodec output buffer right away, instead of deferring release to Frame.close():

private fun createFrame(
codec: MediaCodec, index: Int, ptsInUs: Long, isKeyFrame: Boolean, tag: String
): Frame {
val buffer = requireNotNull(codec.getOutputBuffer(index))
val extra = if (isKeyFrame || !isVideo) extra!! else null
val sourceBuffer = if (extraBuffers != null) buffer.removePrefixes(extraBuffers) else buffer

val rawBuffer = ByteBuffer.allocate(sourceBuffer.remaining())
rawBuffer.put(sourceBuffer)
rawBuffer.rewind()
try {
    codec.releaseOutputBuffer(index, false)
} catch (t: Throwable) {
    Logger.w(tag, "Failed to release output buffer for code: ${t.message}")
}

return pool.get(rawBuffer, ptsInUs, null, isKeyFrame, extra, outputFormat)

}

This adds one buffer copy per frame but fixes the stall reliably in our testing (verified with two consecutive 20s RTSP captures showing continuous ~30fps delivery with no gaps, vs. 8 frames total with multi-second gaps before the fix). Note this issue is independent of the PCR overflow bug we reported separately (#301) — we hit both in the same investigation but they have distinct root causes and this one is the more impactful of the two.

Relevant logs output

[ThibaultBee]

The RTMP path doesn't have this issue because it uses a buffered channel, which decouples encoding from network writes.

I agree the proposed fix isn't ideal: allocating and copying every frame would add significant GC pressure and defeat the goal of avoiding memcpy.

The TS muxer is also quite old. I'd like to rewrite it, but unfortunately I haven't found the time yet. One possible fix there would be to use the SRT socket in non-blocking mode, so network backpressure doesn't block the whole pipeline.

[filippoadessi]

That's a fair point, thanks for looking closely. You're right that per-frame allocation isn't a great trade-off given the memcpy-avoidance goal, and I found the actual gap: SrtSink.write() calls socket.send() directly with no buffering, unlike RtmpEndpoint's ChannelWithCloseableData<FLVTag> (capacity 10, DROP_OLDEST) which already solves exactly this for RTMP. Rather than the encoder-level copy in my original patch, I'd like to rework this as the same bounded-channel pattern applied to SrtSink (buffering Packets, consumed by a coroutine that calls socket.send()), so it's endpoint-scoped like RTMP already is and doesn't touch MediaCodecEncoder at all. Does that direction work for you before I redo the PR? Non-blocking socket mode is also an option if you'd rather avoid the extra buffer/drop-policy entirely — happy to go whichever way you prefer.

[ThibaultBee]

I am working on Non blocking SRT socket.

But I should also changed the Channel size (currently UNLIMITED) in the EncodingPipelineOutput to a specific value like 4. I am looking where the 4 actually comes from but I can't find any source or an API about this value. Is it the same value for audio or video? Or for an HEVC/VC/AV1/...? Not looking for AI answer but real source from the doc.

[filippoadessi]

Hi Thibault,

That's a fair point — and to be honest, the ambiguity is on me: I dropped the number "4" into the issue without explaining where it came from. To clarify, it wasn't from any StreamPack source or documented constant: it was an empirical observation from my field testing. I've been working for several days on a single phone (Samsung Galaxy S10) stress-testing some specific StreamPack features pretty hard (SRT push with constrained uplink, long-lived sessions), and under that load the H.265 output captured at the sink showed bursts of ~4 frames followed by multi-second gaps. I inferred that matched the size of the MediaCodec output buffer pool on that device. It's an observation from one device, not a spec value — I should have said so.

As for a real source to point to: as far as I know, on Android there is no public API to query the output buffer count. The old array-returning getOutputBuffers() was deprecated since API 21 and doesn't reflect the actual runtime pool; the modern access is per-buffer via getOutputBuffer(int), and there's no count getter at all. The pool size is decided by the codec implementation (vendor/HAL), so it can differ per device and per codec (H.264 vs HEVC vs AV1), and can even change at runtime (the codec signals this via INFO_OUTPUT_BUFFERS_CHANGED). So the "4" was specific to my device/codec, not a universal constant — I don't think there's a doc that specifies a fixed number.

On the channel capacity in EncodingPipelineOutput, that's entirely your call — the observation only suggests that a bounded channel with an overflow policy would prevent backpressure from propagating all the way into the codec (which is the actual fix), whatever specific capacity you choose; the value can be tuned. If it helps as a reference point, the RTMP path already uses a bounded channel (capacity 10, and 20 in the FLV endpoint) and doesn't show the stall, but the SRT-side decision is yours.

Thank you for your time and consideration.

Best regards,
Filippo Adessi

[filippoadessi]

That sounds like the right approach — a non-blocking SRT socket plus a bounded channel in EncodingPipelineOutput addresses the root cause much more cleanly than my encoder-level patch (which I agree adds GC pressure and goes against the zero-copy goal).

My PR #304 is closed, and that's fine — you know your codebase best.

If I can be of any help, I remain at your disposal.

Best regards,
Filippo Adessi

[ThibaultBee]

How have you tested this?

You can test this branch on StreamPack: https://github.com/ThibaultBee/StreamPack/pull/new/bugfix/encoder_stalled
You will have to build srtdroid with version 1.10.1 before (with the help of MavenLocal)