Skip to content

feat: adaptive bitrate (ABR) over MoQ — qumo abr transcoder and playground ABR player - #602

Merged
okdaichi merged 13 commits into
mainfrom
ccr-2161bc2c-wkewvo
Nov 17, 2026
Merged

okdaichi merged 13 commits into
mainfrom
ccr-2161bc2c-wkewvo

Conversation

@okdaichi

@okdaichi okdaichi commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Description

This PR adds end-to-end adaptive bitrate streaming over MoQ, to demonstrate a MoQ advantage: a player switches rendition at any group boundary just by changing which track it subscribes to. There are no manifests and no segments.

  1. qumo abr (server side) is a relay client. For every broadcast announced beneath ABR_PREFIX, it publishes an ABR broadcast beside it (/live/cam1 → /live/cam1.abr). That broadcast carries the source's tracks unchanged, plus ffmpeg-encoded renditions of a ladder (default 720p/360p), all in one MSF altGroup.
  2. Playground ABR player (client side): the viewer plays such a broadcast adaptively, switching renditions at group boundaries. The subscribe board gets an Auto/manual rendition picker and ABR stats.
  3. Relay fix: when a subscriber stopped one group's stream, the relay ended that subscriber's whole subscription. Now the relay abandons only that group.

Related Issue

None.

Changes

qumo abr (internal/abr)

  • Each video track gets altGroup: 1, width, height, bitrate, codec and initRef. The source's bitrate is measured from its first few groups.
  • Every rendition numbers its groups as the source does, and each group starts with a keyframe at the same frame (-force_key_frames source). That makes any group boundary a valid switch point.
  • Rendition frames keep the exact LOC timestamp of their source frame. Encoder outputs are matched back to source frames in order and confirmed by timestamp.
  • Timestamps go to ffmpeg relative to the first frame. FLV carries 32-bit milliseconds, which a source on a large clock (µs since 1970) would overflow.
  • One ffmpeg process per source decodes once for all renditions. Frames go in as fragmented MP4 (one fragment per frame) and come out as FLV (one pipe per rendition). Both formats are read and written frame by frame, so neither adds a frame of delay. In a measurement, transcoding added about 8–10 ms per frame.
  • Encoding uses a µs time base (-enc_time_base 1:1000000). ffmpeg's default time base would snap live timestamps to a frame-rate grid.
  • The source track is forwarded without re-encoding and never waits on the encoder. If the encoder falls behind, it drops frames up to the next group.
  • Aliases are not transcoded. A broadcast whose catalog subscription is redirected to another (such as an HLS opt-in under /.hls/) is recognized by that redirect; the broadcast it points to is transcoded at its own path. No path prefix is hard-coded.
  • Requirements: the source must be H.264 without B-frames, and ffmpeg must be on PATH.
  • Docs: docs/site/content/en/docs/cli/abr.md, the README, the CLI index and CHANGELOG.md.

Playground player (playground/src/player)

  • catalog.ts: videoRenditions() returns the first video track and its altGroup alternates, tallest first.
  • abr/switching_track.ts merges the renditions' subscriptions into the one track the existing frame reader consumes:
    • Because renditions share group numbers, the new rendition takes over at its first group past those already handed out.
    • If the new rendition joins at a group that has already played, the next group is reserved for it, so the old rendition can't race it.
    • On a step up, the old subscription stays until it has sent every group before the cutoff (make before break). On a step down, it ends at once to free the link.
  • abr/controller.ts decides when to switch. A live stream is sent only as fast as it's made, so the receive rate can't show spare capacity. The controller uses lateness instead:
    • It steps down when frames fall behind their media time, or when a group is lost. It picks the best rendition that fits, with headroom, the rate measured since the frames fell behind.
    • It steps up one rendition after a calm stretch (4 s), and doubles that wait whenever an automatic step up doesn't hold.
    • It won't step up past the bitrate a subscription's opening burst measured.
  • viewer.ts: the decoder is configured for each group's rendition at its keyframe. A new setRendition() method and onAbr callback support manual selection and stats.
  • SubscribeBoard.tsx adds the rendition picker and abr/rx/burst/lag stats. The committed dist/ is rebuilt with Deno 2.8.1, and scripts/check-webui-dist.sh passes.

Relay (internal/relay/handler.go)

  • deliverGroup treated any WriteFrame failure as the subscriber leaving. A failure while the subscription is still open now returns groupStopped, which abandons only that group. This also affects the existing player, which already cancels late groups.

Testing

  • Unit tests:
    • internal/abr: ladder, FLV reader, frame matcher, track buffer (including group 0), bitrate meter (-race).
    • An encoder test against real ffmpeg, using off-grid, epoch-scale µs timestamps. It caught both the frame-rate time-base problem and the FLV overflow.
    • Playground: 295 tests pass, including the controller and switching track. deno check is clean.
  • Lint: golangci-lint v2.14.0 (built with Go 1.27, as in CI) reports 0 issues.
  • Integration (-tags=integration):
    • TestABR_RenditionsBesideSource: ffmpeg → RTMP ingest → relay → abr → relay → viewer. It checks the catalog and that every shared group has the same timestamps and frame count, and opens with a keyframe. It also announces an HLS opt-in beside the source and checks it is not transcoded (fails with the redirect check disabled).
    • TestRelay_StoppedGroupKeepsSubscription fails without the relay fix.
    • The existing relay, RTMP and ABR integration tests pass.
  • Browser (headless Chromium against qumo playground + qumo rtmp + qumo abr, 1080p source):
    • Playback climbs 360p → 720p → 1080p at 30 fps.
    • A 3 Mbit/s limit on the relay → browser link (tc htb filtered by the browser's port) drops it straight to 360p in about 2.5 s. It climbs back to 1080p once the limit is lifted.
    • Manual selection holds, and switching back to Auto resumes adapting.

Demo:

qumo playground
RELAY_URL=https://127.0.0.1:4433 RELAY_TLS_INSECURE=true qumo rtmp
RELAY_URL=https://127.0.0.1:4433 RELAY_TLS_INSECURE=true ABR_PREFIX=/live/ qumo abr
ffmpeg -re -f lavfi -i testsrc2=size=1920x1080:rate=30 -c:v libx264 -preset veryfast -tune zerolatency -bf 0 -g 60 -b:v 5M -f flv rtmp://127.0.0.1:1935/live/demo
# open http://127.0.0.1:8080/?scenario=echo&path=/live/demo.abr → Start Subscribing

Checklist

  • Comments added for complex logic
  • Documentation updated if needed
  • Tests added/updated

🤖 Generated with Claude Code

https://claude.ai/code/session_01KPWMJL7PH3rPhK5UTGavaB

Add `qumo abr`, a relay client that, for every broadcast announced
beneath ABR_PREFIX, publishes an ABR broadcast at the same path with
ABR_SUFFIX appended (/live/cam1 -> /live/cam1.abr). It carries the
source's tracks as they are plus one rendition per ladder rung shorter
than the source video, encoded by ffmpeg.

- Every video track is in one MSF alternate group (altGroup 1) with its
  width, height, bitrate, codec and initRef; the source's bitrate is
  measured from its first groups.
- Every rendition numbers its groups as the source does and opens each
  with a keyframe at the same frame (-force_key_frames source), so a
  player can switch tracks at any group boundary.
- Rendition frames keep the exact LOC timestamp of the source frame they
  were encoded from: outputs are matched back to source frames by order,
  confirmed by timestamp.
- One ffmpeg decodes the source once for all renditions. Frames go in as
  fragmented MP4 (one fragment per frame) and come out as FLV (one pipe
  per rendition): both are read and written frame by frame, so neither
  adds a frame of delay. Encoding uses a microsecond time base, as the
  default frame-rate time base snaps live timestamps to a grid.
- The source track is forwarded without re-encoding and never waits on
  the encoder; a lagging encoder drops frames up to the next group.

Tested by unit tests (ladder, FLV reader, matcher, track buffer, bitrate
meter), an encoder test against real ffmpeg with off-grid timestamps,
and an integration test: ffmpeg -> RTMP ingest -> relay -> abr -> relay
-> viewer, checking the catalog and group-by-group alignment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPWMJL7PH3rPhK5UTGavaB
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPWMJL7PH3rPhK5UTGavaB
The playground viewer plays an MSF broadcast whose video tracks share an
alternate group (as `qumo abr` publishes) adaptively, switching between
renditions at group boundaries, and lets the user pick one by hand.

- Renditions come from the catalog: the first video track and the video
  tracks in its altGroup, tallest first, each with its decoder config.
- A SwitchingTrack merges the renditions' subscriptions into the one
  track the frame reader consumes. Renditions number their groups alike,
  so the new rendition takes over at its first group past those handed
  out; when it joins at a group already played, the next group is
  reserved for it so the old rendition cannot race it. A step up keeps
  the old subscription until it has sent every group before the cutoff
  (make before break); a step down ends it at once to free the link.
- The decoder is configured for each group's rendition at its keyframe.
- An AbrController steps down when frames fall behind their media time
  (or a group is lost), to the best rendition the receive rate since
  they fell behind carries with headroom; it steps up one rendition
  after a calm stretch, doubling the wait whenever a step up does not
  hold, and not past what a subscription's opening burst measured.
- The subscribe board gains an Auto/rendition picker and ABR stats in
  the overlay.

fix(relay): a subscriber stopping one group keeps its subscription

deliverGroup treated any WriteFrame failure as the subscriber leaving
and ended its egress, so a subscriber that stopped one group's stream
(STOP_SENDING), as a player gives up a late group or one of a rendition
it is leaving, lost the whole subscription. A failure while the
subscription is still open now abandons only that group.

Verified in headless Chromium against playground + rtmp + abr: playback
climbs 360p -> 720p -> 1080p; a 3 Mbit/s limit on the relay-to-browser
link drops it to 360p within ~2.5 s, and it climbs back once lifted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPWMJL7PH3rPhK5UTGavaB
gosec G204: ffmpeg's path is the operator's ABR_FFMPEG and its arguments
are built from the ladder, so the launch is marked as intended.
staticcheck SA4023: encoder.read returns only once its stream ends, so
the nil check on its error was always true.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPWMJL7PH3rPhK5UTGavaB
…aliases

- Rebase source timestamps on the first frame before handing them to
  ffmpeg: its FLV output carries 32-bit milliseconds, which a source on a
  large clock (microseconds since 1970) overflowed, so no rendition frame
  matched its source frame. The encoder test now uses such timestamps.
- Close the encoders already started when a later one fails to start,
  so their ffmpeg is waited for rather than left a zombie (Windows starts
  one per rendition).
- Do not transcode HLS opt-in announcements (/.hls/...): they name a
  broadcast already transcoded at its own path.
- Treat group sequence 0 as a group: readTrack no longer drops it, and
  the track buffer tracks its position with flags, not a 0 sentinel.
- Integration test: no require from the reader goroutines.
- Playground SwitchingTrack: watch each stop signal once, not with a new
  handler on the long-lived signal per acceptGroup call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPWMJL7PH3rPhK5UTGavaB
…efix

A broadcast whose catalog subscription is redirected to another is an
alias of it, as an HLS opt-in is: qumo abr no longer transcodes it, and
transcodes the broadcast it names, announced at its own path. This
replaces the hard-coded token.HLSDirectory check with what makes those
announcements aliases, so any other alias is covered too.

The integration test announces an HLS opt-in beside the source and
checks it is not transcoded; with the redirect check disabled it fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPWMJL7PH3rPhK5UTGavaB
…ge cases

Go unit tests (internal/abr coverage 37% -> 52%):
- outputBroadcast: init data listed under the track's name and replaced,
  not duplicated; TRACK_INFO passes the source's properties on; bitrate.
- renditions: the catalog entry a configuration adds (altGroup, codec,
  size, bitrate, initRef); frames dropped before configuration and before
  a group opens; groups numbered as the source's, with its LOC and wire
  timestamps; a group opening on a non-keyframe continues the last one.
- initData, isAVC, outputTrack, AliasError.
- startEncoders stops the ffmpeg it started when a later one fails.

Integration (TestABR_RenditionsBesideSource): audio is forwarded, the
source's measured bitrate reaches the catalog, and the ABR broadcast is
withdrawn when its source stops.

Playground: the controller does not step below the lowest rendition,
forgets a burst measurement after 30 s, steps up to a rendition of
unknown bitrate, follows catalog updates, and falls to the lowest when
nothing fits; SwitchingTrack lets go of a quiet old rendition after 5 s
and closing ends every subscription and cancels queued groups.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPWMJL7PH3rPhK5UTGavaB
Resolves the CHANGELOG [Unreleased] conflict with #603 by keeping both
sides' entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPWMJL7PH3rPhK5UTGavaB
okdaichi and others added 2 commits November 18, 2026 04:20
The merge of main filed them under the release heading it brought in.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPWMJNAzsnsiqRPREa2JqY
okdaichi and others added 3 commits November 18, 2026 05:00
… fail

- TestStartEncodersClosesStartedOnFailure passed with the cleanup it names
  removed. Its ffmpeg is now the test binary (TestMain), which marks when its
  input is closed, so the test fails unless the started encoder is closed and
  waited for. It no longer needs ffmpeg on PATH.
- renditions.configured needs no mutex: each element is touched only by its
  rendition's goroutine (renditionSink).
- Remove flvTagAudio (unused) and encoder.close's nil-cmd check (close is
  only called on started encoders).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPWMJNAzsnsiqRPREa2JqY
- AbrController's seven tuning options were set by no caller: they are module
  constants beside WINDOW_MS. Only the clock stays an option (tests).
- decoderConfigs built a video config no one read (video is configured per
  rendition): it is now audioConfig. Rendition.width was never read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPWMJNAzsnsiqRPREa2JqY
…anges

f5729d5 changed playground/src without rebuilding the committed dist, so
Web UI dist freshness failed. Rebuilt with deno 2.8.1, as CI pins it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPWMJL7PH3rPhK5UTGavaB

Copy link
Copy Markdown
Collaborator Author

Integration failed on 9c75f9c in internal/relay TestServer_MaxPendingAnonymous (admit_test.go:1321, "a session granted a ticket makes room"). I don't think this PR caused it:

  • 9c75f9c only rebuilds playground/dist, and the relay code is untouched since the last green run.
  • The log shows two session refused: too many sessions hold no credential yet lines, one more than the test expects. So the dial right after the first session's ticket was granted was still refused.

What I think is happening, the same pattern as the TicketSupersedes flake on #608:

  • internal/relay/admit.go answers AUTH_OK (w.Grant) before it settles the anonymous session as granted (settleGranted), which is what frees its pending slot.
  • authTest returns on AUTH_OK, and the test dials again at once.
  • If that dial lands before settleGranted, the slot is still taken and the dial is refused.

I'm re-running the failed job once. The ordering fix in the relay (settle and register before answering the grant, or have the test wait for the slot) is a separate change.


Generated by Claude Code

@okdaichi
okdaichi merged commit 6de0180 into main Nov 17, 2026
21 of 22 checks passed
okdaichi added a commit that referenced this pull request Nov 17, 2026
…o-abr-optin

Merged with -s ours, then #606 and #607's changes applied on top: this
branch already holds #602, whose squash on main would otherwise conflict
with every line this branch changed in it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KPWMJNAzsnsiqRPREa2JqY
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants