Skip to content

feat(moq-mux)!: unwrap fMP4 import to Legacy by default - #6250

Merged
kixelated merged 12 commits into
mainfrom
quest/m1/fmp4-import-unwrap
Oct 22, 2026
Merged

kixelated merged 12 commits into
mainfrom
quest/m1/fmp4-import-unwrap

Conversation

@kixelated

@kixelated kixelated commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Implements /quest/m1/fmp4-import-unwrap.md: first-party fMP4 import stops publishing CMAF unless asked.

What changed

  • moq_mux::container::fmp4::Import now splits each fragment into one Legacy frame per sample by default. Sample i is stamped tfdt + durations before i + cts_i in the track's timescale and shifted through the importer's Timebase, the same catalog clock passthrough uses. The payload is sliced out of the input buffer, not copied. The codec description still comes from stsd.

    • Video groups open on sync samples (the depends_on rule sample::Sample::is_sync already applies). A fragment that joins mid-GOP is skipped up to its first sync sample, as FLV does.
    • Audio keeps the importer's existing segment logic: only a segment's first sample opens a group, so audio follows styp/cut()/video-keyframe segmentation (and stays one group per fragment for unsegmented audio-only input), exactly as passthrough grouped it.
    • Declared segments (styp, cut()) still cut each track's timeline, through a new crate-private container::Producer::boundary.
    • Unwrapped tracks go through catalog::Reserved::{video,audio} like the other demuxers, so bitrate, jitter (plus the cts reorder delay and the per-fragment burst), bandwidth claims, stats, and timelines come from container::Producer.
  • Import::with_passthrough(true) keeps today's CMAF passthrough. The passthrough path is the old code moved into a Passthrough struct; both paths share one trun parser (Fmp4Track::run).

  • import::ContainerFormat gains Cmaf (the passthrough importer). "cmaf" stops being an alias of "fmp4" and parses to it, so string callers that asked for CMAF keep getting CMAF.

  • moq-hls: import::Config::passthrough. moq-cli: import fmp4 --passthrough and import hls --passthrough.

  • FFI: MoqContainerFormat::Cmaf; moq-c: MOQ_CONTAINER_FORMAT_CMAF = 4; go/wrapper: media.ContainerFormatCmaf. py, swift, kt, dart, and cpp alias the generated UniFFI enum, so they get CMAF/.cmaf/kCmaf with no wrapper code. Each binding's doc/lib page says what the two formats do.

  • Bench: cmaf-unwrap-* measures the unwrapping import beside the existing cmaf-import-* passthrough rows. Unwrapping is faster, since it slices samples instead of re-encoding a per-track moof and copying the mdat:

    64 single-sample fragments passthrough unwrap
    1 track, 256 B 328 µs 305 µs
    1 track, 4096 B 658 µs 503 µs
    4 tracks, 256 B 1.92 ms 1.24 ms
    4 tracks, 4096 B 2.79 ms 2.25 ms

Producer audit

Only fmp4::Import defaulted to CMAF. moq-gst's sink defaults to legacy (its source only consumes CMAF), the moq-ffi/moq-c track importers default to Legacy, and moq-video/moq-audio only decode CMAF. The demo (just pub cmaf) and test recipes (test/native-play, test/audio-quality) run moq import fmp4, so they flip with the CLI; none needs CMAF output.

Public API and wire impact

Breaking (behavior): every fMP4 import that didn't opt in now advertises legacy instead of cmaf, publishes one frame per sample instead of per fragment, names its tracks .mp4-v/.mp4-a at the default microsecond timescale instead of .m4s at the mdhd timescale, and ends each video group on the Legacy duration marker. MIGRATE.md entries cover moq-mux, moq-hls, moq-cli, moq-ffi, moq-c, py, swift, kt, dart, go, and cpp.

Additive: fmp4::Import::with_passthrough, ContainerFormat::Cmaf, moq_hls::import::Config::passthrough, --passthrough, the FFI/C/Go Cmaf variants.

No wire format or draft change: legacy and cmaf are both existing containers.

Decisions

Settled by the maintainer, all as implemented:

  1. moq-mux opt-in shape: fmp4::Import::with_passthrough(bool), next to with_select. Rejected: with_container(Container) (allows a Cmaf { init } whose init is ignored) and an fmp4::Output enum.
  2. Container/FFI opt-in shape: a ContainerFormat::Cmaf variant (MoqContainerFormat::Cmaf, MOQ_CONTAINER_FORMAT_CMAF). Rejected: a passthrough: bool on every binding's container init struct.
  3. CLI spelling: import fmp4 --passthrough and import hls --passthrough, mirroring import ts --passthrough. Rejected: import cmaf.

Two readings of the quest plan also stand:

  • Audio keeps per-segment grouping. Packagers flag every audio sample as sync, so unwrapped audio opens a group only at a segment's first sample, as passthrough grouped it, instead of one group per packet.
  • Unwrapped video rolls a group at a declared segment only on a sync sample. A styp or cut() boundary still marks the track's timeline, but video groups open only on sync samples (every sync sample opens one), so a segment that starts on a non-keyframe extends the open group instead of starting one.

Not done here

  • An HLS EXT-X-DISCONTINUITY still writes no discontinuity marker group on media tracks, as before in passthrough. Unwrapped Legacy tracks could call container::Producer::discontinuity(), but that would spend the sequence seek reserved, shifting group numbers off the media sequence.

Checks

  • just check: js, rs, py, kt, go, dart, cpp, and obs pass; swift is skipped on Linux; check flake fails locally because the sandbox refuses unshare (/proc/self/uid_map: Operation not permitted), unrelated to this change.
  • New tests: unwrapped_samples_match_the_passthrough_fragments (bbb.mp4 imported both ways reads back as the same samples, timestamps, and group starts), unwrapped_samples_present_at_their_composition_time (an I/P/B fragment with cts). The segment, timeline, seek, jitter, and sample-defaults fixture tests now run in both modes.
  • quest check passes with this quest file deleted.
  • Not run: just test interop --all (it exercises no fMP4 import, and the FFI change only adds an enum value), test/audio-quality.

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

kixelated and others added 5 commits October 21, 2026 20:51
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fmp4::Import splits each fragment into one Legacy frame per sample;
with_passthrough(true), ContainerFormat::Cmaf, and --passthrough on
import fmp4 / import hls keep publishing CMAF.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… bindings

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Quest run outcome: the quest is implemented. It stays a draft until the maintainer settles the three opt-in shape decisions in the description: with_passthrough(bool), ContainerFormat::Cmaf, and --passthrough, with the alternatives listed there. Run /quest-iterate to settle them, then /quest-ready.

Suggested follow-ups:

  • Write a discontinuity marker on unwrapped media tracks at an HLS EXT-X-DISCONTINUITY, without spending the sequence seek reserved.
  • Add a demo recipe that publishes CMAF (import fmp4 --passthrough), so the lazy CMAF decode path in feat(hang)!: load CMAF decoding on demand #6219 keeps a local way to exercise it.

(Written by Claude Opus 5.5)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Decisions settled by the maintainer, all as implemented:

  1. moq-mux opt-in: fmp4::Import::with_passthrough(bool).
  2. Container/FFI opt-in: ContainerFormat::Cmaf (MoqContainerFormat::Cmaf, MOQ_CONTAINER_FORMAT_CMAF).
  3. CLI: import fmp4 --passthrough and import hls --passthrough.

Both readings of the quest plan stand: audio keeps per-segment grouping, and unwrapped video rolls a group at a declared segment only on a sync sample.

The description now records these instead of open decisions. This push also deletes the finished quest file and its references. Marking ready for review.

(Written by Claude Opus 5.5)

@kixelated
kixelated marked this pull request as ready for review October 22, 2026 05:02
@coderabbitai

coderabbitai Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 6 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 88cedb9b-b085-49c9-afd6-a2a02cdd7a6c
📥 Commits

Reviewing files that changed from the base of the PR and between 58fb8a3 and 7b05bf8.

📒 Files selected for processing: 39
  • cpp/moq/MIGRATE.md
  • dart/moq/MIGRATE.md
  • dart/moq_ffi/lib/src/moq.dart
  • doc/bin/cli.md
  • doc/bin/hls.md
  • doc/lib/c/index.md
  • doc/lib/cpp/index.md
  • doc/lib/dart/index.md
  • doc/lib/go/index.md
  • doc/lib/kt/index.md
  • doc/lib/py/index.md
  • doc/lib/rs/moq-mux.md
  • doc/lib/swift/index.md
  • go/wrapper/MIGRATE.md
  • go/wrapper/media/media.go
  • kt/moq/MIGRATE.md
  • py/moq-rs/MIGRATE.md
  • quest/m1/README.md
  • quest/m1/fmp4-import-unwrap.md
  • rs/moq-c/MIGRATE.md
  • rs/moq-c/src/api.rs
  • rs/moq-cli/MIGRATE.md
  • rs/moq-cli/src/args.rs
  • rs/moq-cli/src/complete.rs
  • rs/moq-cli/src/hls.rs
  • rs/moq-cli/src/main.rs
  • rs/moq-cli/src/publish.rs
  • rs/moq-ffi/MIGRATE.md
  • rs/moq-ffi/src/media.rs
  • rs/moq-hls/MIGRATE.md
  • rs/moq-hls/src/import.rs
  • rs/moq-mux/MIGRATE.md
  • rs/moq-mux/benches/containers.rs
  • rs/moq-mux/src/container/fmp4/import.rs
  • rs/moq-mux/src/container/fmp4/import_test.rs
  • rs/moq-mux/src/container/producer.rs
  • rs/moq-mux/src/import/container.rs
  • rs/moq-mux/src/import/format.rs
  • swift/MIGRATE.md
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-22T12:21:57.344527Z 7b05bf8 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9e91e2a161

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread rs/moq-mux/src/import/format.rs Resolved
@kixelated

Copy link
Copy Markdown
Collaborator Author

Merge summary:

  • Default fMP4 import now unwraps to Legacy (one frame per sample); fmp4::Import::with_passthrough(true) keeps whole-fragment CMAF passthrough. String/FFI callers opt in via ContainerFormat::Cmaf / MoqContainerFormat::Cmaf / MOQ_CONTAINER_FORMAT_CMAF.
  • Addressed the Codex finding: ContainerFormat::Cmaf is appended after Flv so existing discriminants stay stable (matches the FFI and C enums).
  • Merged origin/main; no conflicts. just check passed except the flake module, which failed building the github-runner nix derivation, unrelated to this diff.

Changes since the Codex review are the one-variant reorder plus the main merge, so no fresh review was requested. Enabling auto-merge on 47d2fac.

(Written by Claude Opus 5.5)

@kixelated
kixelated added this pull request to the merge queue Oct 22, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 47d2facd50

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread rs/moq-mux/src/container/fmp4/import.rs Resolved
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Oct 22, 2026
kixelated and others added 2 commits October 22, 2026 02:28
Keep both MIGRATE.md Unreleased entries, and drop the hang-cmaf-lazy
and fmp4-import-unwrap quests, finished by #6219 and this PR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Unwrapped video skips samples until a sync sample, but the skipped
fragment still marked the segment opened, so the next keyframe rolled a
second segment and left audio one group ahead of video.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Re-queue summary after the merge queue dropped this for a main conflict:

  • Merged origin/main. Conflicts were the ## Unreleased MIGRATE.md entries in cpp, dart, go, kt, py, moq-ffi and swift (kept both, this PR's first), and the m1 quests: moq-dev/moq#6219 finished hang-cmaf-lazy while this PR finishes fmp4-import-unwrap, so both files and both README lines are gone.
  • Fixed the outstanding Codex finding: a track marks its segment opened only once it has a group open, so a mid-GOP video join no longer leaves audio one group ahead. New test a_mid_gop_join_keeps_tracks_on_the_same_groups reproduced it first.
  • Local: moq-mux, moq-cli, moq-gst, moq-ffi, moq-hls and moq-c tests and clippy -D warnings pass; quest check passes. Codex reviewed 74831dd clean.

(Written by Claude Opus 5.5)

@kixelated
kixelated added this pull request to the merge queue Oct 22, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Oct 22, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Merged origin/main (d92594b) to clear the merge-queue conflict. Mechanical resolution, no new logic:

  • rs/moq-hls/MIGRATE.md: kept both ## Unreleased entries (this PR's Legacy unwrap and main's ended-playlist import).
  • rs/moq-cli/src/hls.rs: kept this PR's import(target, args: ImportArgs, epoch) signature with main's updated doc comment about serving the finished broadcast.

quest check (plain and --base origin/main), clippy -D warnings, fmt, and tests for moq-mux, moq-hls, moq-cli, moq-ffi, and moq-c pass locally.

(Written by Claude Opus 5.5)

@kixelated
kixelated added this pull request to the merge queue Oct 22, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d92594bd6a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +1282 to +1283
if sample.pts > sample.dts {
producer.reorder(Timestamp::new(sample.pts - sample.dts, timescale)?);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Ignore reorder delay from skipped pre-keyframe samples

When an import begins mid-GOP, producer.write rejects non-sync samples with MissingKeyframe, but their PTS - DTS delay is recorded before that rejection. Because the estimator retains its maximum reorder delay and burst() subsequently publishes it, a dropped sample with a large composition offset can permanently advertise excessive jitter for a track that never emitted that sample, causing consumers to buffer unnecessarily. Record the reorder delay only after the frame is accepted.

Useful? React with 👍 / 👎.

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Oct 22, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Merged origin/main to clear the queue conflict. The only conflict was rs/moq-mux/MIGRATE.md under ## Unreleased, resolved by keeping both entries (this PR's fMP4 unwrap note and main's Container::decode note). No code changes; clippy -D warnings, fmt and tests pass for moq-mux, moq-hls, moq-cli, moq-ffi and moq-c, and quest check is clean.

(Written by Claude Opus 5.5)

@kixelated
kixelated added this pull request to the merge queue Oct 22, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7b05bf8512

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

for (index, sample) in run.samples.iter().enumerate() {
let keyframe = match video {
// Video groups open on sync samples, the `depends_on` rule the rest of moq-mux applies.
true => sample.sync,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Roll audio for every video keyframe in a fragment

When one traf contains multiple sync samples, this marks every sync sample as a video group opener, while the audio path opens at most one group because placement.new_segment advances only once per moof. A valid fragment spanning multiple GOPs therefore advances the video group sequence ahead of audio, breaking the importer's cross-track grouping. Split audio at the corresponding boundaries or ensure one video group is opened per resolved segment.

AGENTS.md reference: rs/moq-mux/AGENTS.md:L3-L5

Useful? React with 👍 / 👎.


// The fragment arrived at once, so a consumer has to buffer its whole span.
producer.burst(end.checked_sub(earliest)?.into())?;
Ok(!producer.needs_keyframe())

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Do not mark dependent video segments as opened

When an styp or cut() declares a segment whose video contains no sync sample, an existing video group remains open, so this returns true even though the new segment never opened a video group. track.segment is then advanced; at the next sync fragment starts_segment() advances again, causing audio to open another group while video is still one sequence behind. Report whether this invocation actually opened the segment rather than whether any old group remains open.

AGENTS.md reference: rs/moq-mux/AGENTS.md:L3-L5

Useful? React with 👍 / 👎.

}
}
track.pending_sequence = Some(sequence);
media!(&mut track.output, producer => producer.seek(sequence)?, passthrough => {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reset segment ownership when seeking unwrapped tracks

After seek() on an already-started muxed import, this closes each producer but leaves track.segment equal to the current segment. If the resumed video is initially mid-GOP, video_rolled() therefore still returns true; audio opens the requested sequence on the skipped fragment, then the first video sync fragment advances the segment and rolls audio again while video only opens the requested sequence. Clear or invalidate the per-track segment marker when the seek closes its group.

AGENTS.md reference: rs/moq-mux/AGENTS.md:L3-L5

Useful? React with 👍 / 👎.

}

// The fragment arrived at once, so a consumer has to buffer its whole span.
producer.burst(end.checked_sub(earliest)?.into())?;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Exclude skipped samples from burst estimates

When video import begins mid-GOP, producer.write intentionally rejects the leading samples, but this still records the span from the run's earliest sample through its end. A fragment containing only rejected samples, or a long rejected prefix before the first sync sample, consequently publishes jitter for media that was never emitted and can make consumers buffer unnecessarily. Compute the burst from accepted samples only, and skip it when none were written.

Useful? React with 👍 / 👎.

Merged via the queue into main with commit a84c31c Oct 22, 2026
11 checks passed
@kixelated
kixelated deleted the quest/m1/fmp4-import-unwrap branch October 22, 2026 12:39
kixelated added a commit that referenced this pull request Oct 22, 2026
…/ingest

Resolve conflicts with main's fMP4 import refactor (#6250), frame arrival
stamps, the clock rename, and the #6165 MIGRATE.md entries. The owned
frame writer now builds frames with `Frame::new`, and main's arrival-stamp
test takes the group writer back from `finish`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kixelated added a commit that referenced this pull request Oct 24, 2026
Replaces main's new ContainerFormat::Cmaf release test with fmp4 passthrough,
and adds the #6250 Fmp4 entry to the raw binding MIGRATE files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant