Skip to content

feat(mux): every frame of a group presents before every frame of a higher group - #9264

Open
kixelated wants to merge 7 commits into
mainfrom
quest/m2/groups-never-overlap
Open

kixelated wants to merge 7 commits into
mainfrom
quest/m2/groups-never-overlap

Conversation

@kixelated

@kixelated kixelated commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Finishes quest/m2/groups-never-overlap.md.

Problem

Groups could go back in time. Since moq-dev/moq#4543, container::Producer opened a group for any keyframe above the open group's start, even one below its newest frame, so a TS content join published an IDR below the old content's last frames, and a later frame could present anywhere above the previous group's start. A reader then can't bound a group that isn't whole by a higher group's lowest timestamp, and HLS edges could list one segment differently (moq-dev/moq#8874). lite-07 publishes the strong rule as a publisher MUST, so first-party producers have to keep it first.

Approach

Every frame of a group presents strictly after every frame of every lower group; frames within a group still reorder.

  • container::Producer (rs/moq-mux/src/container/producer.rs) tracks a ceiling, the newest frame of the groups before the open one.
    • A keyframe at or before the newest frame written continues the open group, so a content join publishes one longer group (about two GOPs), with no timestamp rewritten.
    • A later frame at or before the ceiling is dropped and counted in the catalog's skipped, as a leading picture after a splice would be. So is a keyframe at or before the newest frame with no group open, such as after the TS importer cuts on lost bytes, and with it the frames after it until a keyframe opens a group (skipping).
    • A keyframe below the latest group's start, or any frame below the start of the group before its own, is still refused as TimestampRewind (a restart).
    • A keyframe always cuts at its own timestamp now: the old "overlaps" branch that cut without a bound can't happen.
    • open/commit (the streamed path fMP4 and MKV use) carry a dropped frame as Payload::Dropped, which discards the bytes.
  • fMP4 passthrough (fmp4/import.rs), which writes whole CMAF fragments without the producer, now decides where each fragment lands while the moof is planned (Passthrough::plan), and start() only executes that decision.
    • A fragment that would open a group at or before the newest sample continues the open group. With none open, it is dropped whole and the track is orphaned until its next sync sample.
    • Samples at or before the ceiling of the group the fragment enters are cut out of the run and the traf is rewritten through kept(), as edit-list drops are, so a fragment mixing a leading picture with valid pictures keeps the valid ones.
    • A restart below the latest group's start is refused at planning, before anything of it publishes.
  • JS js/hang legacy Producer mirrors the Rust producer.
  • hang draft: the group rule becomes the strong rule. A group consists of a keyframe followed by frames that decode from those before them, since a splice's continuing keyframe sits mid-group. The draft says how a splicing publisher keeps the rule. "A group MAY start at the same timestamp as the previous group" and "Frames MAY present below the previous group's content" are gone, and the hang-04 changelog bullet says what changed.
  • Readers assume the rule rather than detect it. The Rust and JS reader tests that accepted an overlapping group are deleted, and the comments describing overlap are updated. Nothing else on the reader side existed only for overlap: the gap clamps still serve durations that overrun the next start.
  • The HLS grid's overlap note (rs/moq-hls/src/export/grid.rs, doc/bin/hls.md) is deleted, since a group whose frame passes a line now always contains it.

Sources surveyed: TS, fMP4 (unwrapped and passthrough), FLV/RTMP, MKV, SRT, HLS import, the codec importers, gst, moq-ffi/libmoq media, and WebRTC all go through container::Producer, except the fMP4 passthrough, which this PR handles. moq-transcode and moq-archive copy their source's groups, so they keep whatever their source keeps. The JS publishers never reorder (their encoders refuse B-frames), and JS has no importer.

Choices made without asking, for review:

  • RADL pictures are dropped too, as the quest left open: at a splice one that presents at or before the old content's newest frame would show new content while the old still plays, and its group already opened past it.
  • A keyframe with no group open that doesn't pass the newest frame is dropped rather than refused. Refusing ended a TS import at a content join that coincided with lost bytes (seen on a real ffmpeg splice with a continuity gap, below). The cost is the new content's first GOP after such a loss.
  • A keyframe below the latest group's start is still refused. The plan's "as one at or below its start already does" fits only "at": below the start has always been a restart.

Impact

  • Public API: none.
  • Wire: none. The hang draft's group rule tightens: a group that overlaps a lower one is malformed.
  • Behavior (moq-mux container::Producer::write, the fMP4 importer, @moq/hang legacy Producer.encode): an overlapping keyframe continues its group instead of opening one, and frames that would present with an earlier group are dropped and counted as skipped. A keyframe overlapping a closed group is dropped, not refused (TimestampRewind). Today's MIGRATE line ("refuses a group that does not start strictly after the previous group's start") stays true. The release cut can add the continuation and the drop.

Validation

  • Real input through the real importer: ffmpeg encoded a 4 s x264 B-frame clip and a -bf 0 clip offset so its IDR (pts 486000) lands one frame below the old content's last P-frame (489000), concatenated on one clock.
    • moq import ts into a local moq-relay exits 0, both with the continuity counters fixed (tsp -P continuity --fix) and without.
    • Read back through the TS importer and container::Consumer, the fixed file publishes all 240 frames in 7 groups. The join group holds 60 frames (two GOPs) and every group's frames all follow the previous group's. Without the fix, the cut on lost bytes drops the new content's first GOP: 209 frames, no overlap.
    • An earlier revision of this PR refused that keyframe, and the unfixed file ended the import with TimestampRewind, which is why that case drops.
  • A property test (groups_never_overlap_on_random_streams, 256 random streams with reordering, splices, and cuts) checks that no group overlaps a lower one, and that a frame is dropped only when it presents at or before a frame already published. Disabling the ceiling drop makes it fail.
  • New tests:
    • Producer: an overlapping keyframe continues the group; a leading picture after a splice is dropped; a keyframe overlapping a closed group is dropped.
    • TS: a join below a P-frame continues the group (3 groups, strong rule); a join's leading picture is dropped.
    • fMP4 (video_groups_never_overlap): unwrapped and passthrough both keep the rule, including a mixed fragment, a resume after rejoin() at the newest sample, and one moof of two runs of the track.
    • fMP4 (audio_runs_in_one_moof_share_a_group): a segment's first moof of two audio runs of one track opens one group, unwrapped and passthrough. It failed on the passthrough (four groups instead of two) before the plan loop advanced the segment.
    • fMP4 (video_continuing_a_seeked_group_keeps_its_leading_pictures): after a seek, a sync sample whose key doesn't pass the seek's start continues the group and keeps its leading picture, unwrapped and passthrough. The passthrough dropped it before its plan decided the continuation ahead of trimming (found by the review's shadow in round 5).
    • JS: the producer's continuation and both drops.
  • After merging main (conflict in doc/bin/hls.md: main's overrun text kept, the overlap sentences dropped), cargo test -p moq-mux (2139 passed) and just check (4949 tests) pass.
  • just rs test passes except moq-tokio's QUIC socket tests, which timed out under the full run's load and pass alone (599/599).
  • bun test passes in js/hang/src/container, and just check passes.

Alternatives

Caused by

  • No single PR: moq-dev/moq#4543 allowed overlap on purpose under the weaker rule. The strong rule is new (maintainer, 2026-11-20).

Follow-ups

None added. Tracked in moq-dev/moq#9263 (quest/m2/time-addressing/group-times.md), whose plan this PR updates: container::Producer still finishes a group whole at the next keyframe's timestamp, above that keyframe's open-GOP leading pictures. That's valid under today's End rule (no greater than the next start). Under group times' spans, measured from the lowest frame, the two spans overlap. Which End to write belongs to that quest.

quest/m3/fmp4-group-start-tfdt.md loses its exact-repeat case, which a continuing keyframe now covers. Its near-repeat case remains.

Decisions

  • Unattended choice, for review (orchestrator for an away maintainer, 2026-11-20): at the local review's round-3 checkpoint, with one medium finding open (two audio runs of one track in a segment's first moof opened two passthrough groups, since track.segment advanced only while writing), ✅ continue: the passthrough's plan loop advances track.segment when plan returns Land::Opens, so its plan owns all of its grouping state (rejected: rethink, extracting one grouping state machine shared by container::Producer::prepare and Passthrough::plan, a larger refactor of duplication that predates this PR and that the rule doesn't need). A later review can still pick the rethink.

Models

Size: L
Implementer: Claude, claude-opus-5-5, high effort
Implementer: Claude Code, claude-opus-5-5, unknown effort (iterate, L)

🤖 Generated with Claude Code

Review

Rewritten by quest review after each round.

  • Head: 4ea1ee0

  • Base: origin/main at cb9cb51

  • Risk: high, Tightening the hang wire contract and changing Rust and JS producers to merge groups and drop overlapping frames can break published consumers and deployed peers relying on prior grouping behavior; reverting restores future behavior but cannot recover discarded media.

  • Classified: high by Codex, gpt-6.1-sol, low effort, session 01a1fc77-37de-7030-9d62-81a8eee5d05d

  • Reviewer: Codex, gpt-6.1-sol, xhigh effort

  • Session: 01a1fc0a-e616-7291-8ca5-449b9fee5f79, round 5

  • Score: 10/10, The update correctly decides group placement before trimming, preserves valid leading pictures, and introduces no new findings.

  • Rounds: 5; first round 2 (medium 2); findings 4 (medium 4): fixed 4, refuted 0, open 0

  • Findings: none

  • Reviewer: Codex, gpt-6.1-sol, high effort, shadow

  • Session: 01a1fc65-727f-7b03-a523-38d199028738, round 2

  • Score: 10/10, The incremental change fixes the trimming regression and adds focused coverage, with no further issues found by code inspection; builds and tests were not run.

  • Rounds: 2; first round 1 (medium 1); findings 1 (medium 1): fixed 1, refuted 0, open 0

  • Findings: none

  • The last round's findings, at dfae86b:

    • Fixed: medium, rs/moq-mux/src/container/fmp4/import.rs:1673: Resolve group placement before selecting the trim ceiling; rs/moq-mux/src/container/fmp4/import.rs:1678 now resolves key-based continuation before selecting the trim ceiling, preserving leading pictures in the seeked group; the added regression test covers this scenario.

kixelated and others added 2 commits November 20, 2026 17:12
…e every frame of a higher group

A keyframe at or before the newest frame written continues the open group
(a TS content join), and a later frame presenting at or before an earlier
group's newest frame (a leading picture after a splice) is dropped and
counted as skipped. The Rust producer, the fMP4 passthrough, and the JS
legacy producer keep it; the hang draft states it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title Every frame of a group presents before every frame of a higher group feat(mux): every frame of a group presents before every frame of a higher group Nov 21, 2026
kixelated and others added 2 commits November 20, 2026 19:21
… what decodes from a dropped keyframe

The fMP4 passthrough drops only the samples of a fragment that present with
an earlier group, rewriting it as the edit list does, instead of the whole
fragment. A keyframe dropped with no group open takes the frames decoding
from it along until a keyframe opens a group, in Rust and JS, rather than
refusing them with MissingKeyframe.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…itten

Passthrough::plan advances the track's planned group state (open, start, key,
newest, ceiling) per traf, so a moof with several runs of one track trims each
against the group it actually enters. start() only executes the plan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kixelated and others added 3 commits November 20, 2026 20:22
…ns a group

Two audio runs of one track in a segment's first moof opened two groups,
since the segment state advanced only while writing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…against the group's ceiling

The plan trimmed a fragment against the newest sample before deciding,
from its key, that it continued the open group, so after a seek a
leading picture the group could hold was dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review November 21, 2026 04:58
@coderabbitai

coderabbitai Bot commented Nov 21, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 9 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 344021ed-b38c-4c1e-aebd-564ca9c966d7
Commits

Reviewing files that changed from the base of the PR and between b4496b1 and 4ea1ee0.

Files selected for processing: 17

  • doc/bin/hls.md
  • drafts/draft-lcurley-moq-hang.md
  • js/hang/src/container/consumer.rewind.test.ts
  • js/hang/src/container/consumer.test.ts
  • js/hang/src/container/legacy.ts
  • quest/m2/groups-never-overlap.md
  • quest/m2/lite07-finalize.md
  • quest/m2/time-addressing/group-times.md
  • quest/m3/expire-by-higher-min.md
  • quest/m3/fmp4-group-start-tfdt.md
  • rs/moq-hls/src/export/grid.rs
  • rs/moq-mux/src/container/consumer.rs
  • rs/moq-mux/src/container/fmp4/import.rs
  • rs/moq-mux/src/container/fmp4/import_test.rs
  • rs/moq-mux/src/container/producer.rs
  • rs/moq-mux/src/container/ts/import.rs
  • rs/moq-mux/src/start.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ No critical issues. Two places where the fMP4 passthrough drifts from the unwrapped path; comments inline.

Approach: Sound. Keeping a ceiling at each producer lets an overlapping keyframe continue its group and drops frames that would present with an earlier group, which fixes the cause at the producer rather than shifting timestamps. The passthrough now decides where each fragment lands before writing it, which keeps the two fMP4 paths aligned.

Reviewed changes

This review covers all 7 commits at 4ea1ee0: the Rust and JS producers, the fMP4 passthrough planner, the TS and fMP4 tests, and the draft, doc and quest edits.

  • container::Producer::prepare: a keyframe at or before newest continues the open group. With no group open, it is dropped and skipping begins. A delta at or before ceiling is dropped and counted as skipped. The old overlaps cut is gone, which is safe because an opening keyframe is always past previous_timestamp.
  • Streamed Payload::Dropped: open/commit discard a dropped frame's bytes without taking the group, and still check its size.
  • Passthrough::plan: runs at begin_mdat and lands each fragment as Opens or Continues. It trims samples at or before the ceiling through kept(), and mirrors start/key/newest/ceiling ahead of the writes.
  • JS Producer.encode: mirrors the Rust rules with #ceiling and #skipping.
  • hang draft: the group rule becomes the strong rule. The removed MUSTs follow from it, and the splice guidance and the hang-04 changelog are updated to match.
  • Readers/HLS/quests: the overlap tests and notes are deleted, and no reference to the deleted quest remains.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using claude-opus-5-5 | 𝕏

Comment on lines +1652 to +1663
// A group may not start below the one before it: a restart, refused before anything of it
// publishes, as an unwrapped track does.
if opens
&& let Some(floor) = self.start
&& earliest < floor
{
return Err(crate::container::TimestampRewind {
timestamp: earliest,
floor,
}
.into());
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This restart check compares the fragment's earliest sample with self.start before plan decides whether the fragment continues the open group or is dropped. So a splice fragment that would end up continuing (sync sample at or before newest) fails the import with TimestampRewind if one of its leading pictures sits below the open group's start. The unwrapped container::Producer accepts the same input: it checks the keyframe against start and the deltas against floor/ceiling.

Technical details
# Passthrough restart check runs before the continuation decision

## Affected sites
- rs/moq-mux/src/container/fmp4/import.rs:1652-1663: `if opens && let Some(floor) = self.start && earliest < floor` runs while `opens` is still the tentative `!self.open || (new_segment && keyframe)`.

## Example
G1 starts at 100. G2 opens at 200 with ceiling 180, and newest is 250. A new-segment fragment has a sync sample at 240 and then a leading picture at 190.
- Unwrapped: 240 >= start 200, so it continues. 190 >= floor 100 and > ceiling 180, so it is kept.
- Passthrough: `earliest` = 190 < `start` 200, so it is refused and the import ends.

## Required outcome
- Apply the restart refusal only to a fragment that really opens a group: after the `at <= newest` and key-based continuation decisions. A continuing fragment's below-ceiling samples are already trimmed.
- A test like `video_groups_never_overlap`, with a continuing fragment whose leading picture sits below the open group's start, should give the same frames on both paths.

Comment on lines +1184 to +1186
if let Land::Opens { .. } = land {
track.segment = Some(self.segment);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

track.segment is set here only for Land::Opens. When a sync fragment at a new segment continues the open group (a splice), the segment never counts as rolled. In a stream without styp, video_rolled() then stays false, so the next keyframe moof doesn't move to a new segment, and audio keeps its group across that video keyframe. The unwrapped path marks the segment as soon as it sees a sync sample (open_sample returns opened = keyframe), so after a splice the two paths group audio differently. CMAF input with a styp per segment isn't affected, since pending_cut forces the boundary.

Technical details
# Passthrough continuation at a segment boundary doesn't mark the segment as rolled

## Affected sites
- rs/moq-mux/src/container/fmp4/import.rs:1184-1186: segment marked only on `Land::Opens`
- rs/moq-mux/src/container/fmp4/import.rs:1112-1117 (`video_rolled`) and 1088 (`starts_segment` returns false until video rolls)
- rs/moq-mux/src/container/fmp4/import.rs:1850 (unwrapped `open_sample` reports `opened = keyframe` even when the producer continues or drops)

## Required outcome
- When a sync-led fragment at a new segment continues the group, passthrough marks the segment the same way unwrapped does, so the next video keyframe starts a segment and audio rolls with it.
- A test with both an audio and a video track, no `styp`, and a continuing keyframe should give the same audio groups on both paths.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant