Skip to content

export ts: after re-requesting onto a live route, the first video unit it fetches ends the export ("missed a decode deadline") #5052

Description

@t0ms

On main at 82c3f2fe4, when moq export ts has its subscriptions ended and re-requests them onto a route that is already live, it fetches that route's in-progress video group, drops a few late audio and data frames as designed (frame missed its deadline; dropped), and then exits on Error: MPEG-TS output missed a decode deadline on PID 111 at 9945951 b/s; raise the delay or the multiplex rate. Raising the delay doesn't help: at --delay 1s, which carries this clip through a 540 s run in steady state, both rigs below fail the same way (2 of 2 and 4 of 4 runs), and on the same rigs every path that does not re-request onto a live route survives.

Rigs that reach it

Both on one host, loopback, one moq-relay, CNNiEMEA2.ts (H.264, MPEG audio, AC-3, teletext, SCTE-35, DVB SI, about 10 Mb/s) paced through tsp -P regulate --pcr-synchronous into moq import ts; default export options (--delay 500ms) except as stated.

  1. A crashed publisher replaced, default protocol. export ts --linger 60s. The publisher is SIGKILLed 10 s into each 15 s session, and a new moq import ts starts under the same name 3 s later. On moq-lite-06 there is no epoch on the wire, so the exporter's subscriptions end (internal error) only when the dead session times out, 9.9–12.9 s after the kill. It re-subscribes every track within about 2 ms and fetches the replacement's in-progress 0.avc3 group. In 4 of 4 runs the export then exited on PID 111, 4 ms to 3.3 s after a re-request: at the first re-request in one run, at the second in three, and in those three it wrote nothing between the two. --linger then waits out 60 s for a broadcast that is live, and the process exits 1.

    With MOQ_CONNECT_VERSION / MOQ_LISTEN_VERSION = moq-lite-07-wip on every process the same rig passes, 8 of 8 resumes, exit 0: the replacement's newer epoch ends the old broadcast as it announces, and the exporter resumes without re-requesting onto a live route. So relay: a publisher replacing a killed one is sent the old session's subscriptions before it has published, and moq import ts exits with "rendition is not published" #4945's rig is fixed where the epoch is carried, and this is what is left of it on the default protocol.

  2. A 1+1 pair failing over. Two moq import ts of one forked feed under one name, sharing an --epoch, started 2 s apart, with two exporters subscribed before the second publisher started; every client at --quic-idle-timeout 6s --quic-keep-alive 2s; the serving publisher is SIGKILLed. On moq-lite-06 the subscriptions end with internal error; on moq-lite-07-wip most of them end with not found. Either way the exporter re-subscribes, fetches the other publisher's in-progress video group, and exits on PID 111 within milliseconds: 5 of 5 runs (2 on lite-06, 3 on lite-07). Where moq-lite-07 moves the subscription itself (exporters that subscribed after both publishers were up) there is no re-request, and the export survives the failover unless the two publishers' unaligned group numbering rewinds it (export ts exits with "frame timestamp is below the live edge" when the relay switches to a same-hop standby quickly #4354): 4 of 6 runs.

Rig 2 on moq-lite-06, one exporter (the other is identical):

09:55:30.136114Z  WARN moq_net::lite::subscriber: subscribe error broadcast=rec.hang track=3.ts err=internal error
                  (every track within 1 ms, then each re-subscribed)
09:55:30.141438Z  INFO moq_net::lite::subscriber: fetch started broadcast=rec.hang track=0.avc3 group=13
09:55:30.143087Z  WARN moq_mux::container::ts::export: frame missed its deadline; dropped track=2.ts dropped=1
09:55:30.144657Z  WARN moq_mux::container::ts::export: frame missed its deadline; dropped track=4.ts dropped=2
Error: MPEG-TS output missed a decode deadline on PID 111 at 9945951 b/s; raise the delay or the multiplex rate

What I haven't located

Why a video unit the jitter buffer accepts then misses its deadline in the schedule. jitter::Buffer::queue keeps any frame with arrived <= deadline and drops the rest as late; nothing there asks whether the unit's own packets still fit before that deadline at the multiplex rate. A frame read in the fetch's burst can pass with less slack than that, and Schedule::next then finds it unsent at its deadline and bails, which is right in steady state, where it means the delay or rate is too small. That would explain why it is a video unit that ends it (the large units arriving with the least slack) and why it follows a re-request (the backlog of the in-progress group), but it is reasoned from the code and the logs, not traced. I haven't explained the silence between the two re-requests in rig 1 either. In one rig-2 run both exporters also exited this way 1.4 s after joining mid-GOP, before any failover, which points at the backlog burst rather than at the failover itself.

I can share the relay, publisher and exporter logs for every run.

Activity

  1. t0ms commented on Oct 9, 2026

    @t0ms
    ContributorAuthor

    Located, on main at f8215bc47, by adding logging around the jitter buffer and the schedule and re-running rig 1 in two variants: the replacement replaying the clip from its start, and the replacement starting 15 s further into it. Both exit as before, at the same output length as the uninstrumented runs (61,667 packets).

    The unit that misses was read before the publisher died. A video frame is stamped arrived (when its track was last found empty) and read as it comes off the track (export.rs#L1084), then waits in the track's DecodeClock until a frame with a high enough PTS settles its DTS (L267, pop). When the publisher is killed, the last frames read are still waiting there. On moq-lite-06 the re-request happens inside moq-net, so the track neither ends nor restarts (restarts() and skips() stay 0) and nothing flushes the hold. The frames sit through the whole silence, and leave only when the replacement's frames climb past their PTS, still carrying their old stamps:

    Variant Frames stranded Read this long before release Past their deadline at release
    replay, at the second re-request 13 26.8 s 25.5 s
    continue, at the first re-request 20 13.2 s 11.9 s

    jitter::Buffer::queue keeps a frame when arrival.arrived <= deadline (jitter.rs#L477), and these did arrive before their deadlines, so every one is queued.

    The schedule had already gone past them. It had laid slots up to 1025452 when it drained and stopped (next = 1025453), and the stranded units are due in slots 1025444–1025454. The eleven due before next count as late and go at once, 1,415 packets between them. The one due exactly at next is not late, has no slot of room, and sits behind them in a slot that holds 165 packets, so missed() finds it unsent and the export bails with "raise the delay or the multiplex rate". Neither would reach it: the frames are late by the length of the silence.

    15:51:22.228301Z dbg5052 schedule push pid=111 due=1025443 next=Some(1025423) room=Some(20) packets=140
                     (session 0 killed at 15:51:21.41; nothing on video for 25.5 s)
    15:51:44.525402Z fetch started broadcast=lin-86107.hang track=0.avc3 group=7
    15:51:47.768889Z dbg5052 video push track=0.avc3 decode_ms=25636116 arrived_ago_ms=26823 read_ago_ms=26823 slack_ms=Some(-25522)
                     (12 more, the same)
    15:51:47.768999Z dbg5052 schedule push pid=111 due=1025444 next=Some(1025453) room=Some(-9) packets=105 queued=0
                     (…)
    15:51:47.769159Z dbg5052 schedule push pid=111 due=1025453 next=Some(1025453) room=Some(0) packets=53 queued=11
    15:51:47.769183Z dbg5052 missed index=1025453 pid=111 due=1025453 start=1025453 sent=0 count=53 allowed=165 queued=13
    

    This also accounts for the rest of rig 1:

    • The silence between the two re-requests. The re-request fetches the group the export was on (group=7 here). A replacement replaying from the clip's start reaches that group about 10 s in, so a session killed at 10 s never delivers it, and nothing flows until the next session does.
    • Why a clean end survives. A finished track flushes its hold (L1112), so nothing is stranded.
    • Why moq-lite-07 survives. The newer epoch takes resume(), which resets every track's DecodeClock, the jitter buffer and the schedule (L1439–L1450).

    Rig 2 fits the same path: the exporter holds the dead publisher's last frames through the 6 s idle timeout, and the standby's frames, on the same timeline and further on, settle them at once, hence "within milliseconds". The one rig-2 run that exited 1.4 s after joining, before any failover, is not explained by this; there was no silence to strand anything.

    Two places it could be fixed, either of which stops this exit:

    1. Judge a held frame when it leaves the hold. The earliest a held frame can go out is when the frame that settled it arrived. Giving it that frame's arrived would leave steady state unchanged, since the settling frame comes within the reorder lookahead and so inside the delay, and would drop stranded frames as late. Flushing the hold when a subscription is re-requested would also do it, but the export cannot see that on moq-lite-06 today.
    2. A unit pushed at or behind the schedule's next. It cannot be on time. Counting it late, or dropping it, instead of bailing would keep the export up through any other path that delivers a stale unit, and leave the bail for the case its message describes.

    I can share the logging patch and both runs' logs.

  2. added
    questBeing tracked/planned in a quest. See `quest/`
    on Oct 9, 2026
  3. kixelated commented on Oct 9, 2026

    @kixelated
    Collaborator

    Thanks, the trace pinned it down. The root cause is upstream of the exporter.

    On lite-06, the relay ends the downstream copy with internal error when the dead publisher's session times out. The downstream front treats that as a normal failover (Front::track_ended's Err(_) if delivered arm in rs/moq-net/src/model/front.rs) and re-subscribes through the same relay session about 2 ms later. The relay resolves that epochless SUBSCRIBE to whatever wins the path now, the replacement, and the resume layer fetches the rest of the in-flight group from it. So moq-net stitches two epochless instances across a hop, against the rule that routes without an epoch never splice. The stranded decode-hold frames and the bail are the export's symptom. #5087 does not change this path.

    Decision (planned in #5108):

    • quest/m0/broadcast-epoch/no-stitch.md: a front re-splices after an error only when it has an epoch, and the re-request carries that epoch. An epochless track ends with the error, and consumers recover on the announce Restart (or END then START). On lite-06, any upstream track error then ends the track instead of healing.
    • No exporter-side guard. Once the stitch is gone, a stale unit reaching the schedule is a bug to fail loudly on.
    • quest/m0/broadcast-epoch/export-ts.md: export ts --linger waits only for the same epoch, and a replacement ends the export unless --stitch asks for a fresh ts::Export with a full program switch. resume() is deleted.

    The rig-2 run that exited 1.4 s after a mid-GOP join, before any failover, is not explained by this. Once no-stitch lands, could you re-run both rigs? If that exit reproduces, please file it separately.

    Anyone who wants to donate tokens can run /quest-start quest/m0/broadcast-epoch/no-stitch once #5108 merges.

    (Written by Claude Opus 5.5)

  4. added a commit that references this issue on Oct 9, 2026
  5. kixelated commented on Oct 9, 2026

    @kixelated
    Collaborator

    #5108 merged. quest/m0/broadcast-epoch/no-stitch.md tracks this issue and closes it when it lands, and quest/m0/broadcast-epoch/export-ts.md covers the exporter side. #5087 has also merged, so both quests are ready now, and anyone who wants to donate tokens can run /quest-start quest/m0/broadcast-epoch/no-stitch.

    (Written by Claude Opus 5.5)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    questBeing tracked/planned in a quest. See `quest/`

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions