Description
On main at 82c3f2fe4, when moq export ts has its subscriptions ended and re-requests them onto a route that is already live, it fetches that route's in-progress video group, drops a few late audio and data frames as designed (frame missed its deadline; dropped), and then exits on Error: MPEG-TS output missed a decode deadline on PID 111 at 9945951 b/s; raise the delay or the multiplex rate. Raising the delay doesn't help: at --delay 1s, which carries this clip through a 540 s run in steady state, both rigs below fail the same way (2 of 2 and 4 of 4 runs), and on the same rigs every path that does not re-request onto a live route survives.
Rigs that reach it
Both on one host, loopback, one moq-relay, CNNiEMEA2.ts (H.264, MPEG audio, AC-3, teletext, SCTE-35, DVB SI, about 10 Mb/s) paced through tsp -P regulate --pcr-synchronous into moq import ts; default export options (--delay 500ms) except as stated.
-
A crashed publisher replaced, default protocol.
export ts --linger 60s. The publisher isSIGKILLed 10 s into each 15 s session, and a newmoq import tsstarts under the same name 3 s later. Onmoq-lite-06there is no epoch on the wire, so the exporter's subscriptions end (internal error) only when the dead session times out, 9.9–12.9 s after the kill. It re-subscribes every track within about 2 ms and fetches the replacement's in-progress0.avc3group. In 4 of 4 runs the export then exited on PID 111, 4 ms to 3.3 s after a re-request: at the first re-request in one run, at the second in three, and in those three it wrote nothing between the two.--lingerthen waits out 60 s for a broadcast that is live, and the process exits 1.With
MOQ_CONNECT_VERSION/MOQ_LISTEN_VERSION=moq-lite-07-wipon every process the same rig passes, 8 of 8 resumes, exit 0: the replacement's newer epoch ends the old broadcast as it announces, and the exporter resumes without re-requesting onto a live route. So relay: a publisher replacing a killed one is sent the old session's subscriptions before it has published, andmoq import tsexits with "rendition is not published" #4945's rig is fixed where the epoch is carried, and this is what is left of it on the default protocol. -
A 1+1 pair failing over. Two
moq import tsof one forked feed under one name, sharing an--epoch, started 2 s apart, with two exporters subscribed before the second publisher started; every client at--quic-idle-timeout 6s --quic-keep-alive 2s; the serving publisher isSIGKILLed. Onmoq-lite-06the subscriptions end withinternal error; onmoq-lite-07-wipmost of them end withnot found. Either way the exporter re-subscribes, fetches the other publisher's in-progress video group, and exits on PID 111 within milliseconds: 5 of 5 runs (2 on lite-06, 3 on lite-07). Wheremoq-lite-07moves the subscription itself (exporters that subscribed after both publishers were up) there is no re-request, and the export survives the failover unless the two publishers' unaligned group numbering rewinds it (export tsexits with "frame timestamp is below the live edge" when the relay switches to a same-hop standby quickly #4354): 4 of 6 runs.
Rig 2 on moq-lite-06, one exporter (the other is identical):
09:55:30.136114Z WARN moq_net::lite::subscriber: subscribe error broadcast=rec.hang track=3.ts err=internal error
(every track within 1 ms, then each re-subscribed)
09:55:30.141438Z INFO moq_net::lite::subscriber: fetch started broadcast=rec.hang track=0.avc3 group=13
09:55:30.143087Z WARN moq_mux::container::ts::export: frame missed its deadline; dropped track=2.ts dropped=1
09:55:30.144657Z WARN moq_mux::container::ts::export: frame missed its deadline; dropped track=4.ts dropped=2
Error: MPEG-TS output missed a decode deadline on PID 111 at 9945951 b/s; raise the delay or the multiplex rate
What I haven't located
Why a video unit the jitter buffer accepts then misses its deadline in the schedule. jitter::Buffer::queue keeps any frame with arrived <= deadline and drops the rest as late; nothing there asks whether the unit's own packets still fit before that deadline at the multiplex rate. A frame read in the fetch's burst can pass with less slack than that, and Schedule::next then finds it unsent at its deadline and bails, which is right in steady state, where it means the delay or rate is too small. That would explain why it is a video unit that ends it (the large units arriving with the least slack) and why it follows a re-request (the backlog of the in-progress group), but it is reasoned from the code and the logs, not traced. I haven't explained the silence between the two re-requests in rig 1 either. In one rig-2 run both exporters also exited this way 1.4 s after joining mid-GOP, before any failover, which points at the backlog burst rather than at the failover itself.
I can share the relay, publisher and exporter logs for every run.
Activity
t0ms commentedon Oct 9, 2026
Located, on
mainatf8215bc47, by adding logging around the jitter buffer and the schedule and re-running rig 1 in two variants: the replacement replaying the clip from its start, and the replacement starting 15 s further into it. Both exit as before, at the same output length as the uninstrumented runs (61,667 packets).The unit that misses was read before the publisher died. A video frame is stamped
arrived(when its track was last found empty) andreadas it comes off the track (export.rs#L1084), then waits in the track'sDecodeClockuntil a frame with a high enough PTS settles its DTS (L267,pop). When the publisher is killed, the last frames read are still waiting there. Onmoq-lite-06the re-request happens insidemoq-net, so the track neither ends nor restarts (restarts()andskips()stay 0) and nothing flushes the hold. The frames sit through the whole silence, and leave only when the replacement's frames climb past their PTS, still carrying their old stamps:jitter::Buffer::queuekeeps a frame whenarrival.arrived <= deadline(jitter.rs#L477), and these did arrive before their deadlines, so every one is queued.The schedule had already gone past them. It had laid slots up to 1025452 when it drained and stopped (
next= 1025453), and the stranded units are due in slots 1025444–1025454. The eleven due beforenextcount as late and go at once, 1,415 packets between them. The one due exactly atnextis not late, has no slot of room, and sits behind them in a slot that holds 165 packets, somissed()finds it unsent and the export bails with "raise the delay or the multiplex rate". Neither would reach it: the frames are late by the length of the silence.This also accounts for the rest of rig 1:
group=7here). A replacement replaying from the clip's start reaches that group about 10 s in, so a session killed at 10 s never delivers it, and nothing flows until the next session does.moq-lite-07survives. The newer epoch takesresume(), which resets every track'sDecodeClock, the jitter buffer and the schedule (L1439–L1450).Rig 2 fits the same path: the exporter holds the dead publisher's last frames through the 6 s idle timeout, and the standby's frames, on the same timeline and further on, settle them at once, hence "within milliseconds". The one rig-2 run that exited 1.4 s after joining, before any failover, is not explained by this; there was no silence to strand anything.
Two places it could be fixed, either of which stops this exit:
arrivedwould leave steady state unchanged, since the settling frame comes within the reorder lookahead and so inside the delay, and would drop stranded frames as late. Flushing the hold when a subscription is re-requested would also do it, but the export cannot see that onmoq-lite-06today.next. It cannot be on time. Counting it late, or dropping it, instead of bailing would keep the export up through any other path that delivers a stale unit, and leave the bail for the case its message describes.I can share the logging patch and both runs' logs.
kixelated commentedon Oct 9, 2026
Thanks, the trace pinned it down. The root cause is upstream of the exporter.
On lite-06, the relay ends the downstream copy with
internal errorwhen the dead publisher's session times out. The downstream front treats that as a normal failover (Front::track_ended'sErr(_) if deliveredarm inrs/moq-net/src/model/front.rs) and re-subscribes through the same relay session about 2 ms later. The relay resolves that epochless SUBSCRIBE to whatever wins the path now, the replacement, and the resume layer fetches the rest of the in-flight group from it. So moq-net stitches two epochless instances across a hop, against the rule that routes without an epoch never splice. The stranded decode-hold frames and the bail are the export's symptom. #5087 does not change this path.Decision (planned in #5108):
quest/m0/broadcast-epoch/no-stitch.md: a front re-splices after an error only when it has an epoch, and the re-request carries that epoch. An epochless track ends with the error, and consumers recover on the announceRestart(or END then START). On lite-06, any upstream track error then ends the track instead of healing.quest/m0/broadcast-epoch/export-ts.md:export ts --lingerwaits only for the same epoch, and a replacement ends the export unless--stitchasks for a freshts::Exportwith a full program switch.resume()is deleted.The rig-2 run that exited 1.4 s after a mid-GOP join, before any failover, is not explained by this. Once no-stitch lands, could you re-run both rigs? If that exit reproduces, please file it separately.
Anyone who wants to donate tokens can run
/quest-start quest/m0/broadcast-epoch/no-stitchonce #5108 merges.(Written by Claude Opus 5.5)
quest: import #5052 and #5101 as no-stitch and fresh TS export restar…
kixelated commentedon Oct 9, 2026
#5108 merged.
quest/m0/broadcast-epoch/no-stitch.mdtracks this issue and closes it when it lands, andquest/m0/broadcast-epoch/export-ts.mdcovers the exporter side. #5087 has also merged, so both quests are ready now, and anyone who wants to donate tokens can run/quest-start quest/m0/broadcast-epoch/no-stitch.(Written by Claude Opus 5.5)
fix(net): an epochless track ends on an upstream error instead of re-…