Repository navigation
Description
What happens
One relay, two moq import ts publishers of one broadcast with the same --hop 42, both fed from one
live stream (tsp … -P regulate --pcr-synchronous, forked to two UDP ports), and two moq export ts
subscribers attached once both publishers are up. At t=15 s the serving publisher is stopped.
Now that #4287 closes the session on a signal, the relay switches to the standby at the signal
rather than after the idle timeout; the standby's media subscriptions start in the same second. That is
the behaviour we want from a 1+1 pair. But in the same second both exporters exit:
Error: frame timestamp is below the live edge
| How the serving publisher stops | Publishers started | Runs | Exporters |
|---|---|---|---|
| SIGINT | 2 s apart | 1 | both exit, rewind |
| SIGTERM | 2 s apart | 1 | both exit, rewind |
| end of its input (stdin EOF) | 2 s apart | 1 | both exit, rewind |
| SIGINT | same instant | 1 | both exit, rewind |
| SIGTERM | same instant | 1 | both exit, rewind |
| end of its input | same instant | 1 | both exit, rewind |
| SIGKILL | same instant | 1 | both exit, rewind, on the switch after detection |
| SIGKILL | 2 s apart | 2 | both continue, 3 and 7 s stall |
Before #4287 the same arms on ffa5b81b behaved differently: SIGTERM was a hard kill, and at stdin EOF
the relay passed the completion on without switching and the exporters ended on the track layout
changing. SIGINT rewound there too.
What the logs show at the switch
The exporter returns TimestampRewind from poll_malformed (rs/moq-mux/src/container/consumer.rs):
a group after the last one it read has a minimum timestamp below the live edge it had reached.
In most runs the relay first tries to resume the group in flight on the standby, cannot, and moves on:
WARN moq_relay::relay: connection closed err=no error (serving publisher closed)
WARN moq_net::model::resume: no route can serve the rest of this group group=12 frame=48 err=not found
That line is not the cause on its own. It also appears in both SIGKILL runs that survive, where the
switch comes several seconds later. And two of the rewinding runs, the co-started end-of-input and
SIGKILL runs, do not log it at all.
What the exporter is handed at the switch
To see which group trips the check, I rebuilt moq at 2b689c24 with three log lines in
container/consumer.rs: one per group arriving in poll_read_finish, and one at each
TimestampRewind return, printing the group, its minimum timestamp, the previous group and the live
edge. Seven more arms, same rig:
| How the serving publisher stops | Publishers started | Track | Standby's group | Last group read | Standby's first timestamp vs live edge |
|---|---|---|---|---|---|
| SIGINT | 2 s apart | 1.ts |
72 | 54 | 0.17 s behind |
| SIGTERM | 2 s apart | 6.mp2 |
878 | 673 | 4.15 s behind |
| end of input | 2 s apart | 5.ts |
534 | 471 | 4.05 s behind |
| end of input | same instant | 1.ts |
65 | 45 | 5.61 s behind |
| SIGKILL | same instant | 6.mp2, 2.ts (one per exporter) |
1049, 23 | 740, 18 | 1.00 s, 1.22 s behind |
All five are the across-groups check in poll_malformed; none is the within-group one. The other two
arms survived: the SIGKILL with the publishers 2 s apart switched after a 7 s stall and continued,
and in the co-started SIGINT arm the signalled publisher turned out not to be the one serving, so
nothing switched.
So at the switch, the subscriber is handed the standby's groups under higher sequence numbers
than the last one it read, while their media starts 0.17–5.6 s before what it has already
written. That happens with the publishers started together too: each importer numbers groups from
its own counter, and in these runs the standby's first group was 5 to 309 groups ahead of the last
one read, on older media. Why the counters are that far apart when both start together, and why the relay hands over media from before the live
edge rather than from the standby's newest group, I have not established. The switch that survived
is consistent with it, though: after a 7 s stall the standby's media is past the old edge again.
The converse shows up in a two-relay mesh on ffa5b81b. There the standby is numbered behind the
active publisher, and the splice waits until the standby's numbering reaches the last group delivered.
The standby's lag then adds to the outage (kill + 23 s against kill + 10 s, at a 6 s idle timeout).
Both look like the same thing from the subscriber's side: group sequence is compared across two
publishers whose sequences are not comparable.
The ask, whichever the mechanism
For a TS output, a group whose first timestamps sit behind the live edge after a source switch is an
overlap, not a malformed stream. The exporter ending the process on it turns a working failover into an
outage. Skipping or trimming the overlap, and marking a discontinuity in the output, would keep the
subscriber up. The other half is on the relay side: splicing two publishers of one broadcast by
group sequence only works if their sequences line up, and here they don't, even when the publishers
start together. Whether that should be fixed by resuming on media time, or by the pair agreeing on a
numbering, I'd defer to you.
Reproducer
t6-relay-kill.sh,
one Linux host over loopback:
MODE=standby HOP=42 LATE_SUBS=1 KILL_WHICH=newest SIG=INT STAGGER=2 PORT=4470 KILL=15 END=60 \
t6-relay-kill.sh <dir with moq and moq-relay> <out-dir>With a shared hop the newest publisher serves, so KILL_WHICH=newest signals the serving one.
SIG=TERM and SIG=EOF give the other rows, and STAGGER=0 starts the publishers together; then
which one counts as newest is a race, and a run can signal the idle one. The three diagnostic log
lines are a local patch; happy to paste it if it helps.
Activity
t0ms commentedon Sep 28, 2026
The mesh case from the report behaves differently on
main. Samerig
and arm matrix,
2b689c24, two meshed relays, shared--hop, 6 s idle timeout:no route can serve the rest of this group group=21 frame=30), as onffa5b81b.ffa5b81bthat resume worked once the standby's numbering caught up. Onmain, the momentthe standby's group 21 arrives, every exporter on relay A exits with
frame timestamp is below the live edge: 4 of 4 hard kills and 4 of 4 clean ends where the standby had to catch up.that joined at t=0 (group 23 at the switch), and the 30 s default timeout (group 39).
So in the mesh the lag is no longer just extra outage, it ends the exporter. I haven't run the
diagnostic build in the mesh, so I'm assuming from the message that it's the same across-groups
check as the single-relay switches in the report. Results are in
T6 § Mesh source failover.
t0ms commentedon Oct 8, 2026
Re-measured on
mainat82c3f2fe4, where--epochreplaces--hop, as a before-measurement forquest/m1/hop-aligned-import.md. Same single-relay rig: twomoq import tsof one forked feed under one name with one shared UUIDv7--epoch, started 2 s apart, twoexport ts, 6 s idle timeout, and a hard kill of the serving publisher. Every process onmoq-lite-07-wip, sincemoq-lite-06doesn't carry the epoch and the same setup there fails over by re-request.frame timestamp 25642876266 µs is below the previous group's start 25647116266 µs; the third survived with no stall, for a reason I haven't found.not foundat the failover instead of resuming (3 of 3); the exporter then re-requests and dies as in export ts: after re-requesting onto a live route, the first video unit it fetches ends the export ("missed a decode deadline") #5052. That fits the standby's refusals at its arrival (A--hopstandby that connects to the relay its subscribers are on ends every one of them withnot found#4352) being kept against those subscriptions, which I haven't confirmed in code.--hopstandby that connects to the relay its subscribers are on ends every one of them withnot found#4352's teardown at arrival is gone with a shared epoch.So the rewind is still there on
main, now reached through the epoch resume rather than through reselection, and only in the direction where the standby's counter is ahead. The quest's end-to-end test would needmoq-lite-07on every process to exercise the resume at all. I can share the logs for every run.feat(mux): number same-epoch groups on the catalog clock's grid
feat(mux): number same-epoch groups on the catalog clock's grid
feat(mux): number same-epoch groups on the catalog clock's grid
t0ms commentedon Oct 19, 2026
A correction to my comment above. That rig's standby didn't read the live stream: its
tsp -P forkcopy started at the clip's first packet, about 2 s behind live, and dropped 107–186 datagrams a run while catching up, so the two importers' groups differed by more than their start offsets, and its 2 of 3 and 3 of 3 were the rig's.Re-run with both publishers reading one received copy, byte-identical over their overlap: the rewind still reproduces. On
main, a clean exit of the serving publisher ended both exporters in all four counted runs on the TS lane (three on frame timestamp … below the previous group's start), and on--passthroughended both in three runs and one in the fourth, on that rewind. The rig and the per-build table are in #5895's description, which removes it.(Written by Claude Opus 5.5)