Skip to content

export ts exits with "frame timestamp is below the live edge" when the relay switches to a same-hop standby quickly #4354

Description

@t0ms

What happens

One relay, two moq import ts publishers of one broadcast with the same --hop 42, both fed from one
live stream (tsp … -P regulate --pcr-synchronous, forked to two UDP ports), and two moq export ts
subscribers attached once both publishers are up. At t=15 s the serving publisher is stopped.

Now that #4287 closes the session on a signal, the relay switches to the standby at the signal
rather than after the idle timeout; the standby's media subscriptions start in the same second. That is
the behaviour we want from a 1+1 pair. But in the same second both exporters exit:

Error: frame timestamp is below the live edge
How the serving publisher stops Publishers started Runs Exporters
SIGINT 2 s apart 1 both exit, rewind
SIGTERM 2 s apart 1 both exit, rewind
end of its input (stdin EOF) 2 s apart 1 both exit, rewind
SIGINT same instant 1 both exit, rewind
SIGTERM same instant 1 both exit, rewind
end of its input same instant 1 both exit, rewind
SIGKILL same instant 1 both exit, rewind, on the switch after detection
SIGKILL 2 s apart 2 both continue, 3 and 7 s stall

Before #4287 the same arms on ffa5b81b behaved differently: SIGTERM was a hard kill, and at stdin EOF
the relay passed the completion on without switching and the exporters ended on the track layout
changing. SIGINT rewound there too.

What the logs show at the switch

The exporter returns TimestampRewind from poll_malformed (rs/moq-mux/src/container/consumer.rs):
a group after the last one it read has a minimum timestamp below the live edge it had reached.

In most runs the relay first tries to resume the group in flight on the standby, cannot, and moves on:

WARN moq_relay::relay: connection closed err=no error                                      (serving publisher closed)
WARN moq_net::model::resume: no route can serve the rest of this group group=12 frame=48 err=not found

That line is not the cause on its own. It also appears in both SIGKILL runs that survive, where the
switch comes several seconds later. And two of the rewinding runs, the co-started end-of-input and
SIGKILL runs, do not log it at all.

What the exporter is handed at the switch

To see which group trips the check, I rebuilt moq at 2b689c24 with three log lines in
container/consumer.rs: one per group arriving in poll_read_finish, and one at each
TimestampRewind return, printing the group, its minimum timestamp, the previous group and the live
edge. Seven more arms, same rig:

How the serving publisher stops Publishers started Track Standby's group Last group read Standby's first timestamp vs live edge
SIGINT 2 s apart 1.ts 72 54 0.17 s behind
SIGTERM 2 s apart 6.mp2 878 673 4.15 s behind
end of input 2 s apart 5.ts 534 471 4.05 s behind
end of input same instant 1.ts 65 45 5.61 s behind
SIGKILL same instant 6.mp2, 2.ts (one per exporter) 1049, 23 740, 18 1.00 s, 1.22 s behind

All five are the across-groups check in poll_malformed; none is the within-group one. The other two
arms survived: the SIGKILL with the publishers 2 s apart switched after a 7 s stall and continued,
and in the co-started SIGINT arm the signalled publisher turned out not to be the one serving, so
nothing switched.

So at the switch, the subscriber is handed the standby's groups under higher sequence numbers
than the last one it read, while their media starts 0.17–5.6 s before what it has already
written. That happens with the publishers started together too: each importer numbers groups from
its own counter, and in these runs the standby's first group was 5 to 309 groups ahead of the last
one read, on older media. Why the counters are that far apart when both start together, and why the relay hands over media from before the live
edge rather than from the standby's newest group, I have not established. The switch that survived
is consistent with it, though: after a 7 s stall the standby's media is past the old edge again.

The converse shows up in a two-relay mesh on ffa5b81b. There the standby is numbered behind the
active publisher, and the splice waits until the standby's numbering reaches the last group delivered.
The standby's lag then adds to the outage (kill + 23 s against kill + 10 s, at a 6 s idle timeout).
Both look like the same thing from the subscriber's side: group sequence is compared across two
publishers whose sequences are not comparable.

The ask, whichever the mechanism

For a TS output, a group whose first timestamps sit behind the live edge after a source switch is an
overlap, not a malformed stream. The exporter ending the process on it turns a working failover into an
outage. Skipping or trimming the overlap, and marking a discontinuity in the output, would keep the
subscriber up. The other half is on the relay side: splicing two publishers of one broadcast by
group sequence only works if their sequences line up, and here they don't, even when the publishers
start together. Whether that should be fixed by resuming on media time, or by the pair agreeing on a
numbering, I'd defer to you.

Reproducer

t6-relay-kill.sh,
one Linux host over loopback:

MODE=standby HOP=42 LATE_SUBS=1 KILL_WHICH=newest SIG=INT STAGGER=2 PORT=4470 KILL=15 END=60 \
  t6-relay-kill.sh <dir with moq and moq-relay> <out-dir>

With a shared hop the newest publisher serves, so KILL_WHICH=newest signals the serving one.
SIG=TERM and SIG=EOF give the other rows, and STAGGER=0 starts the publishers together; then
which one counts as newest is a race, and a run can signal the idle one. The three diagnostic log
lines are a local patch; happy to paste it if it helps.

Activity

  1. t0ms commented on Sep 28, 2026

    @t0ms
    ContributorAuthor

    The mesh case from the report behaves differently on main. Same
    rig
    and arm matrix, 2b689c24, two meshed relays, shared --hop, 6 s idle timeout:

    • relay A still resumes at the dead publisher's last video group (no route can serve the rest of this group group=21 frame=30), as on ffa5b81b.
    • On ffa5b81b that resume worked once the standby's numbering caught up. On main, the moment
      the standby's group 21 arrives, every exporter on relay A exits with frame timestamp is below the live edge: 4 of 4 hard kills and 4 of 4 clean ends where the standby had to catch up.
    • The two runs where the standby was already past group 21 at detection resumed fine: a standby
      that joined at t=0 (group 23 at the switch), and the 30 s default timeout (group 39).

    So in the mesh the lag is no longer just extra outage, it ends the exporter. I haven't run the
    diagnostic build in the mesh, so I'm assuming from the message that it's the same across-groups
    check as the single-relay switches in the report. Results are in
    T6 § Mesh source failover.

  2. added
    questBeing tracked/planned in a quest. See `quest/`
    on Sep 28, 2026
  3. t0ms commented on Oct 8, 2026

    @t0ms
    ContributorAuthor

    Re-measured on main at 82c3f2fe4, where --epoch replaces --hop, as a before-measurement for quest/m1/hop-aligned-import.md. Same single-relay rig: two moq import ts of one forked feed under one name with one shared UUIDv7 --epoch, started 2 s apart, two export ts, 6 s idle timeout, and a hard kill of the serving publisher. Every process on moq-lite-07-wip, since moq-lite-06 doesn't carry the epoch and the same setup there fails over by re-request.

    So the rewind is still there on main, now reached through the epoch resume rather than through reselection, and only in the direction where the standby's counter is ahead. The quest's end-to-end test would need moq-lite-07 on every process to exercise the resume at all. I can share the logs for every run.

  4. t0ms commented on Oct 19, 2026

    @t0ms
    ContributorAuthor

    A correction to my comment above. That rig's standby didn't read the live stream: its tsp -P fork copy started at the clip's first packet, about 2 s behind live, and dropped 107–186 datagrams a run while catching up, so the two importers' groups differed by more than their start offsets, and its 2 of 3 and 3 of 3 were the rig's.

    Re-run with both publishers reading one received copy, byte-identical over their overlap: the rewind still reproduces. On main, a clean exit of the serving publisher ended both exporters in all four counted runs on the TS lane (three on frame timestamp … below the previous group's start), and on --passthrough ended both in three runs and one in the fourth, on that rewind. The rig and the per-build table are in #5895's description, which removes it.

    (Written by Claude Opus 5.5)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    questBeing tracked/planned in a quest. See `quest/`

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions