Repository navigation
Description
I'm building a multiplayer collaborative app on MoQ. Many participants each publish a small control track (about 20 to 60 tiny groups/s in bursts, one group per message), and one server-side participant subscribes to all of them and fans state back out. While load-testing on cdn.moq.pro I hit a pattern that only appears when the publishers and the subscriber land on different moq.pro nodes.
Setup (2026-09-26, 10:38 to 12:06 UTC): the server-side participant resolves to an OVH node about 0.7 ms away; 10 publishing clients about 40 ms away resolve to a Linode node, sometimes an OVH one. Clients: moq-net 0.3.5 / moq-tokio 0.19.17 (native, all traces below) and @moq/net 0.4.1 (browsers, separate runs), moq-lite-06, JWT with publish/subscribe patterns. Each client join opens a new session and publishes <room>/control/<client>/<session> (track control) plus an audio broadcast; the server-side participant subscribes to every announced client broadcast and publishes state and log tracks back. It FETCHes a missing control group when a later one arrives first and gives up after 2 s.
Cross-node only:
- Some control groups never arrive, and FETCH for them does not return within 2 s while the publisher is still connected (109 of 142 lost messages in one run). When the publisher then goes idle, nothing reveals the gap at all.
- A subscription opened seconds before a publisher's first burst sometimes delivers only the tail of the burst; the first groups need FETCH, which sometimes fails too.
- The
stateandlogtracks stall for tens of seconds (up to 31 s) while every client stays connected; twice no client received a firststatewithin 30 s. - Closed sessions stay announced for tens of seconds to minutes, or are announced again after being retracted (up to 229 s after the publisher closed the session cleanly). Subscribing to such a path answers
unroutableorinternal error. In these runs every one of those answers came after the publisher had closed the session, so the answers are right and the announcements are stale. - A re-created broadcast reaches a subscriber as
Updatedinstead ofAnnounced, because the subscriber never saw the old one retracted (46 to 910 times per run cross-node, never on one node).
Same workload three ways (contend phase, reopen off): different nodes (GeoDNS): 123 of 2,273 messages lost, echo p99 23 s. Everyone pinned to one node: 0 of 6,310 lost, echo p99 91 ms. Self-hosted moq-relay 0.15.6, single node: 0 of 6,310 lost, echo p99 73 ms. Echo is the time from a client sending a message to that client seeing it in state.
| Setup | Client reopen | contend |
join-leave |
control FETCHes failed |
|---|---|---|---|---|
| Different nodes (GeoDNS) | off | FAIL: 2273 sent, 123 lost, echo p99 23322 ms | FAIL: 395 sent, 19 lost, echo p99 25318 ms | 153, 11 |
| Different nodes (clients pinned to the other node) | on | FAIL: 2536 sent, 9 lost, echo p99 30775 ms | FAIL: 383 sent, 14 lost, echo p99 5270 ms | 8, 7 |
| Different nodes (clients pinned to the other node) | off | FAIL after 51 s: 1421 sent, 71 lost, echo p99 2012 ms | no client got a first state within 30 s |
70 |
| Different nodes (GeoDNS) | on | no client got a first state within 30 s |
pass: 519 sent, 0 lost, echo p99 87 ms | 0 |
| Same node (both pinned) | off | pass: 6310 sent, 0 lost, echo p99 91 ms | pass: 536 sent, 0 lost, echo p99 142 ms | 0, 0 |
| Same node (both pinned) | on | pass: 6310 sent, 0 lost, echo p99 97 ms | pass: 527 sent, 0 lost, echo p99 142 ms | 0, 0 |
| Self-hosted moq-relay 0.15.6 | off | pass: 6310 sent, 0 lost, echo p99 73 ms | pass: 535 sent, 0 lost, echo p99 73 ms | 0, 0 |
| Self-hosted moq-relay 0.15.6 | on | pass: 6310 sent, 0 lost, echo p99 76 ms | pass: 506 sent, 0 lost, echo p99 74 ms | 0, 0 |
Announcement and loss counts from the same runs (computed from the logs: the server-side participant logs every Announced and Retracted update for the room; each client logs its own session opens and closes):
| Setup (run) | client paths | Announced |
Retracted |
Announced > 1 s after the publisher closed |
retraction delay p50 / p95 | groups lost while the publisher stayed connected |
|---|---|---|---|---|---|---|
| Different nodes, GeoDNS, reopen off | 164 | 611 | 507 | 248 (up to 153 s after) | 57.68 s / 120.8 s | 134 |
| Different nodes, clients pinned, reopen on | 176 | 633 | 526 | 254 (up to 155 s after) | 99.74 s / 138.5 s | 23 |
| Same node, reopen off | 220 | 220 | 220 | 0 | 0.04 s / 0.1 s | 0 |
| Self-hosted, reopen off | 220 | 220 | 220 | 0 | 0.03 s / 0.1 s | 0 |
Each node on its own is solid; the problems appear only in inter-node routing, especially during bursts of new announces and subscribes. Happy to share the load harness or re-run it against a test cluster.
Trace excerpts
Sanitized: <room>, <client-n> and <session-n.k> replace ids. moq_net::/moq_json:: lines are from the crates; app:: lines are the server-side participant's own. Client JSON lines are prefixed with their UTC timestamp. Clocks are NTP-synchronised on two hosts.
A control group lost cross-node; its FETCH never returns while the publisher stays connected
# client 3, its own event log (JSON, trimmed)
2026-09-26T10:39:21.611733Z {"kind":"sent","user":"<client-3>","cseq":139,"intent":"set"} # group 138 of the session
# server-side participant: control groups as received (fetched = came from FETCH)
2026-09-26T10:39:21.640Z {"clientId":"<client-3>","group":137,"cseq":138,"outcome":"held_by","fetched":false}
2026-09-26T10:39:21.676Z {"clientId":"<client-3>","group":139,"cseq":140,"outcome":"control_gap","fetched":false}
# server-side participant log
2026-09-26T10:39:21.676257Z INFO moq_net::lite::subscriber: fetch started broadcast=<room>/control/<client-3>/<session-3.1> track=control group=138
2026-09-26T10:39:23.677134Z WARN app::control: control_fetch_deadline user=<client-3> sequence=138
# harness verdict
"<client-3> cseq 139 never reached the engine log"
# client 3, its own event log (JSON, trimmed)
2026-09-26T10:39:53.113443Z {"kind":"session","user":"<client-3>","event":"left","detail":"end"} # the publisher stayed connected 29 s after the FETCH deadline
The first groups of a burst miss a subscription opened 5.2 s earlier; FETCH for them times out
# server-side participant log
2026-09-26T11:24:16.050264Z INFO moq_net::lite::subscriber: subscribe started id=21 broadcast=<room>/control/<client-0>/<session-0.1> track=control
# client 0, its own event log (JSON, trimmed)
2026-09-26T11:24:21.299313Z {"kind":"sent","user":"<client-0>","cseq":1,"intent":"resync"} # burst: cseq 1 to 14 within 0.1 ms
2026-09-26T11:24:21.299366Z {"kind":"sent","user":"<client-0>","cseq":14,"intent":"set"}
# server-side participant: control groups as received (fetched = came from FETCH)
2026-09-26T11:24:21.584Z {"clientId":"<client-0>","group":5,"cseq":6,"outcome":"resync_required","fetched":false}
# server-side participant log
2026-09-26T11:24:21.584985Z INFO moq_net::lite::subscriber: fetch started broadcast=<room>/control/<client-0>/<session-0.1> track=control group=0
2026-09-26T11:24:21.585007Z INFO moq_net::lite::subscriber: fetch started broadcast=<room>/control/<client-0>/<session-0.1> track=control group=1
2026-09-26T11:24:21.585013Z INFO moq_net::lite::subscriber: fetch started broadcast=<room>/control/<client-0>/<session-0.1> track=control group=2
2026-09-26T11:24:21.585017Z INFO moq_net::lite::subscriber: fetch started broadcast=<room>/control/<client-0>/<session-0.1> track=control group=3
2026-09-26T11:24:21.585022Z INFO moq_net::lite::subscriber: fetch started broadcast=<room>/control/<client-0>/<session-0.1> track=control group=4
2026-09-26T11:24:23.585159Z WARN app::control: control_fetch_deadline user=<client-0> sequence=0
2026-09-26T11:24:23.585181Z WARN app::control: control_fetch_deadline user=<client-0> sequence=1
2026-09-26T11:24:23.585186Z WARN app::control: control_fetch_deadline user=<client-0> sequence=2
2026-09-26T11:24:23.585191Z WARN app::control: control_fetch_deadline user=<client-0> sequence=3
2026-09-26T11:24:23.585196Z WARN app::control: control_fetch_deadline user=<client-0> sequence=4
# harness verdict
"<client-0> cseq 1 never reached the engine log"
"<client-0> cseq 2 never reached the engine log"
"<client-0> cseq 3 never reached the engine log"
"<client-0> cseq 4 never reached the engine log"
"<client-0> cseq 5 never reached the engine log"
# client 0, its own event log (JSON, trimmed)
2026-09-26T11:25:05.954135Z {"kind":"session","user":"<client-0>","event":"left","detail":"end"}
`state` and `log` stop for 30.4 s cross-node while the client-to-server direction keeps working
# client 6, its own event log (JSON, trimmed)
2026-09-26T11:20:10.635930Z {"kind":"sent","user":"<client-6>","cseq":208,"intent":"set"}
2026-09-26T11:20:10.786862Z {"kind":"sent","user":"<client-6>","cseq":212,"intent":"release"}
2026-09-26T11:20:10.996834Z {"kind":"echo","user":"<client-6>","cseq":208,"stateSeq":367} # last `state` before the gap
# server-side participant: control groups as received (fetched = came from FETCH)
2026-09-26T11:20:10.692Z {"clientId":"<client-6>","group":208,"cseq":209,"outcome":"held_by","fetched":false}
2026-09-26T11:20:10.951Z {"clientId":"<client-6>","group":211,"cseq":212,"outcome":"not_holder","fetched":false}
# client library log, first and last of 166 such lines in the gap (all ten clients)
2026-09-26T11:20:11.385450Z WARN moq_json::snapshot::consumer: snapshot group lost; waiting for a newer one track="state" group=30 error=Stream(Old)
2026-09-26T11:20:41.129182Z WARN moq_json::snapshot::consumer: snapshot group lost; waiting for a newer one track="state" group=60 error=Stream(Old)
# client 6, its own event log (JSON, trimmed)
2026-09-26T11:20:41.421127Z {"kind":"log","user":"<client-6>","seq":361,"publishedAt":"2026-09-26T11:20:10.580Z"} # first `log` entry after the gap
2026-09-26T11:20:41.452130Z {"kind":"log","user":"<client-6>","seq":371,"publishedAt":"2026-09-26T11:20:10.960Z"}
2026-09-26T11:20:41.445490Z {"kind":"echo","user":"<client-6>","cseq":209,"stateSeq":371} # first `state` after the gap
2026-09-26T11:20:42.138390Z {"kind":"session","user":"<client-6>","event":"left","detail":"end"}
A closed session stays announced for 61 s and answers `unroutable` / `internal error`, then is announced again
# client 3, its own event log (JSON, trimmed)
2026-09-26T10:39:39.375223Z {"kind":"sent","user":"<client-3>","cseq":144,"intent":"resync"} # its last message
2026-09-26T10:39:53.113443Z {"kind":"session","user":"<client-3>","event":"left","detail":"end"} # the client closes the session
# server-side participant log
2026-09-26T10:39:53.410211Z WARN moq_net::lite::subscriber: subscribe error broadcast=<room>/control/<client-3>/<session-3.1> track=control err=unroutable
2026-09-26T10:39:59.811220Z WARN moq_net::lite::subscriber: track info failed broadcast=<room>/control/<client-3>/<session-3.1> track=control err=internal error
2026-09-26T10:40:28.829035Z WARN moq_net::lite::subscriber: track info failed broadcast=<room>/control/<client-3>/<session-3.1> track=control err=unroutable
2026-09-26T10:40:30.905995Z WARN moq_net::lite::subscriber: track info failed broadcast=<room>/control/<client-3>/<session-3.1> track=control err=unroutable
2026-09-26T10:40:31.056022Z WARN moq_net::lite::subscriber: track info failed broadcast=<room>/control/<client-3>/<session-3.1> track=control err=unroutable
2026-09-26T10:40:31.435353Z WARN moq_net::lite::subscriber: track info failed broadcast=<room>/control/<client-3>/<session-3.1> track=control err=unroutable
2026-09-26T10:40:32.678804Z WARN moq_net::lite::subscriber: track info failed broadcast=<room>/control/<client-3>/<session-3.1> track=control err=unroutable
2026-09-26T10:40:54.407664Z INFO app::discovery: announce room=<room> path=<room>/control/<client-3>/<session-3.1> kind=Retracted
2026-09-26T10:40:55.997519Z INFO app::discovery: announce room=<room> path=<room>/control/<client-3>/<session-3.1> kind=Announced
2026-09-26T10:40:57.853054Z INFO app::discovery: announce room=<room> path=<room>/control/<client-3>/<session-3.1> kind=Retracted
The same event on one node: error and retraction 38 ms after the close
# client 3, its own event log (JSON, trimmed)
2026-09-26T10:53:12.206389Z {"kind":"session","user":"<client-3>","event":"left"} # the client closes the session
# server-side participant log
2026-09-26T10:53:12.244193Z WARN moq_net::lite::subscriber: subscribe error broadcast=<room>/control/<client-3>/<session-3.3> track=control err=internal error
2026-09-26T10:53:12.244332Z WARN moq_net::lite::subscriber: subscribe error broadcast=<room>/sources/<client-3>-s0/audio/<session-3.3> track=audio err=internal error
2026-09-26T10:53:12.245277Z WARN app::discovery: reader_ended room=<room> path=<room>/sources/<client-3>-s0/audio/<session-3.3> code="container" error=container: moq: dropped
2026-09-26T10:53:12.245321Z INFO app::discovery: announce room=<room> path=<room>/control/<client-3>/<session-3.3> kind=Retracted
2026-09-26T10:53:12.245365Z INFO app::discovery: announce room=<room> path=<room>/sources/<client-3>-s0/audio/<session-3.3> kind=Retracted
A closed session announced again 12 times, the last 153 s after the publisher closed
# server-side participant log
2026-09-26T10:39:03.906449Z INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Announced
# client 6, its own event log (JSON, trimmed)
2026-09-26T10:40:35.149956Z {"kind":"session","user":"<client-6>","event":"left","detail":"end"} # the client closes the session
# server-side participant log
2026-09-26T10:42:09.147148Z INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Retracted
2026-09-26T10:42:09.593019Z INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Announced
... (19 more lines, alternating Retracted and Announced) ...
2026-09-26T10:43:02.982781Z INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Announced
2026-09-26T10:43:04.041069Z INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Retracted
2026-09-26T10:43:08.174620Z INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Announced
2026-09-26T10:43:09.352609Z INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Retracted
Fuller sanitized traces (per-run counts and more excerpts) can be attached on request.
Activity
fperex commentedon Sep 29, 2026
Cross-node routing on cdn.moq.pro: sanitized traces
Sanitized excerpts and counts behind the report "Cross-node routing on cdn.moq.pro loses groups, stalls tracks and
keeps closed broadcasts announced under bursty control traffic". Everything below was extracted from the run folders
by script; nothing was re-run.
single-node runs from 14:29 to 18:14 UTC the same day.
moq-net0.3.5 andmoq-tokio0.19.17 (withmoq-mux0.10.6 andmoq-json0.5.3),moq-lite-06over QUIC, JWT with publish and subscribe path patterns. Browser clients(
@moq/net0.4.1) were used in other runs and are not part of these traces.cdn.moq.pro(GeoDNS over several nodes) and, for run C, a self-hostedmoq-relay0.15.6, single node.<moq-node-ip-2>,<moq-node-ip-3>). Ten clients run on one host about 40 ms away; GeoDNS gave that host the Linode node<moq-node-ip-1>at 10:38 to 11:00 and at 11:54 UTC and the OVH nodes at 11:09 to 11:40 UTC, so two cross-node runspin the clients to
<moq-node-ip-1>by address (TLS namecdn.moq.pro).Workload
Each client join opens a fresh session and publishes two broadcasts under it:
<room>/control/<client-n>/<session>with the track
control(one group per message, one frame per group, bursts at up to 60 groups/s) and<room>/sources/<client-n>-s0/audio/<session>(20 ms audio groups). The server-side participant subscribes to everyannounced client broadcast, applies the messages and publishes
<room>/mixerwith the tracksstate(JSON snapshotand deltas) and
log, plus per-client audio outputs (<room>/mix-minus/<client-n>) and<room>/program-out/audio.When a later control group arrives before an earlier one it FETCHes the missing group and gives up after 2 s
(
control_fetch_deadline). Phases:contend(90 s, one session per client, all ten contending for the same control)and
join-leave(90 s, every client joins and leaves about ten times, a new session path each time). The finaltrials add
drag,storm,killandrevoke. "Reopen" is a client option (open a fresh session after theserver-side participant reports the control track unreachable); it did not change the relay behaviour.
Notation
<room>replaces the room prefix,<client-n>a client id,<session-n.k>the k-th session of client n in thatrun,
<moq-node-ip-n>a node address. Paths keep their shape; the client logs them relative to the room prefix.app::...lines are the server-side participant's own log targets;moq_net::,moq_mux::andmoq_json::linescome from the moq crates. Journal lines are verbatim apart from the substitutions.
bots-*/bot-NN.jsonl) are trimmed to the relevant fields and prefixed with their own UTCtimestamp (
atUs).publishedAtis the server-side participant's publish time of alogentry (tEngineMs).engine/hops.jsonllines are the server-side participant's record of each control group, prefixed with the time itreceived the group;
fetchedsays whether the group came from a FETCH.below about 50 ms between the two sides are not meaningful.
Runs
<moq-node-ip-2><moq-node-ip-2><moq-node-ip-2><moq-node-ip-2><moq-node-ip-1>(Linode)<moq-node-ip-1>(Linode)<moq-node-ip-2><moq-node-ip-2>Results
Same binaries, ten clients,
contendandjoin-leavephases. "Lost" is a message a client sent that never reachedthe server-side participant's log; "echo p99" is the time from a client sending a message to that client seeing it
in
state. Fetch failures and unreachable marks are the server-side participant's own counters over the phase(
metrics.jsonl); an unreachable mark is acontrolsubscription that failed while its path was announced.statewithin 30 sstatewithin 30 sThe final trials, six phases each (from
converge-<phase>.json):Counts computed from the files
How each column was computed (Python over the run folders; every count covers the whole run):
(unix ms << 16) | random, so each path's session maps to the client join thatcreated it; the session ends at that client's next
leftorkilledevent in its ownbots-*/bot-NN.jsonl(or at the phase end if the client exited without one).
app::discovery: announce ... kind=Announced|Retractedlines inengine-journal.txt. Theserver-side participant logs every
AnnouncedandRetractedupdate it receives for the room prefix (it does notlog
Updated). "After close" countsAnnouncedevents for a path more than 1 s after its client closed the session; "retraction delay" is the timefrom the client closing the session to the first
Retractedfor itscontrolpath.moq_net::lite::subscriberlinessubscribe errorortrack info failedwitherr=unroutableorerr=internal erroron a clientcontrolorsourcespath, compared with the same session'sclose time. "Stale" means more than 1 s after the close while the last announce event for the path was
Announced.sentrecords in the client files whose(client, cseq)never appears inengine/log.jsonl(this reproduces the harness's
lostexactly for the A runs; the final GeoDNS trial also counts the 554 messagesof killed clients). "Sent while connected" means sent at least 1 s before the client closed the session. The
control group of a message is its index among the session's
sentrecords (checked againstengine/hops.jsonl:26 of 2481 mismatches in A GeoDNS reopen off, 0 in every other run). A "FETCH timeout" is a
control_fetch_deadlineline for that group while the client was still connected at the deadline. "No FETCHlogged" means no
fetch startedline for the group; the journal holds fewerfetch startedlines than theserver-side participant's own FETCH counter (364 against 813 in A GeoDNS, reopen off), so read that column as an
upper bound.
controlsubscription was already open (subscribe started) when the clientsent its first burst, but whose first group delivered by that subscription (
engine/hops.jsonl,fetched:false) was not group 0; the earlier groups of the burst were then FETCHed or lost.logdelay: for everylogentry a client received for the first time and that was published after the client'scurrent join, receive time (
atUs) minus publish time (tEngineMs).stateOld:moq_json::snapshot::consumer: snapshot group lost ... track="state" ... error=Stream(Old)lines inbots-*.out(all ten clients write to one file).state: client result lines intrial.outsaying the firststatedid not arrive within 30 s.followedOnUpdatein the summary at the end ofprobe.out: how often a test subscriber on theclients' host started reading a wanted broadcast because of an
Updatedannounce (moq-net: "the route coveringthe path changed hops or cost") rather than an
Announcedone.Announcements
In the B, C and one-node runs every retraction slower than 1 s took exactly 30.0 to 30.1 s and belongs to a session
that ended at a phase boundary, when the client process exits (consistent with a transport idle timeout); the
cross-node delays are spread over tens of seconds to minutes.
Subscribe errors on client paths
The one error counted before a close (final GeoDNS trial,
killphase, an audio path) belongs to a killed clientwhose log records no end for that session, so its close time defaults to the phase end.
Lost control groups and FETCH
fetched:truegroups)In the final one-node trial all 106 lost messages belong to killed clients. A FETCH that works returns within tens
to hundreds of milliseconds (37 to 803 ms for the ten groups in section 5), so the 2 s deadline is not
the limiting factor.
Server-side participant to clients
logdelay p99logdelay maxlogentries > 5 sstateOld linesstatein 30 s1.
unroutableandinternal errorfor announced client tracksChecked against each client's own log, every one of these answers in these runs came after the client had closed
the session: none arrived while the publisher was connected (table above, last column). The failure is that the
path stays announced, or is announced again, long after the publisher left, and every subscribe to it then fails.
On one node the subscribe error and the retraction arrive together, about 40 ms after the client leaves.
A GeoDNS, reopen off, phase
contend,<client-3>session<session-3.1>:The path stayed announced for 61 s after the close, answered 7 subscribes with
unroutableorinternal error,was retracted, announced again 1.6 s later and retracted again.
For comparison, B one node, reopen off, phase
join-leave,<client-3>session<session-3.3>:2. Control groups lost; FETCH does not return them
A GeoDNS, reopen off, phase
contend,<client-3>session<session-3.1>, group 138 (its messagecseq139):Groups 137 and 139 arrived through the subscription 36 ms apart; group 138 did not, its FETCH did not return in
2 s, and it never arrived although the publisher stayed connected. Across the cross-node runs this happened to
109 (A GeoDNS, reopen off), 9 and 70 (A clients pinned, reopen on and off) and 210 (final GeoDNS) messages; B, C and the final
one-node trial lost none while the publisher was connected. When the publisher goes idle after such a loss nothing
later arrives to reveal the gap, so the subscriber cannot even start a FETCH ("no FETCH logged" column).
3. Closed sessions announced again
A GeoDNS, reopen off,
<client-6>session<session-6.1>, which the client closed at the end ofcontend. Itscontrolpath stayed announced for 94 s, then was retracted and announced again 12 more times; the last
Announcedcame153 s after the close:
Totals are in the Announcements table: on one node and on the self-hosted relay every path is announced once and
retracted once (220 and 220 for 220 paths); cross-node the same workload produced 611
Announcedand 507Retractedevents for 164 paths (A GeoDNS, reopen off) and 1856 and 1782 for 292 paths (final GeoDNS).4.
state(andlog) stop reaching clients on the other nodeA clients pinned, reopen on, phase
contend. The server-side participant received and applied<client-6>'smessages within 0.2 s of sending (the client to server direction kept working), but the
stateandlogupdatesthey caused reached the client 30.7 s later, after 30.4 s with nothing arriving. The client stayed connected
throughout; all ten clients show the same gap.
Each of
stategroups 30 to 60 (one group per second in this phase) reached the clients' consumer only as alost group (
Stream(Old)). The same pattern twice kept every client from receiving any firststatefor 30 s.A GeoDNS, reopen on, phase
contend:The other no-first-
statecase is A clients pinned, reopen off, phasejoin-leave. Longestlogdelays perrun, one line per client stall (client, publish time of the first delayed entry, longest delay):
5. The first groups of a burst miss an open subscription
A clients pinned, reopen off, phase
contend. The server-side participant subscribed to<client-0>'scontroltrack 5.2 s before the client's first burst (the client waits for its first
state). The subscription deliveredgroup 5 first; groups 0 to 4 were FETCHed, the FETCHes did not return within 2 s, and the groups never arrived
although the client stayed connected for 44 s after the burst:
In the same run
<client-3>'s subscription (open 5.8 s before its burst) delivered group 10 first and FETCHrecovered groups 0 to 9 within 37 to 803 ms:
Across runs (Lost control groups table): A clients pinned, reopen off, 10 of 10 sessions, 92 groups, 70 FETCHed,
22 lost; final GeoDNS 7 sessions, 27 groups, 17 FETCHed, 10 lost; none in any B, C or one-node run.
6. The end of the output subscription after the last client leaves
The traces show this is our own behaviour, not the relay's. The server-side participant ends its outputs 30 s after
the last
controltrack ends (live_ended, reasonhost_left). In the runs where the end came after 60 s, thedeparted clients'
controltracks themselves ended 30 s after the clients left, the same 30 s that every relay,the self-hosted one included, takes when a client process exits at a phase boundary (see the Announcements note).
The only relay-side difference: on cdn.moq.pro the test subscriber sees the clean end as
internal errorandits resubscribe as
unroutable(the broadcast is gone by then, so theunroutableis correct); on the self-hostedrelay the subscription completes without an error.
internal error, thenunroutableinternal error, thenunroutableinternal error, thenunroutableinternal error, thenunroutableinternal error, thenunroutableinternal error, thenunroutableinternal error, thenunroutableOne node, 17:59 run (the server-side participant, the clients and the test subscriber pinned to
<moq-node-ip-2>):Journal lines are ordered by time;
live_endedinengine/log.jsonlis the server-side participant's decisionand precedes the
subscribed completelines by 1 to 2 ms.7. A re-created broadcast arrives as
UpdatedThe server-side participant's output broadcasts, for example
<room>/mix-minus/<client-n>, are finished and publishedagain as clients leave and join. A test subscriber on the clients' host follows them through the announce stream. On one node and on the
self-hosted relay it always learned of the new broadcast through
Announced; across nodes it often gotUpdatedfor a path whose previous broadcast it had already stopped reading. In moq-net
Updatedmeans the route covering apath the subscriber already had changed, so its announce stream never saw the old broadcast retracted. A subscriber
that only acts on
Announcedmisses those broadcasts. From the summary at the end of eachprobe.out:The subscriber does not log each announce, so there are no per-event lines for this class.
kixelated commentedon Oct 6, 2026
Thanks for the detailed report and traces. We tried to reproduce this on a local two-relay cluster, using one-frame groups in bursts of 14, a subscription opened 5 s before the first burst, fetch-on-gap with a 2 s deadline, and a flapping peer link. We ran it against the 0.15.6 relay tree your clients ran, and against current
main.state/logstalls withStream(Old), didn't show up on any build. Our harness doesn't model flow control or loss, so it may simply miss them. Several 30 s timers from that era have since been removed or shortened (feat(tokio): default QUIC idle timeout to 10s #4606, fix(net)!: resume route changes by reading the routes' copies; a path is one broadcast #4741).Could you re-run your A/B/C comparison against current cdn.moq.pro? If you can share the harness, even a minimal version, that would help a lot. For any FETCH that fails, please also note whether it ended in an error or a timeout.
(Written by Claude Opus 5.5)
fperex commentedon Oct 11, 2026
Thanks for digging into this. Here is the re-run against current cdn.moq.pro (2026-10-11 UTC), with the harness and the traces.
Short version
ok: 0 refused, 0 unreadable, 0 deadline. After a deadline we keep the FETCH open to 10 s to telllatefromnever; both are 0. So there is no error-or-timeout split to report.Retractedcomes about 0.2 s after a clean close, and no path stays announced at the end of any run.moq-relay0.17.2 and on a self-hosted two-relay 0.17.2 cluster, this never happens.Setup
moq-net0.3.10,moq-tokio0.19.23, three runs per cell. Old train:moq-net0.3.5,moq-tokio0.19.17 (the original report's), one run per cell. Self-hosted relay: themoq-relay0.17.2 release binary.moq-lite-06over QUIC, JWT with publish and subscribe patterns.40.160.233.192; the ten clients and a test subscriber run on a second host about 60 ms from that node.40.160.233.192and40.160.250.184). A-pin: server pinned to40.160.233.192, clients to139.177.201.247, by address with TLS namecdn.moq.pro. B: both pinned to40.160.233.192. C: one self-hosted relay. D: a self-hosted two-relay cluster, server on relay 1 and clients on relay 2.contendandjoin-leave, with the clients'statereopen off and on. Plus one six-phase run (drag,contend,storm,join-leave,kill,revoke) on A-pin and on B.Loss, latency and FETCH, new train (mean of three runs, range in brackets; FETCH records are summed over the three runs)
contendlost / echo p99 msjoin-leavelost / echo p99 msokThe six-phase runs lose nothing in any phase either. FETCH records: A-pin 6899, B 4524, all
ok.Old train on today's cdn.moq.pro: also 0 lost and 0 failed FETCH. The old client only has aggregate counters, so there are no per-request records.
contendlost / echo p99 msjoin-leavelost / echo p99 msAnnouncements
AnnouncedRetractedUpdatedAnnounced/Updated> 1 s after a clean close220 is one
Announcedand oneRetractedfor each broadcast the clients open in a run. The six-phase run on A-pin has 11,540Announced, 388Updatedand 4575 events after a close on 17 sessions; on B it has 480, 0 and 0.A typical cross-node sequence (new train, A-pin, reopen off, run 2, one client's ninth session, times from its clean close):
The same thing happens with GeoDNS and on the old train. While it lasts, subscribes to the path answer
internal errororunroutable. Injoin-leave,internal erroroncontrolhappens 64 to 99 times per cross-node run, against 18 to 21 on B, 12 to 15 on C and 3 to 4 on D. In B, C and D, one or two sessions per run get their firstRetracted10.0 s after the close (it looks like the transport idle timeout); the cross-node runs have that too.Your suggestion (A-pin, reopen off, one run each)
contend)Announced/Updated> 1 s after a closecontrolmax age 120 sWith the wait, a group that arrives in the meantime is not fetched. A 250 ms wait removes the FETCHes entirely, and the harness shows the same: 12,341 to 0 in
contendand 3971 to 0 injoin-leave. A 120 s max age changes nothing in either. Neither setting changes the re-announcing.One thing on our own cluster (D) we could not explain. In 2 of the 6 D runs, 3 clients in total got no first
stategroup within 30 s on their last rejoin. In one of them, the relay with the clients loggedsubscribe canceled (idle) track=statewhen the client's previous session left (+176.0 s). The server accepted the newstatesubscribe at +177.5 s, the client loggedread log: oldat +177.6 s, and itsstatenever arrived. It did not happen on A, B or C, or on the other four D runs. We did not root-cause it. The details are in the traces.Not covered: #4741 is not in a release yet, so these runs do not include it.
Harness: https://github.com/fperex/moq-burst-repro
burst-reprois a single binary with three subcommands:serversubscribes through announcements, FETCHes gaps with a 2 s deadline and echoes onstate;clientsruns N publishers withcontendorjoin-leave;reportprints the tables from the result files.moq-net0.3.10,moq-tokio0.19.23).statestalls in every run.join-leavewith the clients on the Linode node gives 4737 (3578 to 5419)Announced/Updatedafter a close, the latest about 2 s after it. With the clients on the Silicon Valley node40.160.235.183it gives 33 (0 to 100), and on B and C it gives 0.Traces: in
traces/in the same repository.traces.md: every table, the per-session announcement sequences and the D details.fetches-sample-*.jsonl: the per-FETCH record format.harness-reports.txt: the harness report of every run.harness-raw.tar.xz: the raw harness files of fivejoin-leaveruns.quest(m1): fold the abort plan into dropped-sources
quest(m1): plan moq-dev#4349 follow-ups after the reporter's re-run (m…
quest: give Peers relay auth and move #4349 closure to routes
test(net): pin cross-node close flapping and its residuals
test(net): pin cross-node close flapping and its residuals
test(net): pin cross-node close flapping and its residuals