Skip to content

Cross-node routing on cdn.moq.pro loses groups, stalls tracks and keeps closed broadcasts announced under bursty control traffic (A/B/C repro inside) #4349

Closed

Description

@fperex

I'm building a multiplayer collaborative app on MoQ. Many participants each publish a small control track (about 20 to 60 tiny groups/s in bursts, one group per message), and one server-side participant subscribes to all of them and fans state back out. While load-testing on cdn.moq.pro I hit a pattern that only appears when the publishers and the subscriber land on different moq.pro nodes.

Setup (2026-09-26, 10:38 to 12:06 UTC): the server-side participant resolves to an OVH node about 0.7 ms away; 10 publishing clients about 40 ms away resolve to a Linode node, sometimes an OVH one. Clients: moq-net 0.3.5 / moq-tokio 0.19.17 (native, all traces below) and @moq/net 0.4.1 (browsers, separate runs), moq-lite-06, JWT with publish/subscribe patterns. Each client join opens a new session and publishes <room>/control/<client>/<session> (track control) plus an audio broadcast; the server-side participant subscribes to every announced client broadcast and publishes state and log tracks back. It FETCHes a missing control group when a later one arrives first and gives up after 2 s.

Cross-node only:

  • Some control groups never arrive, and FETCH for them does not return within 2 s while the publisher is still connected (109 of 142 lost messages in one run). When the publisher then goes idle, nothing reveals the gap at all.
  • A subscription opened seconds before a publisher's first burst sometimes delivers only the tail of the burst; the first groups need FETCH, which sometimes fails too.
  • The state and log tracks stall for tens of seconds (up to 31 s) while every client stays connected; twice no client received a first state within 30 s.
  • Closed sessions stay announced for tens of seconds to minutes, or are announced again after being retracted (up to 229 s after the publisher closed the session cleanly). Subscribing to such a path answers unroutable or internal error. In these runs every one of those answers came after the publisher had closed the session, so the answers are right and the announcements are stale.
  • A re-created broadcast reaches a subscriber as Updated instead of Announced, because the subscriber never saw the old one retracted (46 to 910 times per run cross-node, never on one node).

Same workload three ways (contend phase, reopen off): different nodes (GeoDNS): 123 of 2,273 messages lost, echo p99 23 s. Everyone pinned to one node: 0 of 6,310 lost, echo p99 91 ms. Self-hosted moq-relay 0.15.6, single node: 0 of 6,310 lost, echo p99 73 ms. Echo is the time from a client sending a message to that client seeing it in state.

Setup Client reopen contend join-leave control FETCHes failed
Different nodes (GeoDNS) off FAIL: 2273 sent, 123 lost, echo p99 23322 ms FAIL: 395 sent, 19 lost, echo p99 25318 ms 153, 11
Different nodes (clients pinned to the other node) on FAIL: 2536 sent, 9 lost, echo p99 30775 ms FAIL: 383 sent, 14 lost, echo p99 5270 ms 8, 7
Different nodes (clients pinned to the other node) off FAIL after 51 s: 1421 sent, 71 lost, echo p99 2012 ms no client got a first state within 30 s 70
Different nodes (GeoDNS) on no client got a first state within 30 s pass: 519 sent, 0 lost, echo p99 87 ms 0
Same node (both pinned) off pass: 6310 sent, 0 lost, echo p99 91 ms pass: 536 sent, 0 lost, echo p99 142 ms 0, 0
Same node (both pinned) on pass: 6310 sent, 0 lost, echo p99 97 ms pass: 527 sent, 0 lost, echo p99 142 ms 0, 0
Self-hosted moq-relay 0.15.6 off pass: 6310 sent, 0 lost, echo p99 73 ms pass: 535 sent, 0 lost, echo p99 73 ms 0, 0
Self-hosted moq-relay 0.15.6 on pass: 6310 sent, 0 lost, echo p99 76 ms pass: 506 sent, 0 lost, echo p99 74 ms 0, 0

Announcement and loss counts from the same runs (computed from the logs: the server-side participant logs every Announced and Retracted update for the room; each client logs its own session opens and closes):

Setup (run) client paths Announced Retracted Announced > 1 s after the publisher closed retraction delay p50 / p95 groups lost while the publisher stayed connected
Different nodes, GeoDNS, reopen off 164 611 507 248 (up to 153 s after) 57.68 s / 120.8 s 134
Different nodes, clients pinned, reopen on 176 633 526 254 (up to 155 s after) 99.74 s / 138.5 s 23
Same node, reopen off 220 220 220 0 0.04 s / 0.1 s 0
Self-hosted, reopen off 220 220 220 0 0.03 s / 0.1 s 0

Each node on its own is solid; the problems appear only in inter-node routing, especially during bursts of new announces and subscribes. Happy to share the load harness or re-run it against a test cluster.

Trace excerpts

Sanitized: <room>, <client-n> and <session-n.k> replace ids. moq_net::/moq_json:: lines are from the crates; app:: lines are the server-side participant's own. Client JSON lines are prefixed with their UTC timestamp. Clocks are NTP-synchronised on two hosts.

A control group lost cross-node; its FETCH never returns while the publisher stays connected
# client 3, its own event log (JSON, trimmed)
2026-09-26T10:39:21.611733Z  {"kind":"sent","user":"<client-3>","cseq":139,"intent":"set"}   # group 138 of the session
# server-side participant: control groups as received (fetched = came from FETCH)
2026-09-26T10:39:21.640Z  {"clientId":"<client-3>","group":137,"cseq":138,"outcome":"held_by","fetched":false}
2026-09-26T10:39:21.676Z  {"clientId":"<client-3>","group":139,"cseq":140,"outcome":"control_gap","fetched":false}
# server-side participant log
2026-09-26T10:39:21.676257Z  INFO moq_net::lite::subscriber: fetch started broadcast=<room>/control/<client-3>/<session-3.1> track=control group=138
2026-09-26T10:39:23.677134Z  WARN app::control: control_fetch_deadline user=<client-3> sequence=138
# harness verdict
"<client-3> cseq 139 never reached the engine log"
# client 3, its own event log (JSON, trimmed)
2026-09-26T10:39:53.113443Z  {"kind":"session","user":"<client-3>","event":"left","detail":"end"}   # the publisher stayed connected 29 s after the FETCH deadline
The first groups of a burst miss a subscription opened 5.2 s earlier; FETCH for them times out
# server-side participant log
2026-09-26T11:24:16.050264Z  INFO moq_net::lite::subscriber: subscribe started id=21 broadcast=<room>/control/<client-0>/<session-0.1> track=control
# client 0, its own event log (JSON, trimmed)
2026-09-26T11:24:21.299313Z  {"kind":"sent","user":"<client-0>","cseq":1,"intent":"resync"}   # burst: cseq 1 to 14 within 0.1 ms
2026-09-26T11:24:21.299366Z  {"kind":"sent","user":"<client-0>","cseq":14,"intent":"set"}
# server-side participant: control groups as received (fetched = came from FETCH)
2026-09-26T11:24:21.584Z  {"clientId":"<client-0>","group":5,"cseq":6,"outcome":"resync_required","fetched":false}
# server-side participant log
2026-09-26T11:24:21.584985Z  INFO moq_net::lite::subscriber: fetch started broadcast=<room>/control/<client-0>/<session-0.1> track=control group=0
2026-09-26T11:24:21.585007Z  INFO moq_net::lite::subscriber: fetch started broadcast=<room>/control/<client-0>/<session-0.1> track=control group=1
2026-09-26T11:24:21.585013Z  INFO moq_net::lite::subscriber: fetch started broadcast=<room>/control/<client-0>/<session-0.1> track=control group=2
2026-09-26T11:24:21.585017Z  INFO moq_net::lite::subscriber: fetch started broadcast=<room>/control/<client-0>/<session-0.1> track=control group=3
2026-09-26T11:24:21.585022Z  INFO moq_net::lite::subscriber: fetch started broadcast=<room>/control/<client-0>/<session-0.1> track=control group=4
2026-09-26T11:24:23.585159Z  WARN app::control: control_fetch_deadline user=<client-0> sequence=0
2026-09-26T11:24:23.585181Z  WARN app::control: control_fetch_deadline user=<client-0> sequence=1
2026-09-26T11:24:23.585186Z  WARN app::control: control_fetch_deadline user=<client-0> sequence=2
2026-09-26T11:24:23.585191Z  WARN app::control: control_fetch_deadline user=<client-0> sequence=3
2026-09-26T11:24:23.585196Z  WARN app::control: control_fetch_deadline user=<client-0> sequence=4
# harness verdict
"<client-0> cseq 1 never reached the engine log"
"<client-0> cseq 2 never reached the engine log"
"<client-0> cseq 3 never reached the engine log"
"<client-0> cseq 4 never reached the engine log"
"<client-0> cseq 5 never reached the engine log"
# client 0, its own event log (JSON, trimmed)
2026-09-26T11:25:05.954135Z  {"kind":"session","user":"<client-0>","event":"left","detail":"end"}
`state` and `log` stop for 30.4 s cross-node while the client-to-server direction keeps working
# client 6, its own event log (JSON, trimmed)
2026-09-26T11:20:10.635930Z  {"kind":"sent","user":"<client-6>","cseq":208,"intent":"set"}
2026-09-26T11:20:10.786862Z  {"kind":"sent","user":"<client-6>","cseq":212,"intent":"release"}
2026-09-26T11:20:10.996834Z  {"kind":"echo","user":"<client-6>","cseq":208,"stateSeq":367}   # last `state` before the gap
# server-side participant: control groups as received (fetched = came from FETCH)
2026-09-26T11:20:10.692Z  {"clientId":"<client-6>","group":208,"cseq":209,"outcome":"held_by","fetched":false}
2026-09-26T11:20:10.951Z  {"clientId":"<client-6>","group":211,"cseq":212,"outcome":"not_holder","fetched":false}
# client library log, first and last of 166 such lines in the gap (all ten clients)
2026-09-26T11:20:11.385450Z  WARN moq_json::snapshot::consumer: snapshot group lost; waiting for a newer one track="state" group=30 error=Stream(Old)
2026-09-26T11:20:41.129182Z  WARN moq_json::snapshot::consumer: snapshot group lost; waiting for a newer one track="state" group=60 error=Stream(Old)
# client 6, its own event log (JSON, trimmed)
2026-09-26T11:20:41.421127Z  {"kind":"log","user":"<client-6>","seq":361,"publishedAt":"2026-09-26T11:20:10.580Z"}   # first `log` entry after the gap
2026-09-26T11:20:41.452130Z  {"kind":"log","user":"<client-6>","seq":371,"publishedAt":"2026-09-26T11:20:10.960Z"}
2026-09-26T11:20:41.445490Z  {"kind":"echo","user":"<client-6>","cseq":209,"stateSeq":371}   # first `state` after the gap
2026-09-26T11:20:42.138390Z  {"kind":"session","user":"<client-6>","event":"left","detail":"end"}
A closed session stays announced for 61 s and answers `unroutable` / `internal error`, then is announced again
# client 3, its own event log (JSON, trimmed)
2026-09-26T10:39:39.375223Z  {"kind":"sent","user":"<client-3>","cseq":144,"intent":"resync"}   # its last message
2026-09-26T10:39:53.113443Z  {"kind":"session","user":"<client-3>","event":"left","detail":"end"}   # the client closes the session
# server-side participant log
2026-09-26T10:39:53.410211Z  WARN moq_net::lite::subscriber: subscribe error broadcast=<room>/control/<client-3>/<session-3.1> track=control err=unroutable
2026-09-26T10:39:59.811220Z  WARN moq_net::lite::subscriber: track info failed broadcast=<room>/control/<client-3>/<session-3.1> track=control err=internal error
2026-09-26T10:40:28.829035Z  WARN moq_net::lite::subscriber: track info failed broadcast=<room>/control/<client-3>/<session-3.1> track=control err=unroutable
2026-09-26T10:40:30.905995Z  WARN moq_net::lite::subscriber: track info failed broadcast=<room>/control/<client-3>/<session-3.1> track=control err=unroutable
2026-09-26T10:40:31.056022Z  WARN moq_net::lite::subscriber: track info failed broadcast=<room>/control/<client-3>/<session-3.1> track=control err=unroutable
2026-09-26T10:40:31.435353Z  WARN moq_net::lite::subscriber: track info failed broadcast=<room>/control/<client-3>/<session-3.1> track=control err=unroutable
2026-09-26T10:40:32.678804Z  WARN moq_net::lite::subscriber: track info failed broadcast=<room>/control/<client-3>/<session-3.1> track=control err=unroutable
2026-09-26T10:40:54.407664Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-3>/<session-3.1> kind=Retracted
2026-09-26T10:40:55.997519Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-3>/<session-3.1> kind=Announced
2026-09-26T10:40:57.853054Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-3>/<session-3.1> kind=Retracted
The same event on one node: error and retraction 38 ms after the close
# client 3, its own event log (JSON, trimmed)
2026-09-26T10:53:12.206389Z  {"kind":"session","user":"<client-3>","event":"left"}   # the client closes the session
# server-side participant log
2026-09-26T10:53:12.244193Z  WARN moq_net::lite::subscriber: subscribe error broadcast=<room>/control/<client-3>/<session-3.3> track=control err=internal error
2026-09-26T10:53:12.244332Z  WARN moq_net::lite::subscriber: subscribe error broadcast=<room>/sources/<client-3>-s0/audio/<session-3.3> track=audio err=internal error
2026-09-26T10:53:12.245277Z  WARN app::discovery: reader_ended room=<room> path=<room>/sources/<client-3>-s0/audio/<session-3.3> code="container" error=container: moq: dropped
2026-09-26T10:53:12.245321Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-3>/<session-3.3> kind=Retracted
2026-09-26T10:53:12.245365Z  INFO app::discovery: announce room=<room> path=<room>/sources/<client-3>-s0/audio/<session-3.3> kind=Retracted
A closed session announced again 12 times, the last 153 s after the publisher closed
# server-side participant log
2026-09-26T10:39:03.906449Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Announced
# client 6, its own event log (JSON, trimmed)
2026-09-26T10:40:35.149956Z  {"kind":"session","user":"<client-6>","event":"left","detail":"end"}   # the client closes the session
# server-side participant log
2026-09-26T10:42:09.147148Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Retracted
2026-09-26T10:42:09.593019Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Announced
... (19 more lines, alternating Retracted and Announced) ...
2026-09-26T10:43:02.982781Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Announced
2026-09-26T10:43:04.041069Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Retracted
2026-09-26T10:43:08.174620Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Announced
2026-09-26T10:43:09.352609Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Retracted

Fuller sanitized traces (per-run counts and more excerpts) can be attached on request.

Activity

  1. added
    questBeing tracked/planned in a quest. See `quest/`
    on Sep 28, 2026
  2. fperex commented on Sep 29, 2026

    @fperex
    ContributorAuthor

    Cross-node routing on cdn.moq.pro: sanitized traces

    Sanitized excerpts and counts behind the report "Cross-node routing on cdn.moq.pro loses groups, stalls tracks and
    keeps closed broadcasts announced under bursty control traffic". Everything below was extracted from the run folders
    by script; nothing was re-run.

    • Runs: 2026-09-26, 10:38 to 12:06 UTC (the A/B/C comparison and two final trials). Section 6 also uses four
      single-node runs from 14:29 to 18:14 UTC the same day.
    • Clients and the server-side participant: native, moq-net 0.3.5 and moq-tokio 0.19.17 (with moq-mux 0.10.6 and
      moq-json 0.5.3), moq-lite-06 over QUIC, JWT with publish and subscribe path patterns. Browser clients
      (@moq/net 0.4.1) were used in other runs and are not part of these traces.
    • Relays: cdn.moq.pro (GeoDNS over several nodes) and, for run C, a self-hosted moq-relay 0.15.6, single node.
    • Topology: the server-side participant is about 0.7 ms from the OVH nodes its resolver returns (<moq-node-ip-2>,
      <moq-node-ip-3>). Ten clients run on one host about 40 ms away; GeoDNS gave that host the Linode node
      <moq-node-ip-1> at 10:38 to 11:00 and at 11:54 UTC and the OVH nodes at 11:09 to 11:40 UTC, so two cross-node runs
      pin the clients to <moq-node-ip-1> by address (TLS name cdn.moq.pro).

    Workload

    Each client join opens a fresh session and publishes two broadcasts under it: <room>/control/<client-n>/<session>
    with the track control (one group per message, one frame per group, bursts at up to 60 groups/s) and
    <room>/sources/<client-n>-s0/audio/<session> (20 ms audio groups). The server-side participant subscribes to every
    announced client broadcast, applies the messages and publishes <room>/mixer with the tracks state (JSON snapshot
    and deltas) and log, plus per-client audio outputs (<room>/mix-minus/<client-n>) and <room>/program-out/audio.
    When a later control group arrives before an earlier one it FETCHes the missing group and gives up after 2 s
    (control_fetch_deadline). Phases: contend (90 s, one session per client, all ten contending for the same control)
    and join-leave (90 s, every client joins and leaves about ten times, a new session path each time). The final
    trials add drag, storm, kill and revoke. "Reopen" is a client option (open a fresh session after the
    server-side participant reports the control track unreachable); it did not change the relay behaviour.

    Notation

    • <room> replaces the room prefix, <client-n> a client id, <session-n.k> the k-th session of client n in that
      run, <moq-node-ip-n> a node address. Paths keep their shape; the client logs them relative to the room prefix.
    • app::... lines are the server-side participant's own log targets; moq_net::, moq_mux:: and moq_json:: lines
      come from the moq crates. Journal lines are verbatim apart from the substitutions.
    • Client JSON lines (bots-*/bot-NN.jsonl) are trimmed to the relevant fields and prefixed with their own UTC
      timestamp (atUs). publishedAt is the server-side participant's publish time of a log entry (tEngineMs).
      engine/hops.jsonl lines are the server-side participant's record of each control group, prefixed with the time it
      received the group; fetched says whether the group came from a FETCH.
    • Clocks: the clients and the server-side participant run on different hosts, both NTP-synchronised; differences
      below about 50 ms between the two sides are not meaningful.
    • File names are relative to each run's own folder.

    Runs

    Label Start (UTC) Relay Server-side participant Clients Reopen
    A GeoDNS, reopen off 10:38:58 cdn.moq.pro OVH node by GeoDNS Linode node by GeoDNS off
    B one node, reopen off 10:51:23 cdn.moq.pro pinned to <moq-node-ip-2> pinned to <moq-node-ip-2> off
    C self-hosted, reopen off 10:55:50 moq-relay 0.15.6 same relay same relay off
    A GeoDNS, reopen on 11:00:15 cdn.moq.pro OVH node by GeoDNS GeoDNS (the clients' host still resolved the Linode node) on
    B one node, reopen on 11:09:46 cdn.moq.pro pinned to <moq-node-ip-2> pinned to <moq-node-ip-2> on
    C self-hosted, reopen on 11:14:13 moq-relay 0.15.6 same relay same relay on
    A clients pinned, reopen on 11:19:37 cdn.moq.pro OVH node by GeoDNS pinned to <moq-node-ip-1> (Linode) on
    A clients pinned, reopen off 11:24:11 cdn.moq.pro OVH node by GeoDNS pinned to <moq-node-ip-1> (Linode) off
    Final, GeoDNS 11:40:40 cdn.moq.pro GeoDNS (both OVH addresses returned) GeoDNS (the same two OVH addresses); the address each connection used was not recorded default
    Final, one node 11:54:29 cdn.moq.pro pinned to <moq-node-ip-2> pinned to <moq-node-ip-2> default

    Results

    Same binaries, ten clients, contend and join-leave phases. "Lost" is a message a client sent that never reached
    the server-side participant's log; "echo p99" is the time from a client sending a message to that client seeing it
    in state. Fetch failures and unreachable marks are the server-side participant's own counters over the phase
    (metrics.jsonl); an unreachable mark is a control subscription that failed while its path was announced.

    Run Reopen contend join-leave control fetches failed sessions marked unreachable
    A different nodes (GeoDNS) off FAIL: 2273 sent, 123 lost, echo p99 23322 ms FAIL: 395 sent, 19 lost, echo p99 25318 ms 153, 11 33, 96
    A different nodes (clients pinned) on FAIL: 2536 sent, 9 lost, echo p99 30775 ms FAIL: 383 sent, 14 lost, echo p99 5270 ms 8, 7 109, 14
    A different nodes (clients pinned) off FAIL (stopped after 51 s): 1421 sent, 71 lost, echo p99 2012 ms no client received a first state within 30 s 70 0
    A different nodes (GeoDNS) on no client received a first state within 30 s pass: 519 sent, 0 lost, echo p99 87 ms 0 7
    B same node (both pinned) off pass: 6310 sent, 0 lost, echo p99 91 ms pass: 536 sent, 0 lost, echo p99 142 ms 0, 0 0, 0
    B same node (both pinned) on pass: 6310 sent, 0 lost, echo p99 97 ms pass: 527 sent, 0 lost, echo p99 142 ms 0, 0 0, 0
    C self-hosted moq-relay 0.15.6 off pass: 6310 sent, 0 lost, echo p99 73 ms pass: 535 sent, 0 lost, echo p99 73 ms 0, 0 0, 0
    C self-hosted moq-relay 0.15.6 on pass: 6310 sent, 0 lost, echo p99 76 ms pass: 506 sent, 0 lost, echo p99 74 ms 0, 0 0, 0

    The final trials, six phases each (from converge-<phase>.json):

    Run drag contend storm join-leave kill revoke
    Final, GeoDNS pass: 22330 sent, 0 lost, echo p99 85 ms pass: 6310 sent, 0 lost, echo p99 81 ms FAIL: 15016 sent, 17 lost, echo p99 1268 ms FAIL: 62 sent, 25 lost, echo p99 6086 ms FAIL: 2441 sent, 0 lost, 554 more lost with killed clients, echo p99 1030 ms FAIL: 4491 sent, 255 lost, echo p99 1695 ms
    Final, one node pass: 22330 sent, 0 lost, echo p99 112 ms pass: 6310 sent, 0 lost, echo p99 97 ms pass: 14900 sent, 0 lost, echo p99 96 ms pass: 522 sent, 0 lost, echo p99 142 ms pass: 2890 sent, 0 lost, 106 more lost with killed clients, echo p99 214 ms pass: 20152 sent, 0 lost, echo p99 146 ms

    Counts computed from the files

    How each column was computed (Python over the run folders; every count covers the whole run):

    • Sessions: a client session id is (unix ms << 16) | random, so each path's session maps to the client join that
      created it; the session ends at that client's next left or killed event in its own bots-*/bot-NN.jsonl
      (or at the phase end if the client exited without one).
    • Announce events: app::discovery: announce ... kind=Announced|Retracted lines in engine-journal.txt. The
      server-side participant logs every Announced and Retracted update it receives for the room prefix (it does not
      log Updated). "After close" counts
      Announced events for a path more than 1 s after its client closed the session; "retraction delay" is the time
      from the client closing the session to the first Retracted for its control path.
    • Subscribe errors: moq_net::lite::subscriber lines subscribe error or track info failed with
      err=unroutable or err=internal error on a client control or sources path, compared with the same session's
      close time. "Stale" means more than 1 s after the close while the last announce event for the path was
      Announced.
    • Lost messages: sent records in the client files whose (client, cseq) never appears in engine/log.jsonl
      (this reproduces the harness's lost exactly for the A runs; the final GeoDNS trial also counts the 554 messages
      of killed clients). "Sent while connected" means sent at least 1 s before the client closed the session. The
      control group of a message is its index among the session's sent records (checked against engine/hops.jsonl:
      26 of 2481 mismatches in A GeoDNS reopen off, 0 in every other run). A "FETCH timeout" is a
      control_fetch_deadline line for that group while the client was still connected at the deadline. "No FETCH
      logged" means no fetch started line for the group; the journal holds fewer fetch started lines than the
      server-side participant's own FETCH counter (364 against 813 in A GeoDNS, reopen off), so read that column as an
      upper bound.
    • Burst start missed: sessions whose control subscription was already open (subscribe started) when the client
      sent its first burst, but whose first group delivered by that subscription (engine/hops.jsonl,
      fetched:false) was not group 0; the earlier groups of the burst were then FETCHed or lost.
    • log delay: for every log entry a client received for the first time and that was published after the client's
      current join, receive time (atUs) minus publish time (tEngineMs).
    • state Old: moq_json::snapshot::consumer: snapshot group lost ... track="state" ... error=Stream(Old) lines in
      bots-*.out (all ten clients write to one file).
    • No first state: client result lines in trial.out saying the first state did not arrive within 30 s.
    • Updated follows: followedOnUpdate in the summary at the end of probe.out: how often a test subscriber on the
      clients' host started reading a wanted broadcast because of an Updated announce (moq-net: "the route covering
      the path changed hops or cost") rather than an Announced one.

    Announcements

    Run client paths Announced Retracted Announced > 1 s after close (sessions, latest) retraction delay p50, p95
    A GeoDNS, reopen off 164 611 507 248 (32, 153 s) 57.68 s, 120.8 s
    A GeoDNS, reopen on 220 851 774 220 (36, 229 s) 0.05 s, 111.7 s
    A clients pinned, reopen on 176 633 526 254 (41, 155 s) 99.74 s, 138.5 s
    A clients pinned, reopen off 40 207 189 77 (8, 194 s) 185.34 s, 189.0 s
    Final, GeoDNS 292 1856 1782 720 (61, 373 s) 36.36 s, 335.6 s
    B one node, reopen off 220 220 220 0 0.04 s, 0.1 s
    B one node, reopen on 220 220 220 0 0.04 s, 0.1 s
    Final, one node 480 480 480 0 0.05 s, 0.4 s
    C self-hosted, reopen off 220 220 220 0 0.03 s, 0.1 s
    C self-hosted, reopen on 220 220 220 0 0.03 s, 0.2 s

    In the B, C and one-node runs every retraction slower than 1 s took exactly 30.0 to 30.1 s and belongs to a session
    that ended at a phase boundary, when the client process exits (consistent with a transport idle timeout); the
    cross-node delays are spread over tens of seconds to minutes.

    Subscribe errors on client paths

    Run total within 1 s after the client closed stale (> 1 s after close, still announced) before the client closed
    A GeoDNS, reopen off 369 88 260 0
    A GeoDNS, reopen on 437 70 344 0
    A clients pinned, reopen on 363 109 254 0
    A clients pinned, reopen off 405 37 356 0
    Final, GeoDNS 239 94 132 1
    B one node, reopen off 139 139 0 0
    B one node, reopen on 127 127 0 0
    Final, one node 319 317 2 0
    C self-hosted, reopen off 170 170 0 0
    C self-hosted, reopen on 172 172 0 0

    The one error counted before a close (final GeoDNS trial, kill phase, an audio path) belongs to a killed client
    whose log records no end for that session, so its close time defaults to the phase end.

    Lost control groups and FETCH

    Run lost sent while connected FETCH timeout, client still connected no FETCH logged FETCHes that returned (fetched:true groups) burst start missed: sessions, groups, FETCHed, lost
    A GeoDNS, reopen off 142 134 109 14 447 0
    A GeoDNS, reopen on 0 0 0 0 0 0
    A clients pinned, reopen on 23 23 9 8 251 0
    A clients pinned, reopen off 71 71 70 0 180 10, 92, 70, 22
    Final, GeoDNS 851 280 210 59 706 7, 27, 17, 10
    B one node, reopen off 0 0 0 0 0 0
    B one node, reopen on 0 0 0 0 0 0
    Final, one node 106 0 0 0 65 0
    C self-hosted, reopen off 0 0 0 0 0 0
    C self-hosted, reopen on 0 0 0 0 3 0

    In the final one-node trial all 106 lost messages belong to killed clients. A FETCH that works returns within tens
    to hundreds of milliseconds (37 to 803 ms for the ten groups in section 5), so the 2 s deadline is not
    the limiting factor.

    Server-side participant to clients

    Run log delay p99 log delay max log entries > 5 s state Old lines clients with no first state in 30 s Updated follows
    A GeoDNS, reopen off 23.90 s 25.1 s 482 651 0 117
    A GeoDNS, reopen on 1.01 s 1.0 s 0 144 10 46
    A clients pinned, reopen on 30.83 s 31.4 s 150 530 1 98
    A clients pinned, reopen off 0.84 s 1.0 s 0 85 10 208
    Final, GeoDNS 4.34 s 12.1 s 419 290 1 910
    B one node, reopen off 0.85 s 1.9 s 0 0 0 0
    B one node, reopen on 0.79 s 2.0 s 0 0 0 0
    Final, one node 0.84 s 1.8 s 0 0 0 0
    C self-hosted, reopen off 0.21 s 0.3 s 0 0 0 0
    C self-hosted, reopen on 0.22 s 0.3 s 0 0 0 0

    1. unroutable and internal error for announced client tracks

    Checked against each client's own log, every one of these answers in these runs came after the client had closed
    the session: none arrived while the publisher was connected (table above, last column). The failure is that the
    path stays announced, or is announced again, long after the publisher left, and every subscribe to it then fails.
    On one node the subscribe error and the retraction arrive together, about 40 ms after the client leaves.

    A GeoDNS, reopen off, phase contend, <client-3> session <session-3.1>:

    # bots-contend/bot-03.jsonl
    2026-09-26T10:39:39.375223Z  {"kind":"sent","user":"<client-3>","cseq":144,"intent":"resync"}   # its last message
    2026-09-26T10:39:53.113443Z  {"kind":"session","user":"<client-3>","event":"left","detail":"end"}   # the client closes the session
    # engine-journal.txt
    2026-09-26T10:39:53.410211Z  WARN moq_net::lite::subscriber: subscribe error broadcast=<room>/control/<client-3>/<session-3.1> track=control err=unroutable
    2026-09-26T10:39:59.811220Z  WARN moq_net::lite::subscriber: track info failed broadcast=<room>/control/<client-3>/<session-3.1> track=control err=internal error
    2026-09-26T10:40:28.829035Z  WARN moq_net::lite::subscriber: track info failed broadcast=<room>/control/<client-3>/<session-3.1> track=control err=unroutable
    2026-09-26T10:40:30.905995Z  WARN moq_net::lite::subscriber: track info failed broadcast=<room>/control/<client-3>/<session-3.1> track=control err=unroutable
    2026-09-26T10:40:31.056022Z  WARN moq_net::lite::subscriber: track info failed broadcast=<room>/control/<client-3>/<session-3.1> track=control err=unroutable
    2026-09-26T10:40:31.435353Z  WARN moq_net::lite::subscriber: track info failed broadcast=<room>/control/<client-3>/<session-3.1> track=control err=unroutable
    2026-09-26T10:40:32.678804Z  WARN moq_net::lite::subscriber: track info failed broadcast=<room>/control/<client-3>/<session-3.1> track=control err=unroutable
    2026-09-26T10:40:54.407664Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-3>/<session-3.1> kind=Retracted
    2026-09-26T10:40:55.997519Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-3>/<session-3.1> kind=Announced
    2026-09-26T10:40:57.853054Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-3>/<session-3.1> kind=Retracted
    

    The path stayed announced for 61 s after the close, answered 7 subscribes with unroutable or internal error,
    was retracted, announced again 1.6 s later and retracted again.

    For comparison, B one node, reopen off, phase join-leave, <client-3> session <session-3.3>:

    # bots-join-leave/bot-03.jsonl
    2026-09-26T10:53:12.206389Z  {"kind":"session","user":"<client-3>","event":"left"}   # the client closes the session
    # engine-journal.txt
    2026-09-26T10:53:12.244193Z  WARN moq_net::lite::subscriber: subscribe error broadcast=<room>/control/<client-3>/<session-3.3> track=control err=internal error
    2026-09-26T10:53:12.244332Z  WARN moq_net::lite::subscriber: subscribe error broadcast=<room>/sources/<client-3>-s0/audio/<session-3.3> track=audio err=internal error
    2026-09-26T10:53:12.245277Z  WARN app::discovery: reader_ended room=<room> path=<room>/sources/<client-3>-s0/audio/<session-3.3> code="container" error=container: moq: dropped
    2026-09-26T10:53:12.245321Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-3>/<session-3.3> kind=Retracted
    2026-09-26T10:53:12.245365Z  INFO app::discovery: announce room=<room> path=<room>/sources/<client-3>-s0/audio/<session-3.3> kind=Retracted
    

    2. Control groups lost; FETCH does not return them

    A GeoDNS, reopen off, phase contend, <client-3> session <session-3.1>, group 138 (its message cseq 139):

    # bots-contend/bot-03.jsonl
    2026-09-26T10:39:21.611733Z  {"kind":"sent","user":"<client-3>","cseq":139,"intent":"set"}   # group 138 of the session
    # engine/hops.jsonl
    2026-09-26T10:39:21.640Z  {"clientId":"<client-3>","group":137,"cseq":138,"outcome":"held_by","fetched":false}
    2026-09-26T10:39:21.676Z  {"clientId":"<client-3>","group":139,"cseq":140,"outcome":"control_gap","fetched":false}
    # engine-journal.txt
    2026-09-26T10:39:21.676257Z  INFO moq_net::lite::subscriber: fetch started broadcast=<room>/control/<client-3>/<session-3.1> track=control group=138
    2026-09-26T10:39:23.677134Z  WARN app::control: control_fetch_deadline user=<client-3> sequence=138
    # converge-contend.json, exactlyOnce.failures
    "<client-3> cseq 139 never reached the engine log"
    # bots-contend/bot-03.jsonl
    2026-09-26T10:39:53.113443Z  {"kind":"session","user":"<client-3>","event":"left","detail":"end"}   # the publisher stayed connected 29 s after the FETCH deadline
    

    Groups 137 and 139 arrived through the subscription 36 ms apart; group 138 did not, its FETCH did not return in
    2 s, and it never arrived although the publisher stayed connected. Across the cross-node runs this happened to
    109 (A GeoDNS, reopen off), 9 and 70 (A clients pinned, reopen on and off) and 210 (final GeoDNS) messages; B, C and the final
    one-node trial lost none while the publisher was connected. When the publisher goes idle after such a loss nothing
    later arrives to reveal the gap, so the subscriber cannot even start a FETCH ("no FETCH logged" column).

    3. Closed sessions announced again

    A GeoDNS, reopen off, <client-6> session <session-6.1>, which the client closed at the end of contend. Its control
    path stayed announced for 94 s, then was retracted and announced again 12 more times; the last Announced came
    153 s after the close:

    # bots-contend/bot-06.jsonl
    2026-09-26T10:40:35.149956Z  {"kind":"session","user":"<client-6>","event":"left","detail":"end"}   # the client closes the session
    # engine-journal.txt (control path only)
    2026-09-26T10:39:03.906449Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Announced
    2026-09-26T10:42:09.147148Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Retracted
    2026-09-26T10:42:09.593019Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Announced
    2026-09-26T10:42:23.201862Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Retracted
    2026-09-26T10:42:23.253459Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Announced
    2026-09-26T10:42:24.248804Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Retracted
    2026-09-26T10:42:25.296199Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Announced
    2026-09-26T10:42:25.393154Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Retracted
    2026-09-26T10:42:25.449944Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Announced
    2026-09-26T10:42:25.591936Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Retracted
    2026-09-26T10:42:25.591964Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Announced
    2026-09-26T10:42:25.683347Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Retracted
    2026-09-26T10:42:25.695784Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Announced
    2026-09-26T10:42:25.953760Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Retracted
    2026-09-26T10:42:25.989532Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Announced
    2026-09-26T10:42:26.150077Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Retracted
    2026-09-26T10:42:26.167028Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Announced
    2026-09-26T10:42:26.420725Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Retracted
    2026-09-26T10:42:46.654703Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Announced
    2026-09-26T10:42:56.765483Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Retracted
    2026-09-26T10:42:56.784240Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Announced
    2026-09-26T10:43:01.793980Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Retracted
    2026-09-26T10:43:02.982781Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Announced
    2026-09-26T10:43:04.041069Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Retracted
    2026-09-26T10:43:08.174620Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Announced
    2026-09-26T10:43:09.352609Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-6>/<session-6.1> kind=Retracted
    

    Totals are in the Announcements table: on one node and on the self-hosted relay every path is announced once and
    retracted once (220 and 220 for 220 paths); cross-node the same workload produced 611 Announced and 507
    Retracted events for 164 paths (A GeoDNS, reopen off) and 1856 and 1782 for 292 paths (final GeoDNS).

    4. state (and log) stop reaching clients on the other node

    A clients pinned, reopen on, phase contend. The server-side participant received and applied <client-6>'s
    messages within 0.2 s of sending (the client to server direction kept working), but the state and log updates
    they caused reached the client 30.7 s later, after 30.4 s with nothing arriving. The client stayed connected
    throughout; all ten clients show the same gap.

    # bots-contend/bot-06.jsonl
    2026-09-26T11:20:10.635930Z  {"kind":"sent","user":"<client-6>","cseq":208,"intent":"set"}
    2026-09-26T11:20:10.786862Z  {"kind":"sent","user":"<client-6>","cseq":212,"intent":"release"}
    2026-09-26T11:20:10.996834Z  {"kind":"echo","user":"<client-6>","cseq":208,"stateSeq":367}   # last `state` before the gap
    # engine/hops.jsonl
    2026-09-26T11:20:10.692Z  {"clientId":"<client-6>","group":208,"cseq":209,"outcome":"held_by","fetched":false}
    2026-09-26T11:20:10.951Z  {"clientId":"<client-6>","group":211,"cseq":212,"outcome":"not_holder","fetched":false}
    # bots-contend.out (first and last of 166 such lines in the gap, all ten clients)
    2026-09-26T11:20:11.385450Z  WARN moq_json::snapshot::consumer: snapshot group lost; waiting for a newer one track="state" group=30 error=Stream(Old)
    2026-09-26T11:20:41.129182Z  WARN moq_json::snapshot::consumer: snapshot group lost; waiting for a newer one track="state" group=60 error=Stream(Old)
    # bots-contend/bot-06.jsonl
    2026-09-26T11:20:41.421127Z  {"kind":"log","user":"<client-6>","seq":361,"publishedAt":"2026-09-26T11:20:10.580Z"}   # first `log` entry after the gap
    2026-09-26T11:20:41.452130Z  {"kind":"log","user":"<client-6>","seq":371,"publishedAt":"2026-09-26T11:20:10.960Z"}
    2026-09-26T11:20:41.445490Z  {"kind":"echo","user":"<client-6>","cseq":209,"stateSeq":371}   # first `state` after the gap
    2026-09-26T11:20:42.138390Z  {"kind":"session","user":"<client-6>","event":"left","detail":"end"}
    

    Each of state groups 30 to 60 (one group per second in this phase) reached the clients' consumer only as a
    lost group (Stream(Old)). The same pattern twice kept every client from receiving any first state for 30 s.
    A GeoDNS, reopen on, phase contend:

    # engine-journal.txt
    2026-09-26T11:00:22.011896Z  INFO moq_net::lite::publisher: subscribed started id=3 broadcast=<room>/mixer track=log
    2026-09-26T11:00:22.011921Z  INFO moq_net::lite::publisher: subscribed started id=2 broadcast=<room>/mixer track=state
    # bots-contend.out (first and last of 144 such lines, all ten clients)
    2026-09-26T11:00:22.377007Z  WARN moq_json::snapshot::consumer: snapshot group lost; waiting for a newer one track="state" group=3 error=Stream(Old)
    2026-09-26T11:00:49.354472Z  WARN moq_json::snapshot::consumer: snapshot group lost; waiting for a newer one track="state" group=30 error=Stream(Old)
    # bots-contend.out (the clients give up after 30 s; first of 10)
    2026-09-26T11:00:49.601483Z  WARN moq_net::lite::publisher: subscribed error id=2 broadcast=<room>/control/<client-6>/<session-6.1> track=control err=dropped
    # trial.out (client 3's result per phase; all ten clients failed `contend` the same way)
    bot  3 <client-3>     contend    sent      0 views   0 joins  0 log      0 echoes      0 deadline: the first mixer state did not happen within the deadline
    bot  3 <client-3>     join-leave sent     50 views   9 joins 10 log    120 echoes     50 ok
    

    The other no-first-state case is A clients pinned, reopen off, phase join-leave. Longest log delays per
    run, one line per client stall (client, publish time of the first delayed entry, longest delay):

    # A GeoDNS, reopen off, bots-*/bot-NN.jsonl
    <client-9>  first delayed entry published 2026-09-26T10:41:09.891Z  longest delay 25.1 s  (3 entries > 5 s)
    <client-4>  first delayed entry published 2026-09-26T10:39:15.411Z  longest delay 24.0 s  (37 entries > 5 s)
    <client-9>  first delayed entry published 2026-09-26T10:39:15.411Z  longest delay 23.9 s  (46 entries > 5 s)
    <client-1>  first delayed entry published 2026-09-26T10:39:15.411Z  longest delay 23.9 s  (41 entries > 5 s)
    <client-8>  first delayed entry published 2026-09-26T10:39:15.411Z  longest delay 23.9 s  (39 entries > 5 s)
    <client-2>  first delayed entry published 2026-09-26T10:39:15.411Z  longest delay 23.9 s  (42 entries > 5 s)
    # A clients pinned, reopen on, bots-*/bot-NN.jsonl
    <client-3>  first delayed entry published 2026-09-26T11:20:10.580Z  longest delay 31.4 s  (10 entries > 5 s)
    <client-4>  first delayed entry published 2026-09-26T11:20:10.580Z  longest delay 30.8 s  (11 entries > 5 s)
    <client-9>  first delayed entry published 2026-09-26T11:20:10.600Z  longest delay 30.8 s  (11 entries > 5 s)
    <client-5>  first delayed entry published 2026-09-26T11:20:10.600Z  longest delay 30.8 s  (11 entries > 5 s)
    <client-8>  first delayed entry published 2026-09-26T11:20:10.600Z  longest delay 30.8 s  (11 entries > 5 s)
    <client-7>  first delayed entry published 2026-09-26T11:20:10.600Z  longest delay 30.8 s  (11 entries > 5 s)
    # Final, GeoDNS, bots-*/bot-NN.jsonl
    <client-8>  first delayed entry published 2026-09-26T11:44:56.144Z  longest delay 12.1 s  (184 entries > 5 s)
    <client-6>  first delayed entry published 2026-09-26T11:44:06.144Z  longest delay 9.5 s  (142 entries > 5 s)
    <client-6>  first delayed entry published 2026-09-26T11:47:48.444Z  longest delay 6.9 s  (3 entries > 5 s)
    <client-7>  first delayed entry published 2026-09-26T11:47:48.564Z  longest delay 6.8 s  (1 entry > 5 s)
    <client-4>  first delayed entry published 2026-09-26T11:47:48.404Z  longest delay 6.7 s  (1 entry > 5 s)
    <client-6>  first delayed entry published 2026-09-26T11:43:56.204Z  longest delay 6.7 s  (73 entries > 5 s)
    

    5. The first groups of a burst miss an open subscription

    A clients pinned, reopen off, phase contend. The server-side participant subscribed to <client-0>'s control
    track 5.2 s before the client's first burst (the client waits for its first state). The subscription delivered
    group 5 first; groups 0 to 4 were FETCHed, the FETCHes did not return within 2 s, and the groups never arrived
    although the client stayed connected for 44 s after the burst:

    # engine-journal.txt
    2026-09-26T11:24:16.050264Z  INFO moq_net::lite::subscriber: subscribe started id=21 broadcast=<room>/control/<client-0>/<session-0.1> track=control
    # bots-contend/bot-00.jsonl
    2026-09-26T11:24:21.299313Z  {"kind":"sent","user":"<client-0>","cseq":1,"intent":"resync"}   # burst: cseq 1 to 14 within 0.1 ms
    2026-09-26T11:24:21.299366Z  {"kind":"sent","user":"<client-0>","cseq":14,"intent":"set"}
    # engine/hops.jsonl (first group delivered by the subscription)
    2026-09-26T11:24:21.584Z  {"clientId":"<client-0>","group":5,"cseq":6,"outcome":"resync_required","fetched":false}
    # engine-journal.txt
    2026-09-26T11:24:21.584985Z  INFO moq_net::lite::subscriber: fetch started broadcast=<room>/control/<client-0>/<session-0.1> track=control group=0
    2026-09-26T11:24:21.585007Z  INFO moq_net::lite::subscriber: fetch started broadcast=<room>/control/<client-0>/<session-0.1> track=control group=1
    2026-09-26T11:24:21.585013Z  INFO moq_net::lite::subscriber: fetch started broadcast=<room>/control/<client-0>/<session-0.1> track=control group=2
    2026-09-26T11:24:21.585017Z  INFO moq_net::lite::subscriber: fetch started broadcast=<room>/control/<client-0>/<session-0.1> track=control group=3
    2026-09-26T11:24:21.585022Z  INFO moq_net::lite::subscriber: fetch started broadcast=<room>/control/<client-0>/<session-0.1> track=control group=4
    2026-09-26T11:24:23.585159Z  WARN app::control: control_fetch_deadline user=<client-0> sequence=0
    2026-09-26T11:24:23.585181Z  WARN app::control: control_fetch_deadline user=<client-0> sequence=1
    2026-09-26T11:24:23.585186Z  WARN app::control: control_fetch_deadline user=<client-0> sequence=2
    2026-09-26T11:24:23.585191Z  WARN app::control: control_fetch_deadline user=<client-0> sequence=3
    2026-09-26T11:24:23.585196Z  WARN app::control: control_fetch_deadline user=<client-0> sequence=4
    # converge-contend.json, exactlyOnce.failures
    "<client-0> cseq 1 never reached the engine log"
    "<client-0> cseq 2 never reached the engine log"
    "<client-0> cseq 3 never reached the engine log"
    "<client-0> cseq 4 never reached the engine log"
    "<client-0> cseq 5 never reached the engine log"
    # bots-contend/bot-00.jsonl
    2026-09-26T11:25:05.954135Z  {"kind":"session","user":"<client-0>","event":"left","detail":"end"}
    

    In the same run <client-3>'s subscription (open 5.8 s before its burst) delivered group 10 first and FETCH
    recovered groups 0 to 9 within 37 to 803 ms:

    # engine-journal.txt
    2026-09-26T11:24:15.485999Z  INFO moq_net::lite::subscriber: subscribe started id=5 broadcast=<room>/control/<client-3>/<session-3.1> track=control
    # bots-contend/bot-03.jsonl
    2026-09-26T11:24:21.314977Z  {"kind":"sent","user":"<client-3>","cseq":1,"intent":"resync"}   # burst: cseq 1 to 14
    # engine/hops.jsonl
    2026-09-26T11:24:21.560Z  {"clientId":"<client-3>","group":10,"cseq":11,"outcome":"applied","fetched":false}
    2026-09-26T11:24:21.598Z  {"clientId":"<client-3>","group":5,"cseq":6,"outcome":"applied","fetched":true}
    2026-09-26T11:24:22.303Z  {"clientId":"<client-3>","group":0,"cseq":1,"outcome":"applied","fetched":true}
    2026-09-26T11:24:22.353Z  {"clientId":"<client-3>","group":1,"cseq":2,"outcome":"applied","fetched":true}
    # engine-journal.txt (first of ten)
    2026-09-26T11:24:21.560841Z  INFO moq_net::lite::subscriber: fetch started broadcast=<room>/control/<client-3>/<session-3.1> track=control group=0
    

    Across runs (Lost control groups table): A clients pinned, reopen off, 10 of 10 sessions, 92 groups, 70 FETCHed,
    22 lost; final GeoDNS 7 sessions, 27 groups, 17 FETCHed, 10 lost; none in any B, C or one-node run.

    6. The end of the output subscription after the last client leaves

    The traces show this is our own behaviour, not the relay's. The server-side participant ends its outputs 30 s after
    the last control track ends (live_ended, reason host_left). In the runs where the end came after 60 s, the
    departed clients' control tracks themselves ended 30 s after the clients left, the same 30 s that every relay,
    the self-hosted one included, takes when a client process exits at a phase boundary (see the Announcements note).
    The only relay-side difference: on cdn.moq.pro the test subscriber sees the clean end as internal error and
    its resubscribe as unroutable (the broadcast is gone by then, so the unroutable is correct); on the self-hosted
    relay the subscription completes without an error.

    Run last client left (UTC) output subscription ended subscriber saw
    B one node, reopen off 10:54:28.933 +30.1 s internal error, then unroutable
    B one node, reopen on 11:12:52.107 +30.1 s internal error, then unroutable
    Final, one node 12:03:35.732 +30.1 s internal error, then unroutable
    C self-hosted, reopen off 10:58:56.285 +30.0 s no error, clean end
    C self-hosted, reopen on 11:17:18.849 +30.1 s no error, clean end
    one node, 14:29 14:32:21.771 +60.0 s internal error, then unroutable
    one node, 14:56 14:59:51.347 +60.1 s internal error, then unroutable
    one node, 15:22 15:25:42.385 +60.1 s internal error, then unroutable
    one node, 17:59 18:12:02.704 +60.1 s internal error, then unroutable

    One node, 17:59 run (the server-side participant, the clients and the test subscriber pinned to <moq-node-ip-2>):

    # bots-storm2/bot-03.jsonl (all five clients left within 1 ms)
    2026-09-26T18:12:02.703858Z  {"kind":"session","user":"<client-3>","event":"left","detail":"end"}
    # engine-journal.txt
    2026-09-26T18:12:32.757266Z  WARN moq_net::lite::subscriber: subscribe error broadcast=<room>/control/<client-3>/<session-3.4> track=control err=internal error
    2026-09-26T18:12:32.757403Z  WARN app::discovery: reader_ended room=<room> path=<room>/control/<client-3>/<session-3.4> code="moq" error=moq: control subscription: dropped
    2026-09-26T18:12:32.757463Z  INFO app::discovery: announce room=<room> path=<room>/control/<client-3>/<session-3.4> kind=Retracted
    2026-09-26T18:13:02.771741Z  INFO moq_net::lite::publisher: subscribed complete id=0 broadcast=<room>/program-out/audio track=audio
    2026-09-26T18:13:02.772523Z  INFO moq_net::lite::publisher: subscribed complete id=8 broadcast=<room>/program-out/pcm track=audio
    # engine/log.jsonl
    2026-09-26T18:12:32.770Z  {"seq":68160,"op":{"type":"control_ended","userId":"<client-3>"}}
    2026-09-26T18:13:02.770Z  {"seq":68165,"op":{"type":"live_ended","reason":"host_left"}}
    # probe.out
    2026-09-26T18:13:02.792359Z  WARN moq_net::lite::subscriber: subscribe error broadcast=<room>/program-out/audio track=audio err=internal error
    2026-09-26T18:13:02.795223Z  WARN moq_net::lite::subscriber: subscribe error broadcast=<room>/program-out/pcm track=audio err=internal error
    # probe.out, summary
    "program": {"ended": "moq: request stream: unroutable", "resubscribed": 1}
    "program/pcm": {"ended": "moq: request stream: unroutable", "resubscribed": 1}
    

    Journal lines are ordered by time; live_ended in engine/log.jsonl is the server-side participant's decision
    and precedes the subscribed complete lines by 1 to 2 ms.

    7. A re-created broadcast arrives as Updated

    The server-side participant's output broadcasts, for example <room>/mix-minus/<client-n>, are finished and published
    again as clients leave and join. A test subscriber on the clients' host follows them through the announce stream. On one node and on the
    self-hosted relay it always learned of the new broadcast through Announced; across nodes it often got Updated
    for a path whose previous broadcast it had already stopped reading. In moq-net Updated means the route covering a
    path the subscriber already had changed, so its announce stream never saw the old broadcast retracted. A subscriber
    that only acts on Announced misses those broadcasts. From the summary at the end of each probe.out:

    Run followedOnUpdate
    A GeoDNS, reopen off 117
    A GeoDNS, reopen on 46
    A clients pinned, reopen on 98
    A clients pinned, reopen off 208
    Final, GeoDNS 910
    B one node, reopen off 0
    B one node, reopen on 0
    Final, one node 0
    C self-hosted, reopen off 0
    C self-hosted, reopen on 0

    The subscriber does not log each announce, so there are no per-event lines for this class.

  3. kixelated commented on Oct 6, 2026

    @kixelated
    Collaborator

    Thanks for the detailed report and traces. We tried to reproduce this on a local two-relay cluster, using one-frame groups in bursts of 14, a subscription opened 5 s before the first burst, fetch-on-gap with a 2 s deadline, and a flapping peer link. We ran it against the 0.15.6 relay tree your clients ran, and against current main.

    Could you re-run your A/B/C comparison against current cdn.moq.pro? If you can share the harness, even a minimal version, that would help a lot. For any FETCH that fails, please also note whether it ended in an error or a timeout.

    (Written by Claude Opus 5.5)

  4. fperex commented on Oct 11, 2026

    @fperex
    ContributorAuthor

    Thanks for digging into this. Here is the re-run against current cdn.moq.pro (2026-10-11 UTC), with the harness and the traces.

    Short version

    • Nothing in this workload is lost any more, cross-node or not: 0 lost messages in every phase of every run, on the old client train and the new one.
    • No FETCH failed. Every one of 19,360 per-request FETCH records from the main runs (plus 11,631 from the six-phase and extra runs) ended ok: 0 refused, 0 unreadable, 0 deadline. After a deadline we keep the FETCH open to 10 s to tell late from never; both are 0. So there is no error-or-timeout split to report.
    • Closed broadcasts no longer stay announced for minutes. The first Retracted comes about 0.2 s after a clean close, and no path stays announced at the end of any run.
    • What still reproduces, only when the two sides are on different nodes: right after a clean close, the closed path is announced and retracted again hundreds of times for about 2 s. On one node, on a self-hosted moq-relay 0.17.2 and on a self-hosted two-relay 0.17.2 cluster, this never happens.

    Setup

    • New train: moq-net 0.3.10, moq-tokio 0.19.23, three runs per cell. Old train: moq-net 0.3.5, moq-tokio 0.19.17 (the original report's), one run per cell. Self-hosted relay: the moq-relay 0.17.2 release binary. moq-lite-06 over QUIC, JWT with publish and subscribe patterns.
    • Same client and server-side participant as the original report. The server side runs on a host in Miami, 0.5 ms from 40.160.233.192; the ten clients and a test subscriber run on a second host about 60 ms from that node.
    • A-geo: both sides through GeoDNS (both got the two Miami nodes 40.160.233.192 and 40.160.250.184). A-pin: server pinned to 40.160.233.192, clients to 139.177.201.247, by address with TLS name cdn.moq.pro. B: both pinned to 40.160.233.192. C: one self-hosted relay. D: a self-hosted two-relay cluster, server on relay 1 and clients on relay 2.
    • 10 clients, 90 s phases contend and join-leave, with the clients' state reopen off and on. Plus one six-phase run (drag, contend, storm, join-leave, kill, revoke) on A-pin and on B.

    Loss, latency and FETCH, new train (mean of three runs, range in brackets; FETCH records are summed over the three runs)

    Run reopen contend lost / echo p99 ms join-leave lost / echo p99 ms FETCH records not ok
    A-geo off 0 / 330 (105 to 503) 0 / 405 (216 to 592) 3801 0
    A-geo on 0 / 246 (169 to 384) 0 / 183 (168 to 197) 2257 0
    A-pin off 0 / 337 (127 to 484) 0 / 263 (198 to 309) 3310 0
    A-pin on 0 / 316 (166 to 396) 0 / 292 (176 to 398) 2108 0
    B off 0 / 129 (111 to 163) 0 / 274 (230 to 299) 2315 0
    B on 0 / 240 (110 to 398) 0 / 264 (192 to 300) 2531 0
    C off 0 / 99 (87 to 104) 0 / 100 (90 to 106) 1102 0
    C on 0 / 98 (86 to 103) 0 / 89 (74 to 102) 821 0
    D off 0 / 267 (222 to 302) 0 / 310 (307 to 312) 737 0
    D on 0 / 255 (230 to 292) 0 / 310 (302 to 316) 378 0

    The six-phase runs lose nothing in any phase either. FETCH records: A-pin 6899, B 4524, all ok.

    Old train on today's cdn.moq.pro: also 0 lost and 0 failed FETCH. The old client only has aggregate counters, so there are no per-request records.

    Run reopen contend lost / echo p99 ms join-leave lost / echo p99 ms
    A-pin off 0 / 396 0 / 307
    A-pin on 0 / 120 0 / 202
    B off 0 / 697 0 / 162
    B on 0 / 202 0 / 514

    Announcements

    Run Announced Retracted Updated Announced/Updated > 1 s after a clean close sessions affected latest after the close
    A-geo off, new 4875 (2784 to 6099) 4875 145 (102 to 173) 1840 (1411 to 2337) 5 2.0 s
    A-geo on, new 3629 (2288 to 4810) 3629 144 (112 to 171) 2092 (1345 to 2480) 8 2.1 s
    A-pin off, new 4759 (3678 to 6166) 4759 174 (153 to 190) 1903 (1527 to 2584) 5 1.8 s
    A-pin on, new 5263 (4251 to 6072) 5263 180 (179 to 181) 2445 (2182 to 2606) 9 1.8 s
    B, C and D, new (all 18 runs) 220 220 0 0 0
    A-pin off, old 5596 5596 1019 4 1.9 s
    A-pin on, old 3430 3430 1661 6 1.8 s
    B off and on, old 220 220 0 0

    220 is one Announced and one Retracted for each broadcast the clients open in a run. The six-phase run on A-pin has 11,540 Announced, 388 Updated and 4575 events after a close on 17 sessions; on B it has 480, 0 and 0.

    A typical cross-node sequence (new train, A-pin, reopen off, run 2, one client's ninth session, times from its clean close):

    +0.228 s Retracted        (the control subscription gets `internal error` at the same moment)
    +0.518 s Announced
    +0.624 s Retracted
    +0.640 s Announced
    +0.669 s Retracted
    ... 1140 more, alternating ...
    +1.757 s Announced
    +1.757 s Retracted        (last event)
    

    The same thing happens with GeoDNS and on the old train. While it lasts, subscribes to the path answer internal error or unroutable. In join-leave, internal error on control happens 64 to 99 times per cross-node run, against 18 to 21 on B, 12 to 15 on C and 3 to 4 on D. In B, C and D, one or two sessions per run get their first Retracted 10.0 s after the close (it looks like the transport idle timeout); the cross-node runs have that too.

    Your suggestion (A-pin, reopen off, one run each)

    Server setting FETCH started (contend) lost Announced/Updated > 1 s after a close
    as before 89 0 1344
    250 ms wait before FETCH 0 0 2748
    control max age 120 s 115 0 2631
    both 0 0 1890

    With the wait, a group that arrives in the meantime is not fetched. A 250 ms wait removes the FETCHes entirely, and the harness shows the same: 12,341 to 0 in contend and 3971 to 0 in join-leave. A 120 s max age changes nothing in either. Neither setting changes the re-announcing.

    One thing on our own cluster (D) we could not explain. In 2 of the 6 D runs, 3 clients in total got no first state group within 30 s on their last rejoin. In one of them, the relay with the clients logged subscribe canceled (idle) track=state when the client's previous session left (+176.0 s). The server accepted the new state subscribe at +177.5 s, the client logged read log: old at +177.6 s, and its state never arrived. It did not happen on A, B or C, or on the other four D runs. We did not root-cause it. The details are in the traces.

    Not covered: #4741 is not in a release yet, so these runs do not include it.

    Harness: https://github.com/fperex/moq-burst-repro

    • burst-repro is a single binary with three subcommands: server subscribes through announcements, FETCHes gaps with a 2 s deadline and echoes on state; clients runs N publishers with contend or join-leave; report prints the tables from the result files.
    • It uses published crates only (moq-net 0.3.10, moq-tokio 0.19.23).
    • The README has the exact commands for A, B and C.
    • With it (server in Miami, ten clients in Silicon Valley, three runs per cell): 0 lost, 0 failed FETCH and 0 state stalls in every run.
    • join-leave with the clients on the Linode node gives 4737 (3578 to 5419) Announced/Updated after a close, the latest about 2 s after it. With the clients on the Silicon Valley node 40.160.235.183 it gives 33 (0 to 100), and on B and C it gives 0.

    Traces: in traces/ in the same repository.

    • traces.md: every table, the per-session announcement sequences and the D details.
    • fetches-sample-*.jsonl: the per-FETCH record format.
    • harness-reports.txt: the harness report of every run.
    • harness-raw.tar.xz: the raw harness files of five join-leave runs.
  5. added a commit that references this issue on Oct 11, 2026
    65f1bb7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    questBeing tracked/planned in a quest. See `quest/`

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions