Skip to content

relay: a restarted relay can't stateless-reset its predecessor's connections, so clients wait out the idle timeout #5861

Closed

Description

@t0ms

Measured on main at 3c0daa2: we SIGKILL a moq-relay and a supervisor restarts it on the same
address 1 s later. Its clients don't learn that their connections are gone until the QUIC idle timeout:
moq import ts logs session closed, reconnecting 10.05 s after the kill, and moq export ts times
out at 13 s, then exits with broadcast did not return and is restarted. Over three replicates in a
namespace rig at 20 ms RTT, the exporter's output loses 12.0–14.2 s of video per relay restart.
An SRT relay restarted the same way costs 6.3 s, bounded by libsrt's 5 s peer idle timeout.

The cause, read on today's main (8bcbc7d): both listeners build their endpoint from
moq_quic::EndpointConfig::default() (rs/moq-tokio/src/quic/backend.rs, and
rs/moq-uring/src/quic/driver/mod.rs for io_uring), which draws a random stateless-reset key per
process (rs/moq-quic/src/config/mod.rs). When the restarted relay receives the clients' packets for
connection IDs it doesn't know, the reset tokens it can compute don't match those its predecessor
issued, so the clients ignore them (RFC 9000 §10.3) and wait for the idle timer.

Lowering the idle timeout works around it (4.4–5.1 s lost with MOQ_QUIC_IDLE_TIMEOUT=3s and
MOQ_QUIC_KEEP_ALIVE=1s), but it trades against riding through transport outages, where the
timeout has to exceed the longest outage a route should survive. A stable reset key removes that
trade-off for restarts:

  • derive reset_key from a configured secret (a flag or file), or from the TLS key when the
    certificate is persistent;
  • across a sharded or QUIC-LB cluster, one shared key would let any node reset a dead peer's
    connections.

With it, a publisher (which sends continuously) would learn within one RTT of the relay coming back,
and a subscriber within one keep-alive interval.

Happy to build it if it's wanted.

(Written by Claude Opus 5.5)

Activity

  1. added
    questBeing tracked/planned in a quest. See `quest/`
    on Oct 19, 2026
  2. kixelated commented on Oct 19, 2026

    @kixelated
    Collaborator

    Decision: land #5910 as the fix. A file secret (listen.reset_key) is HKDF-derived into both the reset key and the hashed connection ID key; an unsharded tokio server hashes CIDs per process too, so the reset key alone wasn't enough. A TLS-derived key was rejected because certificate rotation would change it silently.

    Tracked by quest/m1/stateless-reset-key.md; #5910 is its implementation.

    Planned in #5969. Anyone who wants to donate tokens can run /quest-start quest/m1/stateless-reset-key to pick it up.

    (Written by Claude Opus 5.5)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    questBeing tracked/planned in a quest. See `quest/`

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions