Skip to content

fix(net): a reconnecting publisher wins over its lingering session on release - #6435

Closed
kixelated wants to merge 2 commits into
releasefrom
fix/release-reconnect-takeover
Closed

kixelated wants to merge 2 commits into
releasefrom
fix/release-reconnect-takeover

Conversation

@kixelated

Copy link
Copy Markdown
Collaborator

Problem

A publisher that drops without a clean close (a flaky uplink, a SIGKILL) and reconnects leaves its old session lingering on the relay until QUIC idles it out. The new session announces the same path under a new first hop: the client's own random id, or this relay's per-connection stamp for an unnamed publisher. route_order ranks the two by the rendezvous hash before the newest-announcement tiebreak, so about half the time the dead session keeps winning. Viewers who subscribe in that window land on the dead broadcast: they get its cached tail (catalog plus the last group or two) and then nothing, or an immediate clean end.

Reported against moq.pro (on release), not reproducible on staging (on main, where a fresh epoch per publisher run ranks first).

Change

route_order hashes only routes a cluster peer forwarded (RouteEntry::peer). A route that entered on this relay is ranked by no other node, so convergence does not need the hash there, and it now falls through to Reverse(entry.id): the newest announcement wins the moment it lands, as the doc comment already promised. Peer-forwarded routes rank exactly as before.

Side effect: at an otherwise equal rank, a directly ingested route now beats a peer-forwarded one (hash 0 versus a hash). A peer route's chain is at least one hop longer, so this only matters for an exact tie.

Verification

  • New a_reconnect_under_a_new_hop_wins_over_the_old_session covers named ([10] then [11], and reversed) and stamped ([10, 0] then [11, 0], and reversed) reconnects. It fails without the change ([11] never took over from [10]).
  • cargo test -p moq-net -p moq-relay: passes. cargo clippy -p moq-net --all-targets -D warnings: clean. Full just check origin/release not run.
  • End to end, a local moq-relay with moq 0.14.2 publishing synthetic H.264 and killed/restarted each round, a subscriber dialing 1s after each restart and held 12s:
    • live's pin (f1a1deb3b): 6/9 subscribers got only stale bytes or an early end.
    • this branch: 0/12.
  • The same harness against moq.pro's fleet reproduced it across PoPs (fra0 to sjc0, iad0 to dal0) and on a single edge.

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

… release

A publisher that drops without a clean close and reconnects announces under a
new first hop: its own random id, or this relay's per-connection stamp. Until
the old session idles out, route_order's rendezvous hash picks between the dead
and live session by coin flip, so new subscribers land on the dead broadcast and
get its cached tail, then nothing. Hash only routes a cluster peer forwarded;
a route that entered here falls through to the newest announcement.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-23T01:18:33.054964Z 9165f5b PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9165f5be2d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +662 to +664
match entry.peer {
true => fnv_key(prefix.as_str(), entry.hops.iter().copied()),
false => 0,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Update the public route-order documentation

For routes inserted through a non-peer Producer, this now replaces the hash with zero and orders equal candidates directly by recency, but the public Route::cost documentation still states that a deterministic hash always precedes the newest-announcement tie-break. This makes the documented, externally observable selection behavior incorrect for ordinary publisher routes; document the new peer-only hashing distinction alongside this change.

AGENTS.md reference: AGENTS.md:L27-L28

Useful? React with 👍 / 👎.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Closing without backporting. This changes route selection for every directly connected publisher on live, and release code can't be soaked on staging first. The benefit is small: only viewers who join within the old session's idle timeout after an unclean publisher drop are affected. The next release cut from main fixes it for publishers that send an epoch. The repro and analysis are in the description if this comes back.

(Written by Claude Opus 5.5)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant