Skip to content

fix(tokio): escalate the backoff when GOAWAY redirects loop between peers - #6162

Merged
kixelated merged 7 commits into
mainfrom
quest/m1/tokio-redirect-loop
Oct 21, 2026
Merged

kixelated merged 7 commits into
mainfrom
quest/m1/tokio-redirect-loop

Conversation

@kixelated

@kixelated kixelated commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Completes quest/m1/tokio-redirect-loop.md, mirroring moq-dev/moq#6115 in Rust.

Problem

moq-tokio's reconnect loop only damped a GOAWAY redirect that arrived within backoff.initial of connecting. Two relays that each redirect the client to the other after a session outlives that delay bounce it between them forever: every hop scores as healthy, Retry::settle resets the delay and the retry window, and no error is ever reported.

Approach (same rule as JS)

  • While migrations are chained, the loop tracks which endpoints sent it elsewhere (Redirects). A redirect back to one of them skips the healthy-session reset, so the delay escalates to backoff.max and backoff.timeout still ends the run, with the error redirect loop (the JS name). The redirect itself is still honoured.
  • Brent's cycle detection keeps the state constant however many distinct redirects a hostile peer names, and catches a cycle of any length.
  • Endpoints compare without their userinfo, query, or fragment, so a relay minting a fresh credential per handoff can't hide a loop.
  • Only an accepted redirect is a move. An empty or ignored URI, or a GOAWAY naming the current endpoint, is a restart in place and never counts.
  • The run resets when a session ends without a GOAWAY, or when a session outlived backoff.max (it already redials no faster than the backoff would).

Tests

  • redirect_loop_detection_catches_any_cycle_in_constant_memory and an_endpoint_ignores_the_credentials: pure unit tests, ported from JS.
  • tests/integration/reconnect.rs: two relays ping-ponging (with and without rotating credentials) escalate through the jittered backoff; a loop ends at the retry window with redirect loop; a loop no faster than the cap, a one-way rolling restart, and same-place GOAWAYs (named and empty) never back off. Those last cases use a retry window shorter than a session's life, so a wrongly scored hop ends the connection deterministically instead of relying on an upper time bound.
  • Mutations: dropping the loop flag, the cap reset, credential stripping, or the same-place check each fails a test.

These run on the wall clock with lower bounds, not on tokio's paused clock as the quest suggested. Real moq-tokio sessions over TCP or Unix sockets aren't transparent to a paused clock: virtual time jumps by seconds while the runtime waits on socket I/O. rs/AGENTS.md allows wall-clock tests with lower bounds for real sockets. The four loop tests take about 5s in total.

Impact

  • Public API: none. Only the Backoff doc comments change to describe the loop behaviour.
  • Wire: none.
  • Behaviour: a redirect back to an endpoint that already sent the client elsewhere, off a session that lived less than backoff.max, now waits out the jittered backoff and counts against backoff.timeout, which can end the connection with redirect loop. The draining session keeps serving during that wait, up to its handover cap.
  • One small difference from JS: Rust's retry window counts from the last healthy migration, while JS's starts at the first looping hop, so Rust can give up one session life sooner. Both bound the loop.

Verification

  • just check: passes.

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

kixelated and others added 5 commits October 20, 2026 22:51
…eers

Mirrors #6115: while migrations are chained, a redirect back to an
endpoint that already sent the client elsewhere skips the healthy-session
reset, so the backoff escalates to its cap and the retry window ends the loop
with a redirect-loop error. Brent's cycle detection keeps the state constant,
endpoints compare without their query or fragment, and a session that outlived
the capped delay starts a new run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 33 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 5dd41603-fd25-4696-87e4-1532013cbc78
📥 Commits

Reviewing files that changed from the base of the PR and between a42689d and fd26f71.

📒 Files selected for processing: 5
  • quest/m1/README.md
  • quest/m1/js-reload-userinfo.md
  • quest/m1/tokio-redirect-loop.md
  • rs/moq-tokio/src/connection.rs
  • rs/moq-tokio/tests/integration/reconnect.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-21T18:00:07.436019Z fd26f71 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a43b27e1da

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +1365 to +1369
fn endpoint(url: &Url) -> Url {
let mut bare = url.clone();
bare.set_query(None);
bare.set_fragment(None);
bare

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Strip non-dial credentials from redirect endpoint keys

For host-based transports, this retains URL userinfo and path even though userinfo is ignored by dialing and paths may carry rotating credentials. A bouncing pair can therefore return redirects such as tcp://token-N@peer/ on every healthy session; each key remains unique, Redirects::depart never detects the revisit, and the retry timeout keeps resetting, recreating the unbounded redirect loop this change is intended to stop. Build the comparison key from the actual transport endpoint (scheme, host, and effective port), preserving the path only for hostless transports such as Unix sockets.

AGENTS.md reference: AGENTS.md:L18-L18

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Userinfo: fixed in 259e604. It never reaches a dial, so endpoint now strips it along with the query and fragment, and the test covers it.

Path: kept, as in JS (moq-dev/moq#6115). For host-based transports the path is the request target (the WebTransport request, or the SETUP path for moqt/moql/tcp/tls), so relay/a and relay/b are different requests rather than credentials. A hostile peer that wants to dodge detection can already rotate ports under the default same-host policy, so no key catches every evasion. What bounds such a peer is that each hop needs a session outliving backoff.initial. Loop detection is there for fleets that bounce clients by accident, not as a bound on hostile peers.

(Written by Claude Opus 5.5)

Userinfo never reaches a dial, so a peer rotating it could hide a loop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

@codex review

(Written by Claude Opus 5.5)

1 similar comment
@kixelated

Copy link
Copy Markdown
Collaborator Author

@codex review

(Written by Claude Opus 5.5)

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: 259e604eb9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@kixelated
kixelated added this pull request to the merge queue Oct 21, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Oct 21, 2026
@kixelated
kixelated added this pull request to the merge queue Oct 21, 2026
Merged via the queue into main with commit cbab2a8 Oct 21, 2026
11 checks passed
@kixelated
kixelated deleted the quest/m1/tokio-redirect-loop branch October 21, 2026 23:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant