Skip to content

fix(relay): redial a cluster peer from the first backoff ceiling once an attempt connected (backport) - #9104

Merged
kixelated merged 1 commit into
releasefrom
backport/relay-backoff
Nov 18, 2026
Merged

kixelated merged 1 commit into
releasefrom
backport/relay-backoff

Conversation

@kixelated

@kixelated kixelated commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Backport of the redial backoff reset from moq-dev/moq#9055 to release, for quest quest/m1/precut-fixes/relay-backoff-backport.md (claimed by moq-dev/moq#9100).

Problem

A relay redials a cluster peer through Redial, whose ceiling doubles to 30 s. run_remote reset it only on Ok(()) if elapsed >= stable_threshold, but run_remote_once never returns Ok: its session loop runs until reconnect.status() errors. So the ceiling never reset, and a peer that flapped after a long outage waited out a wait drawn below 30 s.

Changes

rs/moq-relay/src/cluster.rs:

  • run_remote_once and run_remote_session return Result<Infallible> and set connected once reconnect.status() reports Status::Connected.
  • Redial::next(connected, rng) starts over at the first ceiling when the attempt connected. The dead stable_threshold branch is gone.
  • New test redial_resets_after_an_attempt_that_connected.

Only the backoff reset is ported, as the maintainer decided when picking #9055. The two startup refusals (cluster.api.root without cluster.api.url, cluster.token with no peer to dial) stay on main.

Differences from #9055

release has no token rotation retry, panic isolation, or dial.observe in this path, so:

  • "Connected" is Status::Connected from reconnect.status(), the release equivalent of main's observation.connects > 0.
  • No token-rotation reset to port: release has no immediate retry on rotation.
  • main's drill mutation peer-link-never-redialed.patch does not exist on release.

Impact

No public API or wire change: Redial and the session functions are private.

Validation

  • cargo test -p moq-relay: all pass, including the new test. cargo clippy -p moq-relay --all-targets -D warnings and cargo fmt --check clean.
  • End to end on a scratch copy of tests/cluster_drain.rs (paused clock, counting forwarder, not committed): relay A dials B through a forwarder that refuses for 600 s, B comes up and A connects, then B shuts down and stays down. Gap from the inner reconnect giving up to the next outer redial:
    • this branch: 0.8 s (later gaps grow, 17.5 s and 20.2 s, as B stays down).
    • origin/release: 14.5 s, drawn below the 30 s ceiling the outage built.

Caused by

No single PR: the backoff branch was dead when written (see #9055).

Models

Size: XS
Implementer: Claude Code, claude-opus-5-5, medium effort

🤖 Generated with Claude Code

(Written by Claude Opus 5.5 with medium)

Review

Rewritten by quest review after each round.

  • Head: 83ab8b9

  • Base: origin/release at c636e03

  • Risk: medium, the base has no RISK.md to name what is always high or customer-facing, so every change is medium

  • Classified: medium by default

  • Reviewer: Codex, gpt-6.1-sol, medium effort

  • Session: 01a1f048-50bd-7370-87b2-4f765eb14f2a, round 1

  • Score: 9/10, Source review found no actionable defects in reconnect backoff handling, but builds and tests were not run as instructed.

  • Rounds: 1; first round 0; findings 0: fixed 0, refuted 0, open 0

  • Findings: none

… an attempt connected (backport)

Backport of the redial backoff reset from #9055.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Approach: sound. Because run_remote_session can never return Ok, the old stable_threshold reset never ran. The PR deletes that dead branch and resets Redial when an attempt actually reached Status::Connected, which matches what #9055 does on main with the signal release has.

Reviewed changes

I reviewed the one-commit backport to rs/moq-relay/src/cluster.rs.

  • Infallible session result: run_remote_once and run_remote_session now return anyhow::Result<Infallible>, so the compiler enforces that an attempt only ends in an error. The let Err(err) = … in run_remote is irrefutable.
  • connected out-param: this is set in the Status::Connected if connection.is_none() arm, and run_remote resets it to false before each attempt.
  • Redial::next(connected, rng): this resets to Redial::default() before drawing, so the first wait after a connected attempt is at most FIRST and the ceiling then doubles as before.
  • Test: the new redial_resets_after_an_attempt_that_connected drives the ceiling to CAP and then checks the reset. The existing jitter test now passes false.

A peer that accepts and then fails right away, such as a redirect straight after the handshake, now also resets the outer ceiling. The inner moq_tokio reconnect's own backoff and give-up still bound how fast that can repeat, and main behaves the same way, so this is not a regression.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

@kixelated
kixelated merged commit 6a26878 into release Nov 18, 2026
34 of 37 checks passed
@kixelated
kixelated deleted the backport/relay-backoff branch November 18, 2026 20:47
kixelated added a commit that referenced this pull request Nov 18, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
shermerL pushed a commit to shermerL/moq that referenced this pull request Nov 19, 2026
… (moq-dev#9100)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant