Skip to content

fix(shaper): keep forwarding after a peer's port goes away - #5796

Merged
kixelated merged 6 commits into
mainfrom
quest/m0/ci/platform-windows-drill
Oct 17, 2026
Merged

kixelated merged 6 commits into
mainfrom
quest/m0/ci/platform-windows-drill

Conversation

@kixelated

@kixelated kixelated commented Oct 16, 2026 •

Copy link
Copy Markdown
Collaborator

Quest: quest/m0/ci/platform-windows-drill.md (deleted here; references removed from quest/m0/ci/README.md and quest/m0/ci/platform-queue.md).

Root cause

The Windows failure is in the test's UDP shaper, not the relay or the reconnect loop.

Windows reports an ICMP port unreachable as WSAECONNRESET (io::ErrorKind::ConnectionReset) on the next recv_from, even on an unconnected UDP socket. Linux and macOS stay silent there. moq-shaper treated any receive error as fatal:

  1. The drill kills the relay. Client keep-alives keep arriving at the shaper, which forwards them to the dead port.
  2. The flow socket's next recv_from returns ConnectionReset. The flow's reply task fails, run propagates it through join_next, and the whole shaper stops forwarding.
  3. Both clients still reach Disconnected (idle timeout). After retarget, nothing forwards, so neither reaches Connected. That matches the panic in expect_status at drills.rs:661, in both lanes, since test(relay): restart the crash drill on a fresh port behind a stable address #5160 put both lanes behind a shaper.

The shaper only records the error in failed, which verify() reads at the end of the drill. The panic comes first, so the CI log never shows the error.

The QUIC driver already skips ConnectionReset on receive (rs/moq-tokio/src/quic/driver/endpoint.rs). The shaper now does the same on both receive paths: the listening socket (a closed client) and each flow's socket (a dead target).

When it started: Windows ran no tests until #5557 (2026-10-15) fixed the clippy-driver path, and Platform moved into the merge queue in #5488. The bug itself goes back to the drill's shaper path (#4054, then #5160 for the loopback lane). I couldn't confirm this from the Windows job log: the REST endpoint for job logs stayed rate-limited.

Regression test

The shaper's unit tests run on an in-memory socket (mem.rs). It now reports a send to an unbound port the way Windows does: the sender's next receive fails with ConnectionReset. That stricter model applies on every platform, so Linux CI catches a regression. All existing tests pass under it. Two new tests fail without the fix (one times out, one gets an empty echo) and pass with it:

  • a_closed_target_loses_datagrams_until_a_retarget
  • a_closed_client_leaves_the_others_forwarding

On Windows, the drill itself (moq-relay::drills relay_killed_mid_group_aborts_then_resumes, both lanes) is the end-to-end check in the merge queue.

Checks

  • cargo nextest run -p moq-shaper: pass (the new tests fail before the fix)
  • cargo nextest run -p moq-relay --test drills on Linux: 14/14 pass
  • cargo clippy -p moq-shaper --all-targets --all-features -D warnings, cargo fmt --check, quest check: clean
  • Windows: not run locally. Platform runs only in the merge queue.

Decisions (settled by the maintainer)

  1. Skip ConnectionReset in the shaper's receive loop, matching the QUIC driver, rather than turning off SIO_UDP_CONNRESET with WSAIoctl. Skipping is portable and needs no windows-sys or unsafe in a test crate.
  2. Don't add shaper tests to the Windows CI job. The in-memory socket models the Windows behavior on every platform, so Linux CI catches a regression.

The Windows proof is the merge-queue Platform run.

API / wire impact

None. moq-shaper is publish = false, and the new helper is private.

Follow-ups

  • Have Path's Drop in the drills print the shaper's failure. The drill would then show "the shaper stopped forwarding" instead of only a status timeout.

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

kixelated and others added 2 commits October 16, 2026 16:15
Windows reports an ICMP port unreachable as ConnectionReset on the next
recv_from, even on an unconnected UDP socket. The shaper treated that as
fatal: once the relay restart drill killed its relay, the next keep-alive
the shaper forwarded drew the error, the flow's reply task failed, and the
whole shaper stopped, so the clients never reconnected after the retarget.

Skip ConnectionReset on both receive paths, as the QUIC driver already
does, and make the in-memory test socket report it the way Windows does so
the unit tests catch a regression on every platform.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-17T02:06:56.820095Z 465378e New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@kixelated

Copy link
Copy Markdown
Collaborator Author

Outcome: root-caused from the code, not the job log (the job-log REST endpoint stayed rate-limited). On Windows, moq-shaper stopped on the ConnectionReset it got after forwarding to the killed relay, so the drill's clients had nothing to reconnect through. Fixed by skipping ConnectionReset on receive, as the QUIC driver already does. The in-memory test socket now reports it the way Windows does, so Linux CI catches a regression.

Choice made without asking: skip the error in the receive loop rather than turn off SIO_UDP_CONNRESET with WSAIoctl. Skipping is portable, needs no windows-sys or unsafe in a test crate, and matches the driver. The queue run is the first Windows confirmation.

(Written by Claude Opus 5.5)

@coderabbitai

coderabbitai Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 20 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 8fb1f724-3880-4caa-a6a6-541c86bf3f92
📥 Commits

Reviewing files that changed from the base of the PR and between 3789ee2 and 465378e.

📒 Files selected for processing: 6
  • quest/m0/ci/README.md
  • quest/m0/ci/nightly-green.md
  • quest/m0/ci/platform-queue.md
  • quest/m0/ci/platform-windows-drill.md
  • rs/moq-shaper/src/lib.rs
  • rs/moq-shaper/src/mem.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

# Conflicts:
#	quest/m0/ci/README.md
#	quest/m0/ci/platform-queue.md
@kixelated

Copy link
Copy Markdown
Collaborator Author

Merging. The shaper now skips ConnectionReset on both receive paths (listening socket and each flow socket), matching the QUIC driver, so a killed relay or closed client no longer stops forwarding. The in-memory socket models the Windows behavior, and two new tests cover it on Linux CI.

Review: Codex reviewed head 5f68f1d with no findings (plus a thumbs-up); the merge commit since the code commit only touched quest files. CI is green.

Decisions were settled earlier (skip rather than SIO_UDP_CONNRESET; no shaper tests in the Windows job). The merge queue's Windows Platform run is the end-to-end proof.

Follow-up: have the drills' Path drop print the shaper's failure so a stopped shaper shows up directly instead of as a status timeout.

(Written by Claude Opus 5.5)

@kixelated
kixelated added this pull request to the merge queue Oct 17, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Oct 17, 2026
@kixelated
kixelated enabled auto-merge October 17, 2026 01:18
@kixelated
kixelated added this pull request to the merge queue Oct 17, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 17, 2026
kixelated and others added 2 commits October 16, 2026 18:59
This PR fixes the drill and deletes its quest, so the link no longer resolves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated enabled auto-merge October 17, 2026 02:05
@kixelated
kixelated added this pull request to the merge queue Oct 17, 2026
Merged via the queue into main with commit dfd8a70 Oct 17, 2026
8 checks passed
@kixelated
kixelated deleted the quest/m0/ci/platform-windows-drill branch October 17, 2026 04:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant