From 4bb93554dec240c0aabbf44a730fa3745efefadd Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 8 Nov 2026 12:24:09 -0800 Subject: [PATCH 1/8] quest: plan the relay port-pair flake and the WebRTC socket-error follow-up Co-Authored-By: Claude Opus 5.5 --- quest/m0/test-flakes-2/README.md | 5 ++ quest/m0/test-flakes-2/relay-port-pair.md | 41 ++++++++++++++++ quest/m3/webrtc/README.md | 1 + quest/m3/webrtc/socket-error.md | 59 +++++++++++++++++++++++ 4 files changed, 106 insertions(+) create mode 100644 quest/m0/test-flakes-2/relay-port-pair.md create mode 100644 quest/m3/webrtc/socket-error.md diff --git a/quest/m0/test-flakes-2/README.md b/quest/m0/test-flakes-2/README.md index 65268b1300a..69508280a0b 100644 --- a/quest/m0/test-flakes-2/README.md +++ b/quest/m0/test-flakes-2/README.md @@ -209,6 +209,10 @@ Added 2026-11-08 (maintainer, /quest-complete): `js/net`'s widened max-age re-drop test, which failed on [moq-dev/moq#8230](https://github.com/moq-dev/moq/pull/8230) with the `Cancel` reset [moq-dev/moq#8195](https://github.com/moq-dev/moq/pull/8195) intends for a reply still pending at the stream's end; the test races its finish against that reply. +Added 2026-11-08 (maintainer's standing rule that flakes join this epic): +`bench/relay.test.py`'s stream-transport test, which binds UDP on the number the +OS picked for TCP and failed a local run while other checks held that UDP port. + Public API: none. Wire: none. ## Required @@ -219,3 +223,4 @@ Public API: none. Wire: none. - [js/net's lite-07 datagram queue test holds up under load](/quest/m0/test-flakes-2/datagram-queues.md) - the createWritable queue test waits on the writes it asserts, and its `test.skip` is removed - [js/net's lite-06 session-close test keeps the end under load](/quest/m0/test-flakes-2/session-close-end.md) - a session close right after a datagram track's end returns the end, not a `remote error: 0`, and its lite-06 skip is removed - [js/net's widened max-age re-drop test ends cleanly, not with a reset](/quest/m0/test-flakes-2/widened-max-age-reset.md) - the test finishes the track only after the re-drop reply arrives, so the end is a FIN, and drops its wall-clock assert +- [bench/relay.test.py's stream-transport test gets a port free for TCP and UDP](/quest/m0/test-flakes-2/relay-port-pair.md) - the test binds TCP and UDP to one number free in both, so parallel checks can't collide with it diff --git a/quest/m0/test-flakes-2/relay-port-pair.md b/quest/m0/test-flakes-2/relay-port-pair.md new file mode 100644 index 00000000000..8207701c4b4 --- /dev/null +++ b/quest/m0/test-flakes-2/relay-port-pair.md @@ -0,0 +1,41 @@ +# [XS] bench/relay.test.py's stream-transport test gets a port free for TCP and UDP + +## Goal + +`bench/relay.test.py`'s `PortOwnership.test_a_stream_transport_needs_its_own_listener` +passes while other checks run in parallel on the same host, never failing its +setup with `Address already in use`. + +## Plan + +Planned 2026-11-08 (maintainer's standing rule that flakes join this epic). It +failed once locally while other sessions ran checks in parallel. None of the 40 +most recent failed merge-queue runs (2026-11-05 to 2026-11-08) failed it, so it +is not quarantined. + +The test does not bind a fixed port: it binds its TCP listener to port 0 +(line 278), then binds UDP to the same number (line 281, +`udp.bind(("127.0.0.1", port))`). TCP and UDP have separate port spaces, so +the OS's free TCP port says nothing about UDP. A parallel check's QUIC client +holding that UDP port fails line 281. + +Decided (rejected options in parentheses): + +- ✅ Find a number free in both protocols: bind UDP to port 0, then TCP to the + same number, and draw again while TCP refuses it. Keep the refused sockets + open until a pair binds, so each draw gets a new number and the search ends. + This searches for a free resource rather than retrying a timing race: no OS + call reserves one number in both protocols. (rejected: a private network + namespace via `unshare`, which needs unprivileged user namespaces that + `sh/dispatch.sh` already probes for and some hosts deny; a fake `lsof`, which + stops testing the real `lsof` filters `relay_owns_port` uses; dropping the + same-number requirement, which is what the relay binds and the check proves.) +- The sibling tests need no change: `test_tcp_without_udp_is_not_owned_as_a_relay_port` + binds only TCP to port 0, and the relay startup tests go through + `start_relay`, which already draws again when a port is taken. + +Reproduce first: hold the UDP port the test's TCP listener will get (for +example, hold UDP sockets across most of the ephemeral range) and watch line +281 fail, then show the fixed test passing under the same pressure. + +Public API: none. Wire: none. diff --git a/quest/m3/webrtc/README.md b/quest/m3/webrtc/README.md index b067898bf5e..ff19bada87e 100644 --- a/quest/m3/webrtc/README.md +++ b/quest/m3/webrtc/README.md @@ -27,6 +27,7 @@ Decided 2026-10-30 with the maintainer (planning interview), after the str0m wor - [WHIP/WHEP builds carry no SCTP](/quest/m3/webrtc/sctp-feature.md) - SCTP compiles only with the `p2p` feature - [The vendored stack keeps only what browsers negotiate](/quest/m3/webrtc/browser-only.md) - PSK suites, `ccm`, and anything else no browser uses are gone - [The vendored stack exposes only what moq-webrtc calls](/quest/m3/webrtc/dead-api.md) - dead upstream API is deleted once clock settles the time seams +- [A failed client socket ends a WebRTC session with an error](/quest/m3/webrtc/socket-error.md) - a client or `p2p` session whose socket fails ends with that error, not `Ok` ## Related diff --git a/quest/m3/webrtc/socket-error.md b/quest/m3/webrtc/socket-error.md new file mode 100644 index 00000000000..e23bb998a16 --- /dev/null +++ b/quest/m3/webrtc/socket-error.md @@ -0,0 +1,59 @@ +# [S] A failed client socket ends a WebRTC session with an error + +## Goal + +When the UDP socket under a moq-webrtc client session fails to receive, +`Client::publish`, `Client::subscribe`, and the `p2p` transport end with an +error that carries the socket's `io::Error`, instead of `Ok` as if the peer +had disconnected. A peer disconnect still ends `Ok`. + +## Plan + +Planned 2026-11-08 from [moq-dev/moq#8268](https://github.com/moq-dev/moq/pull/8268)'s +follow-ups, where CodeRabbit raised it and the author deferred it as +pre-existing. + +Where it goes wrong today: + +1. `ice::reader` (`rs/moq-webrtc/src/ice/wire.rs`) logs a `recv_from` error + and breaks, dropping its queue's sender. Client sessions (`session::bind_udp`) + and the `p2p` transport (`p2p::bind`) use it. +2. `Agent::next` (`rs/moq-webrtc/src/ice/agent.rs`) sees the queue end and + fails with `ice::Error::Closed`, the same variant as an ICE disconnect or + a dead `Rtc`. +3. `Session::run_loop` (`rs/moq-webrtc/src/session.rs`) maps `Closed` to + `Ok(())` (with #8268; before it, to `Error::SessionClosed`, which + `session::end` folded into `Ok`). `p2p`'s `Error::ice` maps it to its own + `Closed`. + +The cause no caller can see is step 1: the reader drops the error. Fix it there. +The reader hands its `io::Error` to the agent, for example as the queue's last +item, and the agent fails with a new `ice::Error` variant named by role (such as +`Socket`). The session maps it to a matching `moq_webrtc::Error` variant, and +`p2p` maps it to its own error. `ice::Error` is `Clone`, so the variant holds +the error behind an `Arc` rather than a string, keeping its kind for callers. +Don't match on error kinds to decide which errors are fatal (AGENTS.md). + +Boundary: the server's shared-socket `server::mux::demux` logs and continues on +every recv error, since one transient error, such as an ICMP unreachable, +must not end every session on the socket. It is out of scope. Do check whether +the client reader has the same transient case. On Windows an unconnected UDP +socket reports an ICMP port unreachable as `WSAECONNRESET` unless +`SIO_UDP_CONNRESET` is off. If tokio leaves it on, a client's ICE check to a +dead candidate ends the session today. Turn it off when the socket is bound, +as a known platform capability, rather than ignoring that error at runtime. + +Reproducing test, failing on main first. On Linux, a connected UDP socket +reports a loopback ICMP port unreachable as `ECONNREFUSED` on its next +receive, with no rate limit on loopback. Connect a socket to a closed loopback +port, send once, and feed the socket to `ice::reader` and an agent, as +`agent.rs`'s `unpolled` fixture does. Then assert that the agent and the +session end with the socket error, not `Closed` or `Ok`. Keep the existing +peer-disconnect path asserting `Ok`. + +Public API: an additive variant on `moq_webrtc::Error` (`#[non_exhaustive]`) +and on `p2p`'s error. Wire: none. + +## Related + +- [A typed DTLS handshake timeout error](/quest/m3/webrtc/dtls-timeout.md) - also adds an `ice::Error` variant mapped through the session loop and `p2p` From f357bcf998411f44ad28dacbd84c37dee1b048bb Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 8 Nov 2026 12:32:42 -0800 Subject: [PATCH 2/8] quest: scope the WebRTC socket-error plan to existing Io variants and pre-open p2p Co-Authored-By: Claude Opus 5.5 --- quest/m3/webrtc/README.md | 2 +- quest/m3/webrtc/socket-error.md | 28 +++++++++++++++++++--------- 2 files changed, 20 insertions(+), 10 deletions(-) diff --git a/quest/m3/webrtc/README.md b/quest/m3/webrtc/README.md index ff19bada87e..c05b15ff780 100644 --- a/quest/m3/webrtc/README.md +++ b/quest/m3/webrtc/README.md @@ -27,7 +27,7 @@ Decided 2026-10-30 with the maintainer (planning interview), after the str0m wor - [WHIP/WHEP builds carry no SCTP](/quest/m3/webrtc/sctp-feature.md) - SCTP compiles only with the `p2p` feature - [The vendored stack keeps only what browsers negotiate](/quest/m3/webrtc/browser-only.md) - PSK suites, `ccm`, and anything else no browser uses are gone - [The vendored stack exposes only what moq-webrtc calls](/quest/m3/webrtc/dead-api.md) - dead upstream API is deleted once clock settles the time seams -- [A failed client socket ends a WebRTC session with an error](/quest/m3/webrtc/socket-error.md) - a client or `p2p` session whose socket fails ends with that error, not `Ok` +- [A failed client socket ends a WebRTC session with an error](/quest/m3/webrtc/socket-error.md) - a client session whose socket fails ends with `Error::Io`, not `Ok` ## Related diff --git a/quest/m3/webrtc/socket-error.md b/quest/m3/webrtc/socket-error.md index e23bb998a16..585fa9db324 100644 --- a/quest/m3/webrtc/socket-error.md +++ b/quest/m3/webrtc/socket-error.md @@ -3,9 +3,15 @@ ## Goal When the UDP socket under a moq-webrtc client session fails to receive, -`Client::publish`, `Client::subscribe`, and the `p2p` transport end with an -error that carries the socket's `io::Error`, instead of `Ok` as if the peer -had disconnected. A peer disconnect still ends `Ok`. +`Client::publish` and `Client::subscribe` end with `Error::Io` carrying the +socket's `io::Error`, instead of `Ok` as if the peer had disconnected. A +`p2p` connection that fails this way before its channel opens hands its opener +`p2p::Error::Io`, not `Closed`. A peer disconnect still ends `Ok`. + +Non-goal: carrying the cause past an opened `p2p` channel. There the driver +drops the channel's reader, so the MoQ session over it already fails with +qmux's `Closed`, an error rather than `Ok`. `qmux::Error` has no I/O variant to +carry the cause. ## Plan @@ -29,10 +35,14 @@ Where it goes wrong today: The cause no caller can see is step 1: the reader drops the error. Fix it there. The reader hands its `io::Error` to the agent, for example as the queue's last item, and the agent fails with a new `ice::Error` variant named by role (such as -`Socket`). The session maps it to a matching `moq_webrtc::Error` variant, and -`p2p` maps it to its own error. `ice::Error` is `Clone`, so the variant holds -the error behind an `Arc` rather than a string, keeping its kind for callers. -Don't match on error kinds to decide which errors are fatal (AGENTS.md). +`Socket`). `Agent::next` returns a clone of the stored failure on every later +call, so `ice::Error` stays `Clone` and the variant holds an `Arc`. +The session maps it to the existing `moq_webrtc::Error::Io`, which is already +documented as covering a receive error on the media socket. `p2p`'s +`Error::ice` maps it to `p2p::Error::Io`, whose doc widens from bind and STUN +failures to cover it. `io::Error::new(err.kind(), err)` rebuilds an owned error +from the `Arc` that keeps its kind and source. Don't match on error kinds to +decide which errors are fatal (AGENTS.md). Boundary: the server's shared-socket `server::mux::demux` logs and continues on every recv error, since one transient error, such as an ICMP unreachable, @@ -51,8 +61,8 @@ port, send once, and feed the socket to `ice::reader` and an agent, as session end with the socket error, not `Closed` or `Ok`. Keep the existing peer-disconnect path asserting `Ok`. -Public API: an additive variant on `moq_webrtc::Error` (`#[non_exhaustive]`) -and on `p2p`'s error. Wire: none. +Public API: none. The errors map to existing `Io` variants, and the new +`ice::Error` variant is crate-private. Wire: none. ## Related From 096117e9cf09ad2fbb41c1750c55dc0935a0e9fb Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 8 Nov 2026 12:40:35 -0800 Subject: [PATCH 3/8] quest: carry the WebRTC socket error past an opened p2p channel Co-Authored-By: Claude Opus 5.5 --- quest/m3/webrtc/README.md | 2 +- quest/m3/webrtc/socket-error.md | 21 +++++++++++---------- 2 files changed, 12 insertions(+), 11 deletions(-) diff --git a/quest/m3/webrtc/README.md b/quest/m3/webrtc/README.md index c05b15ff780..19a1d9dfb3c 100644 --- a/quest/m3/webrtc/README.md +++ b/quest/m3/webrtc/README.md @@ -27,7 +27,7 @@ Decided 2026-10-30 with the maintainer (planning interview), after the str0m wor - [WHIP/WHEP builds carry no SCTP](/quest/m3/webrtc/sctp-feature.md) - SCTP compiles only with the `p2p` feature - [The vendored stack keeps only what browsers negotiate](/quest/m3/webrtc/browser-only.md) - PSK suites, `ccm`, and anything else no browser uses are gone - [The vendored stack exposes only what moq-webrtc calls](/quest/m3/webrtc/dead-api.md) - dead upstream API is deleted once clock settles the time seams -- [A failed client socket ends a WebRTC session with an error](/quest/m3/webrtc/socket-error.md) - a client session whose socket fails ends with `Error::Io`, not `Ok` +- [A failed client socket ends a WebRTC session with an error](/quest/m3/webrtc/socket-error.md) - a client or `p2p` session whose socket fails ends with its I/O error, not `Ok` or `Closed` ## Related diff --git a/quest/m3/webrtc/socket-error.md b/quest/m3/webrtc/socket-error.md index 585fa9db324..408b56158ef 100644 --- a/quest/m3/webrtc/socket-error.md +++ b/quest/m3/webrtc/socket-error.md @@ -5,13 +5,9 @@ When the UDP socket under a moq-webrtc client session fails to receive, `Client::publish` and `Client::subscribe` end with `Error::Io` carrying the socket's `io::Error`, instead of `Ok` as if the peer had disconnected. A -`p2p` connection that fails this way before its channel opens hands its opener -`p2p::Error::Io`, not `Closed`. A peer disconnect still ends `Ok`. - -Non-goal: carrying the cause past an opened `p2p` channel. There the driver -drops the channel's reader, so the MoQ session over it already fails with -qmux's `Closed`, an error rather than `Ok`. `qmux::Error` has no I/O variant to -carry the cause. +`p2p` connection that fails this way hands its opener `p2p::Error::Io` before +its channel opens, and fails the MoQ session over an opened channel with +`qmux::Error::Io`, instead of `Closed`. A peer disconnect still ends `Ok`. ## Plan @@ -41,8 +37,12 @@ The session maps it to the existing `moq_webrtc::Error::Io`, which is already documented as covering a receive error on the media socket. `p2p`'s `Error::ice` maps it to `p2p::Error::Io`, whose doc widens from bind and STUN failures to cover it. `io::Error::new(err.kind(), err)` rebuilds an owned error -from the `Arc` that keeps its kind and source. Don't match on error kinds to -decide which errors are fatal (AGENTS.md). +from the `Arc` that keeps its kind and source. Once `p2p`'s channel has opened, +the driver drops the channel's `Inbox`, which its `Reader` reads as +`qmux::Error::Closed`. On a socket failure the driver instead pushes +`Err(qmux::Error::Io(arc))` into that `Inbox` first, since `Inbox::push` +already carries errors. Don't match on error kinds to decide which errors are +fatal (AGENTS.md). Boundary: the server's shared-socket `server::mux::demux` logs and continues on every recv error, since one transient error, such as an ICMP unreachable, @@ -58,7 +58,8 @@ reports a loopback ICMP port unreachable as `ECONNREFUSED` on its next receive, with no rate limit on loopback. Connect a socket to a closed loopback port, send once, and feed the socket to `ice::reader` and an agent, as `agent.rs`'s `unpolled` fixture does. Then assert that the agent and the -session end with the socket error, not `Closed` or `Ok`. Keep the existing +session end with the socket error, not `Closed` or `Ok`, and that an opened +`p2p` channel's reader returns `qmux::Error::Io`. Keep the existing peer-disconnect path asserting `Ok`. Public API: none. The errors map to existing `Io` variants, and the new From abec23b55b0b0a30c2eead0b6420688a0f2c821e Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 8 Nov 2026 12:48:17 -0800 Subject: [PATCH 4/8] quest: cover the redraw, pre-open p2p, and deadline cases in the follow-up plans Co-Authored-By: Claude Opus 5.5 --- quest/m0/test-flakes-2/relay-port-pair.md | 4 +++- quest/m3/webrtc/socket-error.md | 14 +++++++++----- 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/quest/m0/test-flakes-2/relay-port-pair.md b/quest/m0/test-flakes-2/relay-port-pair.md index 8207701c4b4..dacdc4ad846 100644 --- a/quest/m0/test-flakes-2/relay-port-pair.md +++ b/quest/m0/test-flakes-2/relay-port-pair.md @@ -36,6 +36,8 @@ Decided (rejected options in parentheses): Reproduce first: hold the UDP port the test's TCP listener will get (for example, hold UDP sockets across most of the ephemeral range) and watch line -281 fail, then show the fixed test passing under the same pressure. +281 fail, then show the fixed test passing under the same pressure. Also hold +TCP listeners across most of the range, so TCP refuses the first draws, and +show the search redraws until a pair binds. Public API: none. Wire: none. diff --git a/quest/m3/webrtc/socket-error.md b/quest/m3/webrtc/socket-error.md index 408b56158ef..d539590af36 100644 --- a/quest/m3/webrtc/socket-error.md +++ b/quest/m3/webrtc/socket-error.md @@ -41,8 +41,10 @@ from the `Arc` that keeps its kind and source. Once `p2p`'s channel has opened, the driver drops the channel's `Inbox`, which its `Reader` reads as `qmux::Error::Closed`. On a socket failure the driver instead pushes `Err(qmux::Error::Io(arc))` into that `Inbox` first, since `Inbox::push` -already carries errors. Don't match on error kinds to decide which errors are -fatal (AGENTS.md). +already carries errors. `Agent::next` judges the establishment deadline before +it reads the inbound queue, so a socket error queued just before that deadline +would surface as `Timeout`: read a queued socket error first, since it is the +cause. Don't match on error kinds to decide which errors are fatal (AGENTS.md). Boundary: the server's shared-socket `server::mux::demux` logs and continues on every recv error, since one transient error, such as an ICMP unreachable, @@ -58,9 +60,11 @@ reports a loopback ICMP port unreachable as `ECONNREFUSED` on its next receive, with no rate limit on loopback. Connect a socket to a closed loopback port, send once, and feed the socket to `ice::reader` and an agent, as `agent.rs`'s `unpolled` fixture does. Then assert that the agent and the -session end with the socket error, not `Closed` or `Ok`, and that an opened -`p2p` channel's reader returns `qmux::Error::Io`. Keep the existing -peer-disconnect path asserting `Ok`. +session end with the socket error, not `Closed` or `Ok`. Also assert that a +`p2p` opener still waiting gets `p2p::Error::Io` with the underlying error, that +an opened `p2p` channel's reader returns `qmux::Error::Io`, and that an error +queued past the establishment deadline still wins over `Timeout`. Keep the +existing peer-disconnect path asserting `Ok`. Public API: none. The errors map to existing `Io` variants, and the new `ice::Error` variant is crate-private. Wire: none. From d768e1233289071e031740ad03764edebef14223 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 8 Nov 2026 12:56:28 -0800 Subject: [PATCH 5/8] quest: size the WebRTC socket-error quest M and cover late p2p openers Co-Authored-By: Claude Opus 5.5 --- quest/m3/webrtc/socket-error.md | 28 +++++++++++++++++++--------- 1 file changed, 19 insertions(+), 9 deletions(-) diff --git a/quest/m3/webrtc/socket-error.md b/quest/m3/webrtc/socket-error.md index d539590af36..a0778c1d2bd 100644 --- a/quest/m3/webrtc/socket-error.md +++ b/quest/m3/webrtc/socket-error.md @@ -1,4 +1,4 @@ -# [S] A failed client socket ends a WebRTC session with an error +# [M] A failed client socket ends a WebRTC session with an error ## Goal @@ -6,8 +6,10 @@ When the UDP socket under a moq-webrtc client session fails to receive, `Client::publish` and `Client::subscribe` end with `Error::Io` carrying the socket's `io::Error`, instead of `Ok` as if the peer had disconnected. A `p2p` connection that fails this way hands its opener `p2p::Error::Io` before -its channel opens, and fails the MoQ session over an opened channel with -`qmux::Error::Io`, instead of `Closed`. A peer disconnect still ends `Ok`. +its channel opens, including an opener that asks after the failure, and fails +the MoQ session over an opened channel with `qmux::Error::Io`, instead of +`Closed`. A peer disconnect still ends `Ok`. The first cause wins: a peer +close that lands before the socket fails stays a close. ## Plan @@ -40,8 +42,12 @@ failures to cover it. `io::Error::new(err.kind(), err)` rebuilds an owned error from the `Arc` that keeps its kind and source. Once `p2p`'s channel has opened, the driver drops the channel's `Inbox`, which its `Reader` reads as `qmux::Error::Closed`. On a socket failure the driver instead pushes -`Err(qmux::Error::Io(arc))` into that `Inbox` first, since `Inbox::push` -already carries errors. `Agent::next` judges the establishment deadline before +`Err(qmux::Error::Io(arc))` into that `Inbox` before it drops the `Inbox` or +the outbox, since `Inbox::push` already carries errors, so a pending write +cannot end the MoQ session with `Closed` first. A `p2p` opener that asks after +the driver exited gets `Closed` from the dropped command channel today, whatever +the cause. The driver leaves its terminal error where the handle reads it once +the channel is gone, which fixes it for every cause, not only the socket. `Agent::next` judges the establishment deadline before it reads the inbound queue, so a socket error queued just before that deadline would surface as `Timeout`: read a queued socket error first, since it is the cause. Don't match on error kinds to decide which errors are fatal (AGENTS.md). @@ -52,7 +58,9 @@ must not end every session on the socket. It is out of scope. Do check whether the client reader has the same transient case. On Windows an unconnected UDP socket reports an ICMP port unreachable as `WSAECONNRESET` unless `SIO_UDP_CONNRESET` is off. If tokio leaves it on, a client's ICE check to a -dead candidate ends the session today. Turn it off when the socket is bound, +dead candidate ends the session today. Linux reports ICMP errors on an +unconnected UDP socket only with `IP_RECVERR`, which nothing here sets, so there +a receive error is the socket failing. Turn it off when the socket is bound, as a known platform capability, rather than ignoring that error at runtime. Reproducing test, failing on main first. On Linux, a connected UDP socket @@ -61,9 +69,11 @@ receive, with no rate limit on loopback. Connect a socket to a closed loopback port, send once, and feed the socket to `ice::reader` and an agent, as `agent.rs`'s `unpolled` fixture does. Then assert that the agent and the session end with the socket error, not `Closed` or `Ok`. Also assert that a -`p2p` opener still waiting gets `p2p::Error::Io` with the underlying error, that -an opened `p2p` channel's reader returns `qmux::Error::Io`, and that an error -queued past the establishment deadline still wins over `Timeout`. Keep the +`p2p` opener gets `p2p::Error::Io` with the underlying error whether it asked +before or after the failure. With a channel opened between two peers and a +write pending, failing one side's socket ends its MoQ session with +`qmux::Error::Io`. An error queued past the establishment deadline still wins +over `Timeout`. Keep the existing peer-disconnect path asserting `Ok`. Public API: none. The errors map to existing `Io` variants, and the new From 4916bf7f05f6c71aef6aa0f03fae3369a0a1329b Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 8 Nov 2026 13:04:29 -0800 Subject: [PATCH 6/8] quest: keep OS error codes, drain before the deadline, and isolate port pressure Co-Authored-By: Claude Opus 5.5 --- quest/m0/test-flakes-2/relay-port-pair.md | 5 ++++- quest/m3/webrtc/socket-error.md | 13 +++++++------ 2 files changed, 11 insertions(+), 7 deletions(-) diff --git a/quest/m0/test-flakes-2/relay-port-pair.md b/quest/m0/test-flakes-2/relay-port-pair.md index dacdc4ad846..fbdbab53db5 100644 --- a/quest/m0/test-flakes-2/relay-port-pair.md +++ b/quest/m0/test-flakes-2/relay-port-pair.md @@ -38,6 +38,9 @@ Reproduce first: hold the UDP port the test's TCP listener will get (for example, hold UDP sockets across most of the ephemeral range) and watch line 281 fail, then show the fixed test passing under the same pressure. Also hold TCP listeners across most of the range, so TCP refuses the first draws, and -show the search redraws until a pair binds. +show the search redraws until a pair binds. Hold the pressure sockets in a +separate process: `relay_owns_port` checks the test's own PID, so a pressure +listener the test owns at the drawn port plus two would make the negative `tls` +assertion fail, and the test's descriptors stay free for the search. Public API: none. Wire: none. diff --git a/quest/m3/webrtc/socket-error.md b/quest/m3/webrtc/socket-error.md index a0778c1d2bd..0d3c7d15fec 100644 --- a/quest/m3/webrtc/socket-error.md +++ b/quest/m3/webrtc/socket-error.md @@ -38,8 +38,9 @@ call, so `ice::Error` stays `Clone` and the variant holds an `Arc`. The session maps it to the existing `moq_webrtc::Error::Io`, which is already documented as covering a receive error on the media socket. `p2p`'s `Error::ice` maps it to `p2p::Error::Io`, whose doc widens from bind and STUN -failures to cover it. `io::Error::new(err.kind(), err)` rebuilds an owned error -from the `Arc` that keeps its kind and source. Once `p2p`'s channel has opened, +failures to cover it. Rebuild an owned error from the `Arc` that keeps what +callers read: `io::Error::from_raw_os_error` when it has an OS code, so +`raw_os_error()` still answers, else `io::Error::new(err.kind(), err)`. Once `p2p`'s channel has opened, the driver drops the channel's `Inbox`, which its `Reader` reads as `qmux::Error::Closed`. On a socket failure the driver instead pushes `Err(qmux::Error::Io(arc))` into that `Inbox` before it drops the `Inbox` or @@ -49,8 +50,8 @@ the driver exited gets `Closed` from the dropped command channel today, whatever the cause. The driver leaves its terminal error where the handle reads it once the channel is gone, which fixes it for every cause, not only the socket. `Agent::next` judges the establishment deadline before it reads the inbound queue, so a socket error queued just before that deadline -would surface as `Timeout`: read a queued socket error first, since it is the -cause. Don't match on error kinds to decide which errors are fatal (AGENTS.md). +would surface as `Timeout`: drain the queue, datagrams and any socket error +behind them, before judging the deadline, since the socket error is the cause. Don't match on error kinds to decide which errors are fatal (AGENTS.md). Boundary: the server's shared-socket `server::mux::demux` logs and continues on every recv error, since one transient error, such as an ICMP unreachable, @@ -72,8 +73,8 @@ session end with the socket error, not `Closed` or `Ok`. Also assert that a `p2p` opener gets `p2p::Error::Io` with the underlying error whether it asked before or after the failure. With a channel opened between two peers and a write pending, failing one side's socket ends its MoQ session with -`qmux::Error::Io`. An error queued past the establishment deadline still wins -over `Timeout`. Keep the +`qmux::Error::Io` and the OS code. A socket error queued behind a datagram, +read after the establishment deadline, still wins over `Timeout`. Keep the existing peer-disconnect path asserting `Ok`. Public API: none. The errors map to existing `Io` variants, and the new From 6792a2865908be811343e98eacc9843e24a73408 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 8 Nov 2026 13:24:24 -0800 Subject: [PATCH 7/8] quest: share one p2p terminal error across handles and cap port pressure Co-Authored-By: Claude Opus 5.5 --- quest/m0/test-flakes-2/relay-port-pair.md | 3 +- quest/m3/webrtc/socket-error.md | 38 +++++++++++++---------- 2 files changed, 24 insertions(+), 17 deletions(-) diff --git a/quest/m0/test-flakes-2/relay-port-pair.md b/quest/m0/test-flakes-2/relay-port-pair.md index fbdbab53db5..2bbd020313f 100644 --- a/quest/m0/test-flakes-2/relay-port-pair.md +++ b/quest/m0/test-flakes-2/relay-port-pair.md @@ -41,6 +41,7 @@ TCP listeners across most of the range, so TCP refuses the first draws, and show the search redraws until a pair binds. Hold the pressure sockets in a separate process: `relay_owns_port` checks the test's own PID, so a pressure listener the test owns at the drawn port plus two would make the negative `tls` -assertion fail, and the test's descriptors stay free for the search. +assertion fail. The search keeps one UDP socket per refused draw, so hold fewer +TCP listeners than the test's spare descriptors; real collisions are rare. Public API: none. Wire: none. diff --git a/quest/m3/webrtc/socket-error.md b/quest/m3/webrtc/socket-error.md index 0d3c7d15fec..eb5b80b3c92 100644 --- a/quest/m3/webrtc/socket-error.md +++ b/quest/m3/webrtc/socket-error.md @@ -40,18 +40,23 @@ documented as covering a receive error on the media socket. `p2p`'s `Error::ice` maps it to `p2p::Error::Io`, whose doc widens from bind and STUN failures to cover it. Rebuild an owned error from the `Arc` that keeps what callers read: `io::Error::from_raw_os_error` when it has an OS code, so -`raw_os_error()` still answers, else `io::Error::new(err.kind(), err)`. Once `p2p`'s channel has opened, -the driver drops the channel's `Inbox`, which its `Reader` reads as -`qmux::Error::Closed`. On a socket failure the driver instead pushes -`Err(qmux::Error::Io(arc))` into that `Inbox` before it drops the `Inbox` or -the outbox, since `Inbox::push` already carries errors, so a pending write -cannot end the MoQ session with `Closed` first. A `p2p` opener that asks after -the driver exited gets `Closed` from the dropped command channel today, whatever -the cause. The driver leaves its terminal error where the handle reads it once -the channel is gone, which fixes it for every cause, not only the socket. `Agent::next` judges the establishment deadline before -it reads the inbound queue, so a socket error queued just before that deadline -would surface as `Timeout`: drain the queue, datagrams and any socket error -behind them, before judging the deadline, since the socket error is the cause. Don't match on error kinds to decide which errors are fatal (AGENTS.md). +`raw_os_error()` still answers, else `io::Error::new(err.kind(), err)`. + +Every handle on a `p2p` connection reads its end as `Closed` today, whatever the +cause: an opener that asks after the driver exited (the dropped command channel), +and an opened channel's `Reader` (the dropped `Inbox`) and `Writer` (the closed +outbox). Pushing the error into the `Inbox` alone would race: qmux's driver can +find the outbox closed before it reads the `Inbox`. Instead the driver records +its terminal error in one slot the handles share before it drops anything, and +each handle that finds its channel gone reports that error, as `p2p::Error` or +`qmux::Error::Io`, instead of `Closed`. That fixes every cause, not only the +socket. + +`Agent::next` judges the establishment deadline before it reads the inbound +queue, so a socket error queued just before that deadline would surface as +`Timeout`: drain the queue, datagrams and any socket error behind them, before +judging the deadline, since the socket error is the cause. +Don't match on error kinds to decide which errors are fatal (AGENTS.md). Boundary: the server's shared-socket `server::mux::demux` logs and continues on every recv error, since one transient error, such as an ICMP unreachable, @@ -59,10 +64,10 @@ must not end every session on the socket. It is out of scope. Do check whether the client reader has the same transient case. On Windows an unconnected UDP socket reports an ICMP port unreachable as `WSAECONNRESET` unless `SIO_UDP_CONNRESET` is off. If tokio leaves it on, a client's ICE check to a -dead candidate ends the session today. Linux reports ICMP errors on an -unconnected UDP socket only with `IP_RECVERR`, which nothing here sets, so there -a receive error is the socket failing. Turn it off when the socket is bound, +dead candidate ends the session today. Turn it off when the socket is bound, as a known platform capability, rather than ignoring that error at runtime. +Linux reports ICMP errors on an unconnected UDP socket only with `IP_RECVERR`, +which nothing here sets, so there a receive error is the socket failing. Reproducing test, failing on main first. On Linux, a connected UDP socket reports a loopback ICMP port unreachable as `ECONNREFUSED` on its next @@ -72,7 +77,8 @@ port, send once, and feed the socket to `ice::reader` and an agent, as session end with the socket error, not `Closed` or `Ok`. Also assert that a `p2p` opener gets `p2p::Error::Io` with the underlying error whether it asked before or after the failure. With a channel opened between two peers and a -write pending, failing one side's socket ends its MoQ session with +write pending, and the outbox's closure reaching qmux before the `Inbox`'s, +failing one side's socket ends its MoQ session with `qmux::Error::Io` and the OS code. A socket error queued behind a datagram, read after the establishment deadline, still wins over `Timeout`. Keep the existing peer-disconnect path asserting `Ok`. From 3aa7d54d69fc5fc1efa2b25d330272dc313d4d99 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sun, 8 Nov 2026 13:32:04 -0800 Subject: [PATCH 8/8] quest: make the relay port-pair reproduction force refusals deterministically Co-Authored-By: Claude Opus 5.5 --- quest/m0/test-flakes-2/relay-port-pair.md | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/quest/m0/test-flakes-2/relay-port-pair.md b/quest/m0/test-flakes-2/relay-port-pair.md index 2bbd020313f..94361508330 100644 --- a/quest/m0/test-flakes-2/relay-port-pair.md +++ b/quest/m0/test-flakes-2/relay-port-pair.md @@ -34,14 +34,17 @@ Decided (rejected options in parentheses): binds only TCP to port 0, and the relay startup tests go through `start_relay`, which already draws again when a port is taken. -Reproduce first: hold the UDP port the test's TCP listener will get (for -example, hold UDP sockets across most of the ephemeral range) and watch line -281 fail, then show the fixed test passing under the same pressure. Also hold -TCP listeners across most of the range, so TCP refuses the first draws, and -show the search redraws until a pair binds. Hold the pressure sockets in a -separate process: `relay_owns_port` checks the test's own PID, so a pressure -listener the test owns at the drawn port plus two would make the negative `tls` -assertion fail. The search keeps one UDP socket per refused draw, so hold fewer -TCP listeners than the test's spare descriptors; real collisions are rare. +Reproduce first, by hand rather than as a committed test: from a helper process, +hold UDP sockets across all but a few ports of the ephemeral range, and watch +line 281 fail. Then show the fixed test passing under the same pressure. Next, +have the helper hold TCP listeners on all but a few ports instead, so TCP +refuses nearly every draw, and show the search redraws until a pair binds. +Each held socket costs a descriptor: the helper holds one per port, and the +test keeps one UDP socket per refused draw. Raise both processes' soft +`RLIMIT_NOFILE` to cover the range plus a few spare (the Linux default hard +limit, 524288, allows it). The helper is a separate process because +`relay_owns_port` checks the test's own PID, so a pressure listener the test +owned at the drawn port plus two would make the negative `tls` assertion fail. +Real collisions are rare, so the search itself needs no cap. Public API: none. Wire: none.