diff --git a/quest/m0/test-flakes-2/README.md b/quest/m0/test-flakes-2/README.md index abc884cbeca..f78fe51d7c5 100644 --- a/quest/m0/test-flakes-2/README.md +++ b/quest/m0/test-flakes-2/README.md @@ -209,6 +209,10 @@ Added 2026-11-08 (maintainer, /quest-complete): `js/net`'s widened max-age re-drop test, which failed on [moq-dev/moq#8230](https://github.com/moq-dev/moq/pull/8230) with the `Cancel` reset [moq-dev/moq#8195](https://github.com/moq-dev/moq/pull/8195) intends for a reply still pending at the stream's end; the test races its finish against that reply. +Added 2026-11-08 (maintainer's standing rule that flakes join this epic): +`bench/relay.test.py`'s stream-transport test, which binds UDP on the number the +OS picked for TCP and failed a local run while other checks held that UDP port. + Added 2026-11-08 (maintainer's standing rule that flakes join this epic): moq-ffi's raw media warmup test, which failed in the merge queue for [moq-dev/moq#8272](https://github.com/moq-dev/moq/pull/8272) when its reader skipped one of three @@ -224,4 +228,5 @@ Public API: none. Wire: none. - [js/net's lite-07 datagram queue test holds up under load](/quest/m0/test-flakes-2/datagram-queues.md) - the createWritable queue test waits on the writes it asserts, and its `test.skip` is removed - [js/net's lite-06 session-close test keeps the end under load](/quest/m0/test-flakes-2/session-close-end.md) - a session close right after a datagram track's end returns the end, not a `remote error: 0`, and its lite-06 skip is removed - [js/net's widened max-age re-drop test ends cleanly, not with a reset](/quest/m0/test-flakes-2/widened-max-age-reset.md) - the test finishes the track only after the re-drop reply arrives, so the end is a FIN, and drops its wall-clock assert +- [bench/relay.test.py's stream-transport test gets a port free for TCP and UDP](/quest/m0/test-flakes-2/relay-port-pair.md) - the test binds TCP and UDP to one number free in both, so parallel checks can't collide with it - [moq-ffi's warmup test keeps every group under load](/quest/m0/test-flakes-2/ffi-warmup-wall-clock.md) - `raw_media_catalog_warmup_reaches_the_subscription` keeps the group between its warmup start and the live edge, and its `#[ignore]` is removed diff --git a/quest/m0/test-flakes-2/relay-port-pair.md b/quest/m0/test-flakes-2/relay-port-pair.md new file mode 100644 index 00000000000..94361508330 --- /dev/null +++ b/quest/m0/test-flakes-2/relay-port-pair.md @@ -0,0 +1,50 @@ +# [XS] bench/relay.test.py's stream-transport test gets a port free for TCP and UDP + +## Goal + +`bench/relay.test.py`'s `PortOwnership.test_a_stream_transport_needs_its_own_listener` +passes while other checks run in parallel on the same host, never failing its +setup with `Address already in use`. + +## Plan + +Planned 2026-11-08 (maintainer's standing rule that flakes join this epic). It +failed once locally while other sessions ran checks in parallel. None of the 40 +most recent failed merge-queue runs (2026-11-05 to 2026-11-08) failed it, so it +is not quarantined. + +The test does not bind a fixed port: it binds its TCP listener to port 0 +(line 278), then binds UDP to the same number (line 281, +`udp.bind(("127.0.0.1", port))`). TCP and UDP have separate port spaces, so +the OS's free TCP port says nothing about UDP. A parallel check's QUIC client +holding that UDP port fails line 281. + +Decided (rejected options in parentheses): + +- ✅ Find a number free in both protocols: bind UDP to port 0, then TCP to the + same number, and draw again while TCP refuses it. Keep the refused sockets + open until a pair binds, so each draw gets a new number and the search ends. + This searches for a free resource rather than retrying a timing race: no OS + call reserves one number in both protocols. (rejected: a private network + namespace via `unshare`, which needs unprivileged user namespaces that + `sh/dispatch.sh` already probes for and some hosts deny; a fake `lsof`, which + stops testing the real `lsof` filters `relay_owns_port` uses; dropping the + same-number requirement, which is what the relay binds and the check proves.) +- The sibling tests need no change: `test_tcp_without_udp_is_not_owned_as_a_relay_port` + binds only TCP to port 0, and the relay startup tests go through + `start_relay`, which already draws again when a port is taken. + +Reproduce first, by hand rather than as a committed test: from a helper process, +hold UDP sockets across all but a few ports of the ephemeral range, and watch +line 281 fail. Then show the fixed test passing under the same pressure. Next, +have the helper hold TCP listeners on all but a few ports instead, so TCP +refuses nearly every draw, and show the search redraws until a pair binds. +Each held socket costs a descriptor: the helper holds one per port, and the +test keeps one UDP socket per refused draw. Raise both processes' soft +`RLIMIT_NOFILE` to cover the range plus a few spare (the Linux default hard +limit, 524288, allows it). The helper is a separate process because +`relay_owns_port` checks the test's own PID, so a pressure listener the test +owned at the drawn port plus two would make the negative `tls` assertion fail. +Real collisions are rare, so the search itself needs no cap. + +Public API: none. Wire: none. diff --git a/quest/m3/webrtc/README.md b/quest/m3/webrtc/README.md index b067898bf5e..19a1d9dfb3c 100644 --- a/quest/m3/webrtc/README.md +++ b/quest/m3/webrtc/README.md @@ -27,6 +27,7 @@ Decided 2026-10-30 with the maintainer (planning interview), after the str0m wor - [WHIP/WHEP builds carry no SCTP](/quest/m3/webrtc/sctp-feature.md) - SCTP compiles only with the `p2p` feature - [The vendored stack keeps only what browsers negotiate](/quest/m3/webrtc/browser-only.md) - PSK suites, `ccm`, and anything else no browser uses are gone - [The vendored stack exposes only what moq-webrtc calls](/quest/m3/webrtc/dead-api.md) - dead upstream API is deleted once clock settles the time seams +- [A failed client socket ends a WebRTC session with an error](/quest/m3/webrtc/socket-error.md) - a client or `p2p` session whose socket fails ends with its I/O error, not `Ok` or `Closed` ## Related diff --git a/quest/m3/webrtc/socket-error.md b/quest/m3/webrtc/socket-error.md new file mode 100644 index 00000000000..e8b9b96d82d --- /dev/null +++ b/quest/m3/webrtc/socket-error.md @@ -0,0 +1,89 @@ +# [M] A failed client socket ends a WebRTC session with an error + +## Goal + +When the UDP socket under a moq-webrtc client session fails to receive, +`Client::publish` and `Client::subscribe` end with `Error::Io` carrying the +socket's `io::Error`, instead of `Ok` as if the peer had disconnected. A +`p2p` connection that fails this way hands its opener `p2p::Error::Io` before +its channel opens, including an opener that asks after the failure, and fails +the MoQ session over an opened channel with `qmux::Error::Io`, instead of +`Closed`. A peer disconnect still ends `Ok`. The first cause wins: a peer +close that lands before the socket fails stays a close. + +## Plan + +Planned 2026-11-08 from [moq-dev/moq#8268](https://github.com/moq-dev/moq/pull/8268)'s +follow-ups, where CodeRabbit raised it and the author deferred it as +pre-existing. + +Where it goes wrong today: + +1. `ice::reader` (`rs/moq-webrtc/src/ice/wire.rs`) logs a `recv_from` error + and breaks, dropping its queue's sender. Client sessions (`session::bind_udp`) + and the `p2p` transport (`p2p::bind`) use it. +2. `Agent::next` (`rs/moq-webrtc/src/ice/agent.rs`) sees the queue end and + fails with `ice::Error::Closed`, the same variant as an ICE disconnect or + a dead `Rtc`. +3. `Session::run_loop` (`rs/moq-webrtc/src/session.rs`) maps `Closed` to + `Ok(())`. `p2p`'s `Error::ice` maps it to its own `Closed`. + +The cause no caller can see is step 1: the reader drops the error. Fix it there. +The reader hands its `io::Error` to the agent, for example as the queue's last +item, and the agent fails with a new `ice::Error` variant named by role (such as +`Socket`). `Agent::next` returns a clone of the stored failure on every later +call, so `ice::Error` stays `Clone` and the variant holds an `Arc`. +The session maps it to the existing `moq_webrtc::Error::Io`, which is already +documented as covering a receive error on the media socket. `p2p`'s +`Error::ice` maps it to `p2p::Error::Io`, whose doc widens from bind and STUN +failures to cover it. Rebuild an owned error from the `Arc` that keeps what +callers read: `io::Error::from_raw_os_error` when it has an OS code, so +`raw_os_error()` still answers, else `io::Error::new(err.kind(), err)`. + +Every handle on a `p2p` connection reads its end as `Closed` today, whatever the +cause: an opener that asks after the driver exited (the dropped command channel), +and an opened channel's `Reader` (the dropped `Inbox`) and `Writer` (the closed +outbox). Pushing the error into the `Inbox` alone would race: qmux's driver can +find the outbox closed before it reads the `Inbox`. Instead the driver records +its terminal error in one slot the handles share before it drops anything, and +each handle that finds its channel gone reports that error, as `p2p::Error` or +`qmux::Error::Io`, instead of `Closed`. That fixes every cause, not only the +socket. + +`Agent::next` judges the establishment deadline before it reads the inbound +queue, so a socket error queued just before that deadline would surface as +`Timeout`: drain the queue, datagrams and any socket error behind them, before +judging the deadline, since the socket error is the cause. +Don't match on error kinds to decide which errors are fatal (AGENTS.md). + +Boundary: the server's shared-socket `server::mux::demux` logs and continues on +every recv error, since one transient error, such as an ICMP unreachable, +must not end every session on the socket. It is out of scope. Do check whether +the client reader has the same transient case. On Windows an unconnected UDP +socket reports an ICMP port unreachable as `WSAECONNRESET` unless +`SIO_UDP_CONNRESET` is off. If tokio leaves it on, a client's ICE check to a +dead candidate ends the session today. Turn it off when the socket is bound, +as a known platform capability, rather than ignoring that error at runtime. +Linux reports ICMP errors on an unconnected UDP socket only with `IP_RECVERR`, +which nothing here sets, so there a receive error is the socket failing. + +Reproducing test, failing on main first. On Linux, a connected UDP socket +reports a loopback ICMP port unreachable as `ECONNREFUSED` on its next +receive, with no rate limit on loopback. Connect a socket to a closed loopback +port, send once, and feed the socket to `ice::reader` and an agent, as +`agent.rs`'s `unpolled` fixture does. Then assert that the agent and the +session end with the socket error, not `Closed` or `Ok`. Also assert that a +`p2p` opener gets `p2p::Error::Io` with the underlying error whether it asked +before or after the failure. With a channel opened between two peers and a +write pending, and the outbox's closure reaching qmux before the `Inbox`'s, +failing one side's socket ends its MoQ session with +`qmux::Error::Io` and the OS code. A socket error queued behind a datagram, +read after the establishment deadline, still wins over `Timeout`. Keep the +existing peer-disconnect path asserting `Ok`. + +Public API: none. The errors map to existing `Io` variants, and the new +`ice::Error` variant is crate-private. Wire: none. + +## Related + +- [A typed DTLS handshake timeout error](/quest/m3/webrtc/dtls-timeout.md) - also adds an `ice::Error` variant mapped through the session loop and `p2p`