diff --git a/quest/m0/test-flakes-2/README.md b/quest/m0/test-flakes-2/README.md index 381c52bd7f0..90357e7dd8c 100644 --- a/quest/m0/test-flakes-2/README.md +++ b/quest/m0/test-flakes-2/README.md @@ -145,10 +145,11 @@ SUBSCRIBE_FINISH arrived before its FIN aborted with the session's error. Its datagrams now finish whenever the stream track's settled end survives the abort, fixed by [moq-dev/moq#8157](https://github.com/moq-dev/moq/pull/8157). -Added 2026-11-07 (maintainer, /quest-complete): moq-relay's -`embed::worker_tokio_custom_route_and_quic`, which failed -[moq-dev/moq#8138](https://github.com/moq-dev/moq/pull/8138)'s Check with "the -owner's sockets outlived `run`". It is `#[ignore]`d until its child fixes it. +moq-relay's `embed::worker_tokio_custom_route_and_quic` was never load-bound +either. Its socket-ownership check matched the HTTP and QUIC ports without +their protocol, so the test's own QUIC client, still draining, counted as the +relay's whenever its UDP port shared the HTTP listener's TCP port number. +Fixed by [moq-dev/moq#8236](https://github.com/moq-dev/moq/pull/8236). `sh/dispatch-query.test.py`'s terminate test was never load-bound. A TERM landing just after the dispatcher's `wait -n` returned left Bash marking the @@ -193,7 +194,6 @@ Public API: none. Wire: none. - [Fallback playback](/quest/m0/test-flakes-2/ietf22-fallback.md) - a browser viewer follows a fallback within the media lane's bound without dropping its session - [moq-gst's stale-completion sink test survives load](/quest/m0/test-flakes-2/gst-stale-completion.md) - `a_stale_completion_cannot_end_the_next_publication` passes under stress, and its `#[ignore]` is removed -- [The embedded relay's sockets are closed when run returns](/quest/m0/test-flakes-2/embed-sockets-outlived.md) - `embed::worker_tokio_custom_route_and_quic` stops failing with "the owner's sockets outlived `run`" - [The http:// fingerprint tests survive a slow fetch](/quest/m0/test-flakes-2/fingerprint-host-clock.md) - a fingerprint fetch slower than the 500 ms hedge no longer trips the host-clock guard - [A lite-07 reply FINs instead of resetting when the subscriber FINs mid-write](/quest/m0/test-flakes-2/lite07-reply-fin.md) - `js/net`'s empty-window reply test passes under `--rerun-each 200`, fixed in the publisher - [moq-tokio's stream-stopped tests survive a loaded host](/quest/m0/test-flakes-2/quic-stream-stopped.md) - `stream_stopped` awaits its stop event instead of a 100 ms wall-clock deadline, and its `#[ignore]` is removed diff --git a/quest/m0/test-flakes-2/embed-sockets-outlived.md b/quest/m0/test-flakes-2/embed-sockets-outlived.md deleted file mode 100644 index 3537b16ba0b..00000000000 --- a/quest/m0/test-flakes-2/embed-sockets-outlived.md +++ /dev/null @@ -1,41 +0,0 @@ -# [S] The embedded relay's sockets are closed when run returns - -## Goal - -moq-relay's `embed::worker_tokio_custom_route_and_quic` -(`rs/moq-relay/tests/integration/embed.rs`) passes reliably under load, -fixed at its cause under the epic's rules, and its `#[ignore]` and -`sh/gh/test-inventory.allow` entry are removed. Either `run` returns only -after every worker's socket is closed, or the test's check stops racing a -close that `run` already guarantees. - -## Plan - -Planned 2026-11-07 (maintainer, /quest-complete) from -[moq-dev/moq#8138](https://github.com/moq-dev/moq/pull/8138)'s optional -Check ([run](https://github.com/moq-dev/moq/actions/runs/42352819862)). That PR -touches only `sh/` and quest docs, yet the test panicked at `embed.rs:351` -with "the owner's sockets outlived `run`". Following AGENTS.md's rule to -disable a flaky test at once, the test is `#[ignore]`d until this quest fixes it. - -[moq-dev/moq#7823](https://github.com/moq-dev/moq/pull/7823) already moved -Linux to the socket-inode ownership proof (`bound` before `stop`, then -`open_sockets` after), so a reused port cannot explain this failure. A failure -there means an inode bound before `stop` was still in the process's fd table -after `run` returned. With `runtime.workers = 2`, the thread-per-core Tokio -workers each bind their own `SO_REUSEPORT` socket. Suspect a worker, or a -socket it owns, that drops after `run` joins it, such as a spawned task or a -connection that outlives the join. - -Reproduce it first, for example with repeated runs of this one test beside a -CPU-heavy job. Record which inode outlived `run` and who owns it. Then make -`run` close it before returning, or show that the close is already ordered -before `run` returns and fix the check instead. No retry, sleep, or settle -window. - -The assertion lives in the shared `embed_and_stop` helper, so -`shared_tokio_custom_route_and_quic` (default CI) and -`uring_custom_route_and_quic` (nightly's `just rs uring`) assert the same close. -Verify the fix on all three before removing the `#[ignore]`. If -the cause is the check racing a close rather than a leak in the worker mode, -the fix covers the other two as well. diff --git a/rs/moq-relay/tests/integration/embed.rs b/rs/moq-relay/tests/integration/embed.rs index 8cd2f762b10..a0bc09bd6ad 100644 --- a/rs/moq-relay/tests/integration/embed.rs +++ b/rs/moq-relay/tests/integration/embed.rs @@ -78,15 +78,17 @@ async fn assert_owner_stopped(quic: SocketAddr, http: SocketAddr) { ); } -/// The socket inodes this process holds bound to `port`: every UDP socket, but -/// only a TCP listener, since accepted connections are not the owner's to close. +/// The socket inodes this process holds bound to `port` of type `kind`: every +/// UDP socket, but only a TCP listener, since accepted connections are not the +/// owner's to close. The type matters because UDP and TCP number ports +/// separately: the test's own QUIC client can hold the HTTP port's number. /// /// Asked of this process's own descriptors, for two reasons. Rebinding the port /// races any concurrent process's ephemeral bind, which may take a freed port. /// And `/proc/net/udp` is served in chunks that skip entries while other /// processes churn sockets. #[cfg(target_os = "linux")] -fn bound(port: u16) -> std::collections::HashSet { +fn bound(kind: socket2::Type, port: u16) -> std::collections::HashSet { use std::os::fd::FromRawFd; use std::os::unix::fs::{FileTypeExt, MetadataExt}; @@ -111,8 +113,8 @@ fn bound(port: u16) -> std::collections::HashSet { } let socket = socket2::SockRef::from(&file); let owned = match socket.r#type() { - Ok(socket2::Type::DGRAM) => true, - Ok(socket2::Type::STREAM) => socket.is_listener().unwrap_or(false), + Ok(socket2::Type::DGRAM) => kind == socket2::Type::DGRAM, + Ok(socket2::Type::STREAM) => kind == socket2::Type::STREAM && socket.is_listener().unwrap_or(false), _ => false, }; let on_port = socket @@ -336,8 +338,8 @@ async fn embed_and_stop(mut config: Config) { #[cfg(target_os = "linux")] let owned = { - let quic_sockets = bound(quic.port()); - let http_sockets = bound(http.port()); + let quic_sockets = bound(socket2::Type::DGRAM, quic.port()); + let http_sockets = bound(socket2::Type::STREAM, http.port()); assert!(!quic_sockets.is_empty(), "no QUIC socket found on {quic}"); assert!(!http_sockets.is_empty(), "no HTTP listener found on {http}"); &quic_sockets | &http_sockets @@ -410,7 +412,6 @@ async fn shared_tokio_custom_route_and_quic() { /// Thread-per-core Tokio workers. Linux-only: the mode binds with `SO_REUSEPORT`. #[cfg(target_os = "linux")] #[tokio::test] -#[ignore = "flaky: the owner's sockets outlive `run` under load; quest/m0/test-flakes-2/embed-sockets-outlived.md"] async fn worker_tokio_custom_route_and_quic() { let dir = tempfile::tempdir().expect("tempdir"); let (cert, key) = certificate(dir.path()); diff --git a/sh/gh/test-inventory.allow b/sh/gh/test-inventory.allow index 76ccc34519f..3c9d43f6d8c 100644 --- a/sh/gh/test-inventory.allow +++ b/sh/gh/test-inventory.allow @@ -38,9 +38,6 @@ ignore moq-video::render::export::dx12::tests::media_foundation_encodes_the_expo # Quarantined flakes, re-enabled by the quest that fixes them. ignore moq-gst::sink::imp::tests::a_stale_completion_cannot_end_the_next_publication /quest/m0/test-flakes-2/gst-stale-completion.md -# Flaky, disabled until its quest fixes the cause. -ignore moq-relay::embed::worker_tokio_custom_route_and_quic /quest/m0/test-flakes-2/embed-sockets-outlived.md - # Need another implementation to speak the route wire; the quest ports it. ignore moq-net::test_interop::lite_tracks_interop /quest/m2/cluster-routing/routes/js.md