From e5c9982ed58cd8cabf72f499551c449079f63678 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 6 Nov 2026 11:49:52 -0800 Subject: [PATCH 1/3] chore(quest): claim quest/m0/test-flakes-2/uring-memlock.md Quest-Claim: 273274b425d0ea00 From 73734da403d0c9a8c2506c9cdb9921fe425feb5b Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 6 Nov 2026 12:57:41 -0800 Subject: [PATCH 2/3] docs(moq-uring): measure the shared locked-memory budget; quest waits on the host limit Co-Authored-By: Claude Opus 5.5 --- quest/m0/test-flakes-2/README.md | 2 +- quest/m0/test-flakes-2/uring-memlock.md | 115 +++++++++--------------- rs/moq-uring/README.md | 11 ++- 3 files changed, 53 insertions(+), 75 deletions(-) diff --git a/quest/m0/test-flakes-2/README.md b/quest/m0/test-flakes-2/README.md index 2682dacac1b..d48c7fc979a 100644 --- a/quest/m0/test-flakes-2/README.md +++ b/quest/m0/test-flakes-2/README.md @@ -112,5 +112,5 @@ Public API: none. Wire: none. - [Test PulseAudio servers stay off the desktop's D-Bus](/quest/m0/test-flakes-2/pulse-dbus.md) - concurrent media runs on one desktop never collide on `org.PulseAudio1` - [moq-net's pool-debt flight test holds its bound in parallel runs](/quest/m0/test-flakes-2/pool-debt-flight.md) - `pool_debt_sheds_a_flight_without_its_cached_group` stays under `2 * POOL` when it shares a process with moq-net's other tests - [The Python audio playout-delay test reads its frames](/quest/m0/test-flakes-2/audio-playout-delay.md) - `test_audio_consumer_reports_its_playout_delay` stops timing out on its first read in the merge queue -- [io_uring tests fit the shared locked-memory budget](/quest/m0/test-flakes-2/uring-memlock.md) - concurrent moq-relay and moq-uring test runs stop exhausting the user's `RLIMIT_MEMLOCK`, fixed at the cause rather than by raising the limit +- [The dev host's locked-memory limit fits parallel sessions](/quest/m0/test-flakes-2/uring-memlock.md) - the maintainer's host raises the user's `RLIMIT_MEMLOCK`, which Node.js rings from parallel agent sessions fill before io_uring tests start - [The FFI concurrent-ready test wakes both calls under load](/quest/m0/test-flakes-2/ffi-concurrent-ready.md) - `concurrent_ready_calls_all_wake` stops stalling its reader drain under load and is re-enabled diff --git a/quest/m0/test-flakes-2/uring-memlock.md b/quest/m0/test-flakes-2/uring-memlock.md index 3baefd8b2d0..2525c1c4ec4 100644 --- a/quest/m0/test-flakes-2/uring-memlock.md +++ b/quest/m0/test-flakes-2/uring-memlock.md @@ -1,80 +1,49 @@ -# [S] io_uring tests fit the shared locked-memory budget +# [XS] The dev host's locked-memory limit fits parallel sessions ## Goal moq-relay's and moq-uring's io_uring tests stop failing ring setup with -`ENOMEM` on the user's shared `RLIMIT_MEMLOCK` when several test runs share -one host, as they do under parallel quest sessions. The fix works at the -cause, by bounding how much locked memory the tests take at once across -invocations or by sizing their rings to fit, not by raising the limit -blindly. Done when the PR shows concurrent runs that failed before now -passing on a host with the default 8 MiB limit. +`ENOMEM` on the user's shared `RLIMIT_MEMLOCK` when many agent sessions +share the maintainer's host. Done when `ulimit -Hl` in a new agent session +on that host prints at least 65536, the 64 MiB hosted CI already uses. ## Plan -Added 2026-11-05 (maintainer, /quest-complete follow-ups). - -What [moq-dev/moq#7593](https://github.com/moq-dev/moq/pull/7593) (merged -2026-11-02 17:04Z) did: it widened the `io-uring` nextest group in -`.config/nextest.toml` to every moq-relay test naming `uring` as a whole -segment, with `package(moq-uring)` already in it. It also made a refused -relay worker report its `RLIMIT_MEMLOCK` error through `Ready::wait` -instead of a timeout. It kept the group at `max-threads = 4`. Its quest -scoped out the rest on purpose: "The group caps rings only within one -nextest invocation; separate invocations still share the budget, as -`rs/moq-uring/README.md` documents, and this quest does not change that." -It gives no reason beyond that scope. A nextest group also does nothing -under plain `cargo test`, which the land fixers run. - -So the budget is still shared by: - -- separate nextest invocations from the same user (several checkouts or - sessions running `just check` at once), each allowed four ring tests; -- plain `cargo test`, which runs moq-uring's and moq-relay's ring tests in - parallel with no group at all. - -`sh/rs/nextest.sh` raises hosted CI to 64 MiB, and no CI or merge-queue run -from 2026-11-03 to 2026-11-06 failed on it. The one io_uring failure in that -window, -[run 41736459964](https://github.com/moq-dev/moq/actions/runs/41736459964)'s -`embed::uring_custom_route_and_quic`, was the socket-ownership check -[moq-dev/moq#7823](https://github.com/moq-dev/moq/pull/7823) fixed. The -failures below are local runs, on hosts with an 8 MiB limit where the -report names it, and all but the last are on bases that include #7593: - -- [moq-dev/moq#7951](https://github.com/moq-dev/moq/pull/7951) (PR body, - 2026-11-05): two `quest gate just ci` runs on `eb0c56312b` "failed only in - io_uring tests in moq-relay and moq-uring: `io_uring setup failed: Cannot - allocate memory ... RLIMIT_MEMLOCK (8192 KiB), shared by every process of - this user`". The third run passed unchanged. -- [moq-dev/moq#7375](https://github.com/moq-dev/moq/pull/7375) (PR body, - merged 2026-11-03): `quest gate just check` failed once each in - `bitrate_caps::uring_session_is_held_to_its_caps` and - `uring::tests::a_group_stopped_before_serving_joins`, both `ENOMEM` on the - 8 MiB limit. -- [moq-dev/moq#7313](https://github.com/moq-dev/moq/pull/7313) (PR body, - merged 2026-11-03): a moq-relay uring test in `just check`. Then 11 - moq-uring integration tests failed in a single - `just rs test --workspace --no-fail-fast`. One invocation stays within its - four-test group, so other processes of the same user must have held part - of the budget. -- [moq-dev/moq#7508](https://github.com/moq-dev/moq/pull/7508) (PR body, - merged 2026-11-03): `moq-uring::integration mtu::discovery_finds_a_larger_mtu` - with `Cannot allocate memory`. The PR doesn't say whether that run - included #7593. - -Reproduce first: on a host with an 8 MiB limit, run two or three -`just rs test -p moq-uring` or relay `--features io-uring` invocations at -once (and one plain `cargo test`) until a ring setup fails. Then measure what -one ring test locks. `rs/moq-uring/src/worker.rs` counts 16 bytes per CQ -entry, but the registered buffer rings (`register_buf_ring_with_flags` in -`udp.rs`) may be most of it. Integration tests keep the production -256/2048 geometry, while unit tests use 128/512. - -Choose the fix from that measurement and prefer the simplest. Options -include serializing ring tests across invocations through a user-wide lock -that both nextest and `cargo test` respect, or sizing the test rings and -buffer pools so the worst-case concurrency fits. Raise the limit only with -a measured reason, not as the fix. - -Public API: none. Wire: none. +Measured 2026-11-06 in [moq-dev/moq#8051](https://github.com/moq-dev/moq/pull/8051) +on that host (kernel 7.0, 8 MiB limit). The tests are not what fills the +budget: + +- Processes other than tests held 4.1 to 4.9 MiB. Most of it was Node.js: + libuv gives every event loop a 256-entry epoll-batching ring (28 KiB), and + `UV_USE_IO_URING` does not turn that ring off. Each agent session's + chrome-devtools-mcp alone holds nine of them (about 250 KiB), so the + baseline grows with the number of sessions. +- One `just rs test -p moq-uring -p moq-relay --features moq-relay/io-uring` + run needs 0.5 to 1 MiB of headroom. Twelve such runs started at once all + passed at that baseline. +- The failures reported in [moq-dev/moq#7313](https://github.com/moq-dev/moq/pull/7313) + (eleven moq-uring tests in one workspace run) leave the baseline above + 7 MiB at the time. No test-side cap brings that back under 8 MiB. + +Measured and set aside: + +- Capping the `io-uring` nextest group at one thread halves the headroom a + run needs, but slows `just rs test --workspace --features moq-relay/io-uring` + from 101 to 108 s to 149 to 153 s. Three endpoint tests in + `rs/moq-uring/tests/integration/endpoint.rs` each sit out moq-quic's 10 s + handshake timeout while holding a ring, which makes the serial group the + tail. +- A lock shared by every invocation counts its wait against nextest's + hang timeout. It also needs nextest's experimental wrapper scripts or a + hook in every ring test. +- Giving integration tests the smaller unit-test rings needs a public + geometry knob and stops testing production geometry. + +## Waiting + +- Maintainer: raise the host user's hard `RLIMIT_MEMLOCK` to 64 MiB, for + example `LimitMEMLOCK=64M` in a `/etc/systemd/system/user@.service.d/` + drop-in plus ` hard memlock 65536` in `/etc/security/limits.d/` for + login shells. An agent cannot do this because it needs root; + `sh/rs/nextest.sh` raises the limit with `sudo prlimit` only on hosted CI. + Check with `ulimit -Hl` in a fresh session. diff --git a/rs/moq-uring/README.md b/rs/moq-uring/README.md index f76be803dae..d9dfa167b40 100644 --- a/rs/moq-uring/README.md +++ b/rs/moq-uring/README.md @@ -153,10 +153,19 @@ Unit tests use 128 SQ entries and 512 CQ entries instead of the production 256/2048 geometry. Integration tests still exercise production rings. The nextest group allows four ring tests per invocation. +Other programs draw on the same budget. libuv gives every event loop a +256-entry ring of 28 KiB, and it creates these without checking +`UV_USE_IO_URING`. A desktop running nine agent sessions held 4 to 5 MiB of an +8 MiB limit before any test started, mostly in Node.js tool servers. A +production-geometry worker takes 52 KiB and a unit-test worker 20 KiB. +One `moq-uring` plus `moq-relay` test invocation needs about 0.5 to 1 MiB +of what is left. Count the rings with `ls -l /proc/*/fd 2>/dev/null | grep -c io_uring`. + The Rust recipes that run nextest use `sh/rs/nextest.sh`: local Linux shells raise their soft locked-memory limit to the host's hard limit, while GitHub-hosted jobs raise both to 64 MiB before launching nextest. A NixOS self-hosted runner should set `serviceConfig.LimitMEMLOCK = "64M"` on its runner service. For many simultaneous local checks, configure the same user limit through -PAM limits or the service that starts the development shell; a Nix shell +PAM limits or the service that starts the development shell (for a systemd +user session, `LimitMEMLOCK=64M` in a `user@.service` drop-in); a Nix shell cannot raise its inherited hard limit. Verify with `ulimit -Sl` and `ulimit -Hl`. From 5189cbbcd5e66213a174b5e5ed1895afdea76534 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 6 Nov 2026 13:45:21 -0800 Subject: [PATCH 3/3] quest: record the uring-memlock direction decision Co-Authored-By: Claude Opus 5.5 --- quest/m0/test-flakes-2/uring-memlock.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/quest/m0/test-flakes-2/uring-memlock.md b/quest/m0/test-flakes-2/uring-memlock.md index 2525c1c4ec4..419676d4d2c 100644 --- a/quest/m0/test-flakes-2/uring-memlock.md +++ b/quest/m0/test-flakes-2/uring-memlock.md @@ -9,6 +9,11 @@ on that host prints at least 65536, the 64 MiB hosted CI already uses. ## Plan +Decided 2026-11-06 (maintainer, /quest-complete): ✅ raise the host limit and +keep this quest as a host action. Not chosen: cap the `io-uring` test group at +one thread (about +45 s per workspace check), or first make the 10 s endpoint +tests fast; add a public ring-geometry knob so tests use small rings. + Measured 2026-11-06 in [moq-dev/moq#8051](https://github.com/moq-dev/moq/pull/8051) on that host (kernel 7.0, 8 MiB limit). The tests are not what fills the budget: