Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion quest/m0/test-flakes-2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,7 @@ Public API: none. Wire: none.
- [Test PulseAudio servers stay off the desktop's D-Bus](/quest/m0/test-flakes-2/pulse-dbus.md) - concurrent media runs on one desktop never collide on `org.PulseAudio1`
- [moq-net's pool-debt flight test holds its bound in parallel runs](/quest/m0/test-flakes-2/pool-debt-flight.md) - `pool_debt_sheds_a_flight_without_its_cached_group` stays under `2 * POOL` when it shares a process with moq-net's other tests
- [The Python audio playout-delay test reads its frames](/quest/m0/test-flakes-2/audio-playout-delay.md) - `test_audio_consumer_reports_its_playout_delay` stops timing out on its first read in the merge queue
- [io_uring tests fit the shared locked-memory budget](/quest/m0/test-flakes-2/uring-memlock.md) - concurrent moq-relay and moq-uring test runs stop exhausting the user's `RLIMIT_MEMLOCK`, fixed at the cause rather than by raising the limit
- [The dev host's locked-memory limit fits parallel sessions](/quest/m0/test-flakes-2/uring-memlock.md) - the maintainer's host raises the user's `RLIMIT_MEMLOCK`, which Node.js rings from parallel agent sessions fill before io_uring tests start
- [The FFI concurrent-ready test wakes both calls under load](/quest/m0/test-flakes-2/ffi-concurrent-ready.md) - `concurrent_ready_calls_all_wake` stops stalling its reader drain under load and is re-enabled
- [lite-05 copy fill checks idleness without wall time](/quest/m0/test-flakes-2/copy-fill-idle.md) - the copy-fill test proves no FETCH went out on the `Manual` clock, not a 200 ms real-time wait
- [OBS TSan runs report no leaked FFI worker thread](/quest/m0/test-flakes-2/obs-tsan-thread-leak.md) - the OBS TSan tests stop flagging a finished `moq-ffi-worker` thread at exit under load
120 changes: 47 additions & 73 deletions quest/m0/test-flakes-2/uring-memlock.md
Original file line number Diff line number Diff line change
@@ -1,80 +1,54 @@
# [S] io_uring tests fit the shared locked-memory budget
# [XS] The dev host's locked-memory limit fits parallel sessions

## Goal

moq-relay's and moq-uring's io_uring tests stop failing ring setup with
`ENOMEM` on the user's shared `RLIMIT_MEMLOCK` when several test runs share
one host, as they do under parallel quest sessions. The fix works at the
cause, by bounding how much locked memory the tests take at once across
invocations or by sizing their rings to fit, not by raising the limit
blindly. Done when the PR shows concurrent runs that failed before now
passing on a host with the default 8 MiB limit.
`ENOMEM` on the user's shared `RLIMIT_MEMLOCK` when many agent sessions
share the maintainer's host. Done when `ulimit -Hl` in a new agent session
on that host prints at least 65536, the 64 MiB hosted CI already uses.

## Plan

Added 2026-11-05 (maintainer, /quest-complete follow-ups).

What [moq-dev/moq#7593](https://github.com/moq-dev/moq/pull/7593) (merged
2026-11-02 17:04Z) did: it widened the `io-uring` nextest group in
`.config/nextest.toml` to every moq-relay test naming `uring` as a whole
segment, with `package(moq-uring)` already in it. It also made a refused
relay worker report its `RLIMIT_MEMLOCK` error through `Ready::wait`
instead of a timeout. It kept the group at `max-threads = 4`. Its quest
scoped out the rest on purpose: "The group caps rings only within one
nextest invocation; separate invocations still share the budget, as
`rs/moq-uring/README.md` documents, and this quest does not change that."
It gives no reason beyond that scope. A nextest group also does nothing
under plain `cargo test`, which the land fixers run.

So the budget is still shared by:

- separate nextest invocations from the same user (several checkouts or
sessions running `just check` at once), each allowed four ring tests;
- plain `cargo test`, which runs moq-uring's and moq-relay's ring tests in
parallel with no group at all.

`sh/rs/nextest.sh` raises hosted CI to 64 MiB, and no CI or merge-queue run
from 2026-11-03 to 2026-11-06 failed on it. The one io_uring failure in that
window,
[run 41736459964](https://github.com/moq-dev/moq/actions/runs/41736459964)'s
`embed::uring_custom_route_and_quic`, was the socket-ownership check
[moq-dev/moq#7823](https://github.com/moq-dev/moq/pull/7823) fixed. The
failures below are local runs, on hosts with an 8 MiB limit where the
report names it, and all but the last are on bases that include #7593:

- [moq-dev/moq#7951](https://github.com/moq-dev/moq/pull/7951) (PR body,
2026-11-05): two `quest gate just ci` runs on `eb0c56312b` "failed only in
io_uring tests in moq-relay and moq-uring: `io_uring setup failed: Cannot
allocate memory ... RLIMIT_MEMLOCK (8192 KiB), shared by every process of
this user`". The third run passed unchanged.
- [moq-dev/moq#7375](https://github.com/moq-dev/moq/pull/7375) (PR body,
merged 2026-11-03): `quest gate just check` failed once each in
`bitrate_caps::uring_session_is_held_to_its_caps` and
`uring::tests::a_group_stopped_before_serving_joins`, both `ENOMEM` on the
8 MiB limit.
- [moq-dev/moq#7313](https://github.com/moq-dev/moq/pull/7313) (PR body,
merged 2026-11-03): a moq-relay uring test in `just check`. Then 11
moq-uring integration tests failed in a single
`just rs test --workspace --no-fail-fast`. One invocation stays within its
four-test group, so other processes of the same user must have held part
of the budget.
- [moq-dev/moq#7508](https://github.com/moq-dev/moq/pull/7508) (PR body,
merged 2026-11-03): `moq-uring::integration mtu::discovery_finds_a_larger_mtu`
with `Cannot allocate memory`. The PR doesn't say whether that run
included #7593.

Reproduce first: on a host with an 8 MiB limit, run two or three
`just rs test -p moq-uring` or relay `--features io-uring` invocations at
once (and one plain `cargo test`) until a ring setup fails. Then measure what
one ring test locks. `rs/moq-uring/src/worker.rs` counts 16 bytes per CQ
entry, but the registered buffer rings (`register_buf_ring_with_flags` in
`udp.rs`) may be most of it. Integration tests keep the production
256/2048 geometry, while unit tests use 128/512.

Choose the fix from that measurement and prefer the simplest. Options
include serializing ring tests across invocations through a user-wide lock
that both nextest and `cargo test` respect, or sizing the test rings and
buffer pools so the worst-case concurrency fits. Raise the limit only with
a measured reason, not as the fix.

Public API: none. Wire: none.
Decided 2026-11-06 (maintainer, /quest-complete): ✅ raise the host limit and
keep this quest as a host action. Not chosen: cap the `io-uring` test group at
one thread (about +45 s per workspace check), or first make the 10 s endpoint
tests fast; add a public ring-geometry knob so tests use small rings.

Measured 2026-11-06 in [moq-dev/moq#8051](https://github.com/moq-dev/moq/pull/8051)
on that host (kernel 7.0, 8 MiB limit). The tests are not what fills the
budget:

- Processes other than tests held 4.1 to 4.9 MiB. Most of it was Node.js:
libuv gives every event loop a 256-entry epoll-batching ring (28 KiB), and
`UV_USE_IO_URING` does not turn that ring off. Each agent session's
chrome-devtools-mcp alone holds nine of them (about 250 KiB), so the
baseline grows with the number of sessions.
- One `just rs test -p moq-uring -p moq-relay --features moq-relay/io-uring`
run needs 0.5 to 1 MiB of headroom. Twelve such runs started at once all
passed at that baseline.
- The failures reported in [moq-dev/moq#7313](https://github.com/moq-dev/moq/pull/7313)
(eleven moq-uring tests in one workspace run) leave the baseline above
7 MiB at the time. No test-side cap brings that back under 8 MiB.

Measured and set aside:

- Capping the `io-uring` nextest group at one thread halves the headroom a
run needs, but slows `just rs test --workspace --features moq-relay/io-uring`
from 101 to 108 s to 149 to 153 s. Three endpoint tests in
`rs/moq-uring/tests/integration/endpoint.rs` each sit out moq-quic's 10 s
handshake timeout while holding a ring, which makes the serial group the
tail.
- A lock shared by every invocation counts its wait against nextest's
hang timeout. It also needs nextest's experimental wrapper scripts or a
hook in every ring test.
- Giving integration tests the smaller unit-test rings needs a public
geometry knob and stops testing production geometry.

## Waiting

- Maintainer: raise the host user's hard `RLIMIT_MEMLOCK` to 64 MiB, for
example `LimitMEMLOCK=64M` in a `/etc/systemd/system/user@.service.d/`
drop-in plus `<user> hard memlock 65536` in `/etc/security/limits.d/` for
login shells. An agent cannot do this because it needs root;
`sh/rs/nextest.sh` raises the limit with `sudo prlimit` only on hosted CI.
Check with `ulimit -Hl` in a fresh session.
11 changes: 10 additions & 1 deletion rs/moq-uring/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,10 +153,19 @@ Unit tests use 128 SQ entries and 512 CQ entries instead of the production
256/2048 geometry. Integration tests still exercise production rings.
The nextest group allows four ring tests per invocation.

Other programs draw on the same budget. libuv gives every event loop a
256-entry ring of 28 KiB, and it creates these without checking
`UV_USE_IO_URING`. A desktop running nine agent sessions held 4 to 5 MiB of an
8 MiB limit before any test started, mostly in Node.js tool servers. A
Comment on lines +157 to +159

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UV_USE_IO_URING=0 does drop this ring on released libuv below 1.50: uv__use_io_uring() consults the env var for every ring and uv__iou_init returns before io_uring_setup, so Node 20 (libuv 1.46) creates none of these rings when it is set. The unconditional return 1 for non-SQPOLL rings is only on v1.50.0 and later (the body links v1.x), so as written the sentence is false for anyone on that boundary and hides a remedy that still works there. The same sentence appears in quest/m0/test-flakes-2/uring-memlock.md:22-24.

Suggested change
256-entry ring of 28 KiB, and it creates these without checking
`UV_USE_IO_URING`. A desktop running nine agent sessions held 4 to 5 MiB of an
8 MiB limit before any test started, mostly in Node.js tool servers. A
256-entry ring of 28 KiB, and libuv 1.50 and later creates these without checking
`UV_USE_IO_URING`; earlier versions let `UV_USE_IO_URING=0` turn it off. A desktop
running nine agent sessions held 4 to 5 MiB of an

production-geometry worker takes 52 KiB and a unit-test worker 20 KiB.
One `moq-uring` plus `moq-relay` test invocation needs about 0.5 to 1 MiB
of what is left. Count the rings with `ls -l /proc/*/fd 2>/dev/null | grep -c io_uring`.

The Rust recipes that run nextest use `sh/rs/nextest.sh`: local Linux shells raise their
soft locked-memory limit to the host's hard limit, while GitHub-hosted jobs
raise both to 64 MiB before launching nextest. A NixOS self-hosted runner
should set `serviceConfig.LimitMEMLOCK = "64M"` on its runner service.
For many simultaneous local checks, configure the same user limit through
PAM limits or the service that starts the development shell; a Nix shell
PAM limits or the service that starts the development shell (for a systemd
user session, `LimitMEMLOCK=64M` in a `user@.service` drop-in); a Nix shell
cannot raise its inherited hard limit. Verify with `ulimit -Sl` and `ulimit -Hl`.
Loading