Repository navigation
docs(moq-uring): the shared locked-memory budget is mostly other programs' rings #8051
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
e5c9982
chore(quest): claim quest/m0/test-flakes-2/uring-memlock.md
kixelated 73734da
docs(moq-uring): measure the shared locked-memory budget; quest waits…
kixelated 5189cbb
quest: record the uring-memlock direction decision
kixelated f0b7075
Merge remote-tracking branch 'origin/main' into HEAD
kixelated File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,80 +1,54 @@ | ||
| # [S] io_uring tests fit the shared locked-memory budget | ||
| # [XS] The dev host's locked-memory limit fits parallel sessions | ||
|
|
||
| ## Goal | ||
|
|
||
| moq-relay's and moq-uring's io_uring tests stop failing ring setup with | ||
| `ENOMEM` on the user's shared `RLIMIT_MEMLOCK` when several test runs share | ||
| one host, as they do under parallel quest sessions. The fix works at the | ||
| cause, by bounding how much locked memory the tests take at once across | ||
| invocations or by sizing their rings to fit, not by raising the limit | ||
| blindly. Done when the PR shows concurrent runs that failed before now | ||
| passing on a host with the default 8 MiB limit. | ||
| `ENOMEM` on the user's shared `RLIMIT_MEMLOCK` when many agent sessions | ||
| share the maintainer's host. Done when `ulimit -Hl` in a new agent session | ||
| on that host prints at least 65536, the 64 MiB hosted CI already uses. | ||
|
|
||
| ## Plan | ||
|
|
||
| Added 2026-11-05 (maintainer, /quest-complete follow-ups). | ||
|
|
||
| What [moq-dev/moq#7593](https://github.com/moq-dev/moq/pull/7593) (merged | ||
| 2026-11-02 17:04Z) did: it widened the `io-uring` nextest group in | ||
| `.config/nextest.toml` to every moq-relay test naming `uring` as a whole | ||
| segment, with `package(moq-uring)` already in it. It also made a refused | ||
| relay worker report its `RLIMIT_MEMLOCK` error through `Ready::wait` | ||
| instead of a timeout. It kept the group at `max-threads = 4`. Its quest | ||
| scoped out the rest on purpose: "The group caps rings only within one | ||
| nextest invocation; separate invocations still share the budget, as | ||
| `rs/moq-uring/README.md` documents, and this quest does not change that." | ||
| It gives no reason beyond that scope. A nextest group also does nothing | ||
| under plain `cargo test`, which the land fixers run. | ||
|
|
||
| So the budget is still shared by: | ||
|
|
||
| - separate nextest invocations from the same user (several checkouts or | ||
| sessions running `just check` at once), each allowed four ring tests; | ||
| - plain `cargo test`, which runs moq-uring's and moq-relay's ring tests in | ||
| parallel with no group at all. | ||
|
|
||
| `sh/rs/nextest.sh` raises hosted CI to 64 MiB, and no CI or merge-queue run | ||
| from 2026-11-03 to 2026-11-06 failed on it. The one io_uring failure in that | ||
| window, | ||
| [run 41736459964](https://github.com/moq-dev/moq/actions/runs/41736459964)'s | ||
| `embed::uring_custom_route_and_quic`, was the socket-ownership check | ||
| [moq-dev/moq#7823](https://github.com/moq-dev/moq/pull/7823) fixed. The | ||
| failures below are local runs, on hosts with an 8 MiB limit where the | ||
| report names it, and all but the last are on bases that include #7593: | ||
|
|
||
| - [moq-dev/moq#7951](https://github.com/moq-dev/moq/pull/7951) (PR body, | ||
| 2026-11-05): two `quest gate just ci` runs on `eb0c56312b` "failed only in | ||
| io_uring tests in moq-relay and moq-uring: `io_uring setup failed: Cannot | ||
| allocate memory ... RLIMIT_MEMLOCK (8192 KiB), shared by every process of | ||
| this user`". The third run passed unchanged. | ||
| - [moq-dev/moq#7375](https://github.com/moq-dev/moq/pull/7375) (PR body, | ||
| merged 2026-11-03): `quest gate just check` failed once each in | ||
| `bitrate_caps::uring_session_is_held_to_its_caps` and | ||
| `uring::tests::a_group_stopped_before_serving_joins`, both `ENOMEM` on the | ||
| 8 MiB limit. | ||
| - [moq-dev/moq#7313](https://github.com/moq-dev/moq/pull/7313) (PR body, | ||
| merged 2026-11-03): a moq-relay uring test in `just check`. Then 11 | ||
| moq-uring integration tests failed in a single | ||
| `just rs test --workspace --no-fail-fast`. One invocation stays within its | ||
| four-test group, so other processes of the same user must have held part | ||
| of the budget. | ||
| - [moq-dev/moq#7508](https://github.com/moq-dev/moq/pull/7508) (PR body, | ||
| merged 2026-11-03): `moq-uring::integration mtu::discovery_finds_a_larger_mtu` | ||
| with `Cannot allocate memory`. The PR doesn't say whether that run | ||
| included #7593. | ||
|
|
||
| Reproduce first: on a host with an 8 MiB limit, run two or three | ||
| `just rs test -p moq-uring` or relay `--features io-uring` invocations at | ||
| once (and one plain `cargo test`) until a ring setup fails. Then measure what | ||
| one ring test locks. `rs/moq-uring/src/worker.rs` counts 16 bytes per CQ | ||
| entry, but the registered buffer rings (`register_buf_ring_with_flags` in | ||
| `udp.rs`) may be most of it. Integration tests keep the production | ||
| 256/2048 geometry, while unit tests use 128/512. | ||
|
|
||
| Choose the fix from that measurement and prefer the simplest. Options | ||
| include serializing ring tests across invocations through a user-wide lock | ||
| that both nextest and `cargo test` respect, or sizing the test rings and | ||
| buffer pools so the worst-case concurrency fits. Raise the limit only with | ||
| a measured reason, not as the fix. | ||
|
|
||
| Public API: none. Wire: none. | ||
| Decided 2026-11-06 (maintainer, /quest-complete): ✅ raise the host limit and | ||
| keep this quest as a host action. Not chosen: cap the `io-uring` test group at | ||
| one thread (about +45 s per workspace check), or first make the 10 s endpoint | ||
| tests fast; add a public ring-geometry knob so tests use small rings. | ||
|
|
||
| Measured 2026-11-06 in [moq-dev/moq#8051](https://github.com/moq-dev/moq/pull/8051) | ||
| on that host (kernel 7.0, 8 MiB limit). The tests are not what fills the | ||
| budget: | ||
|
|
||
| - Processes other than tests held 4.1 to 4.9 MiB. Most of it was Node.js: | ||
| libuv gives every event loop a 256-entry epoll-batching ring (28 KiB), and | ||
| `UV_USE_IO_URING` does not turn that ring off. Each agent session's | ||
| chrome-devtools-mcp alone holds nine of them (about 250 KiB), so the | ||
| baseline grows with the number of sessions. | ||
| - One `just rs test -p moq-uring -p moq-relay --features moq-relay/io-uring` | ||
| run needs 0.5 to 1 MiB of headroom. Twelve such runs started at once all | ||
| passed at that baseline. | ||
| - The failures reported in [moq-dev/moq#7313](https://github.com/moq-dev/moq/pull/7313) | ||
| (eleven moq-uring tests in one workspace run) leave the baseline above | ||
| 7 MiB at the time. No test-side cap brings that back under 8 MiB. | ||
|
|
||
| Measured and set aside: | ||
|
|
||
| - Capping the `io-uring` nextest group at one thread halves the headroom a | ||
| run needs, but slows `just rs test --workspace --features moq-relay/io-uring` | ||
| from 101 to 108 s to 149 to 153 s. Three endpoint tests in | ||
| `rs/moq-uring/tests/integration/endpoint.rs` each sit out moq-quic's 10 s | ||
| handshake timeout while holding a ring, which makes the serial group the | ||
| tail. | ||
| - A lock shared by every invocation counts its wait against nextest's | ||
| hang timeout. It also needs nextest's experimental wrapper scripts or a | ||
| hook in every ring test. | ||
| - Giving integration tests the smaller unit-test rings needs a public | ||
| geometry knob and stops testing production geometry. | ||
|
|
||
| ## Waiting | ||
|
|
||
| - Maintainer: raise the host user's hard `RLIMIT_MEMLOCK` to 64 MiB, for | ||
| example `LimitMEMLOCK=64M` in a `/etc/systemd/system/user@.service.d/` | ||
| drop-in plus `<user> hard memlock 65536` in `/etc/security/limits.d/` for | ||
| login shells. An agent cannot do this because it needs root; | ||
| `sh/rs/nextest.sh` raises the limit with `sudo prlimit` only on hosted CI. | ||
| Check with `ulimit -Hl` in a fresh session. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
UV_USE_IO_URING=0does drop this ring on released libuv below 1.50:uv__use_io_uring()consults the env var for every ring anduv__iou_initreturns beforeio_uring_setup, so Node 20 (libuv 1.46) creates none of these rings when it is set. The unconditionalreturn 1for non-SQPOLL rings is only on v1.50.0 and later (the body linksv1.x), so as written the sentence is false for anyone on that boundary and hides a remedy that still works there. The same sentence appears inquest/m0/test-flakes-2/uring-memlock.md:22-24.