From 696a290f0b65bd654af0d217851737e955bed882 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 13 Nov 2026 13:47:24 -0800 Subject: [PATCH 1/5] chore(quest): claim quest/m0/test-flakes-2/README.md Quest-Claim: c8c31e11336b7b12 From 892f6ea5ee5af04cbf0b68d3d3e107c0f541fc81 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 13 Nov 2026 13:59:20 -0800 Subject: [PATCH 2/5] quest: plan the io_uring setup ENOMEM seen under load Co-Authored-By: Claude Opus 5.5 --- quest/m0/test-flakes-2/uring-setup-enomem.md | 41 ++++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100644 quest/m0/test-flakes-2/uring-setup-enomem.md diff --git a/quest/m0/test-flakes-2/uring-setup-enomem.md b/quest/m0/test-flakes-2/uring-setup-enomem.md new file mode 100644 index 00000000000..4a06eb1dd94 --- /dev/null +++ b/quest/m0/test-flakes-2/uring-setup-enomem.md @@ -0,0 +1,41 @@ +# [S] io_uring ring setup survives a fragmented host + +## Goal + +moq-relay's io_uring tests, such as `bitrate_caps::uring_session_is_held_to_its_caps`, +stop failing at `moq_uring::Worker::new` with `ENOMEM` on a loaded Linux 7.3 +host, and the relay's startup error names the cause the kernel actually hit. + +## Plan + +Seen 2026-11-13 on fedora (Linux 7.3.3, `ulimit -l` 8192) during a loaded +`just check`: the test's capability probe got `Cannot allocate memory (os error +12)` from `io_uring_setup` within 10 ms, and the rerun passed. + +It was not `RLIMIT_MEMLOCK`, despite the message blaming it. Since +[moq-dev/moq#8641](https://github.com/moq-dev/moq/pull/8641) showed 7.3.3 sets +up a ring under a limit of 0, and v7.3's `io_uring.c` creates both ring regions +with `io_create_region(NULL, ...)`, so nothing is charged to the user. The +host's memory cgroups recorded no `max` or `oom` events either. + +The likely cause, from v7.3's `io_uring/memmap.c` `io_region_allocate_pages`: +it first tries one compound allocation of `get_order(size)`, and on failure +falls back to `alloc_pages_bulk_node`, returning `ENOMEM` whenever the bulk +call hands back fewer pages than asked. The bulk allocator returns a partial +batch rather than reclaiming. The production CQ of 2048 entries is 32 KiB of +CQEs plus the ring header and SQ array, so the region spills just past 32 KiB +into an order-4 (64 KiB) compound allocation, above `PAGE_ALLOC_COSTLY_ORDER`, +which the page allocator gives up on quickly when memory is fragmented. + +Open (implementer): confirm the cause, for example by fragmenting memory or +tracing `io_region_allocate_pages`'s returns while the test loops, before +changing code. Candidate fixes, in order of preference: + +- Size the rings so each region stays at or below order 3, such as + `IORING_SETUP_NO_SQARRAY` to drop the SQ array, if that alone brings the CQ + region under 32 KiB plus the header. This touches the CQ sizing rationale in + `rs/moq-uring/src/worker.rs`. +- Report it upstream if the bulk fallback's partial return is the whole cause. + +Never retry setup in the test. `Error::ring`'s message should stop naming +`RLIMIT_MEMLOCK` first on kernels that do not charge it. From 7176aef32d17e57448d3dc771fa9474a8224f4c7 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 13 Nov 2026 14:05:34 -0800 Subject: [PATCH 3/5] quest: record the loaded just check runs Co-Authored-By: Claude Opus 5.5 --- quest/m0/test-flakes-2/README.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/quest/m0/test-flakes-2/README.md b/quest/m0/test-flakes-2/README.md index 9fb69a97173..fa1ee93ad65 100644 --- a/quest/m0/test-flakes-2/README.md +++ b/quest/m0/test-flakes-2/README.md @@ -28,7 +28,12 @@ Rules every child keeps: budget). Wait out the jitter's whole window, not a count of small steps. This README's own work, after the children: run `just check --all` several -times on a loaded machine, as the first round did. +times on a loaded machine, as the first round did. Done 2026-11-13 on fedora +(32 cores, load average 4 to 14 from other agents' sessions): three runs all +passed. The one failure seen on that host the same day, moq-relay's +io_uring `ENOMEM` at ring setup, is planned in +[uring-setup-enomem](/quest/m0/test-flakes-2/uring-setup-enomem.md). The epic +completes when that child lands. The redundant failover drift moved to m0 on 2026-10-20 (maintainer) and is fixed by [moq-dev/moq#6113](https://github.com/moq-dev/moq/pull/6113). From 4898825d4e3125ddb4171cbd92b183a135c4436e Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 13 Nov 2026 19:14:12 -0800 Subject: [PATCH 4/5] quest: record the uring-setup-enomem placement decision Co-Authored-By: Claude Sonnet 5.5 --- quest/m0/test-flakes-2/README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/quest/m0/test-flakes-2/README.md b/quest/m0/test-flakes-2/README.md index fa1ee93ad65..f25d1eefb69 100644 --- a/quest/m0/test-flakes-2/README.md +++ b/quest/m0/test-flakes-2/README.md @@ -32,8 +32,10 @@ times on a loaded machine, as the first round did. Done 2026-11-13 on fedora (32 cores, load average 4 to 14 from other agents' sessions): three runs all passed. The one failure seen on that host the same day, moq-relay's io_uring `ENOMEM` at ring setup, is planned in -[uring-setup-enomem](/quest/m0/test-flakes-2/uring-setup-enomem.md). The epic -completes when that child lands. +[uring-setup-enomem](/quest/m0/test-flakes-2/uring-setup-enomem.md). +Decided 2026-11-13 (maintainer, `/quest-complete`): that child stays in m0 +under this epic, so the epic stays open until it lands (rejected: moving it to +m1, moving it to m3). The epic completes when that child lands. The redundant failover drift moved to m0 on 2026-10-20 (maintainer) and is fixed by [moq-dev/moq#6113](https://github.com/moq-dev/moq/pull/6113). From 8df9e005aec32ed7537446b1c4564d35818a6ee5 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 13 Nov 2026 19:20:19 -0800 Subject: [PATCH 5/5] quest: correct the order-3 ring sizing fix for the ring header Co-Authored-By: Claude Sonnet 5.5 --- quest/m0/test-flakes-2/uring-setup-enomem.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/quest/m0/test-flakes-2/uring-setup-enomem.md b/quest/m0/test-flakes-2/uring-setup-enomem.md index 4a06eb1dd94..3acf33db539 100644 --- a/quest/m0/test-flakes-2/uring-setup-enomem.md +++ b/quest/m0/test-flakes-2/uring-setup-enomem.md @@ -31,10 +31,11 @@ Open (implementer): confirm the cause, for example by fragmenting memory or tracing `io_region_allocate_pages`'s returns while the test loops, before changing code. Candidate fixes, in order of preference: -- Size the rings so each region stays at or below order 3, such as - `IORING_SETUP_NO_SQARRAY` to drop the SQ array, if that alone brings the CQ - region under 32 KiB plus the header. This touches the CQ sizing rationale in - `rs/moq-uring/src/worker.rs`. +- Size the rings so each region stays at or below order 3 (32 KiB). The + rings region holds the header and SQ array besides the CQEs, and 2048 CQEs + already fill 32 KiB, so `IORING_SETUP_NO_SQARRAY` alone cannot do it: the CQ + must shrink too, such as to 1024 entries. This touches the CQ sizing + rationale in `rs/moq-uring/src/worker.rs`. - Report it upstream if the bulk fallback's partial return is the whole cause. Never retry setup in the test. `Error::ring`'s message should stop naming