ring: read errno, so the transient-failure handling actually works

[MDA2AV]

Fixes #220.

The bug, in three adjacent lines

[DllImport("libc", EntryPoint = "syscall")]                       // io_uring_setup
[DllImport("libc", EntryPoint = "syscall")]                       // io_uring_enter
[DllImport("libc", EntryPoint = "syscall", SetLastError = true)]  // io_uring_register  ← only this one

glibc's syscall() reports failure as -1 with the code in errno; it never returns -errno. Without SetLastError the code is lost, so three comparisons against specific errnos were dead:

Ring.cs:46	if (fd == -EINVAL) — the fallback for kernels without IORING_SETUP_NO_SQARRAY
SharedRing.cs:60	rc != -EINTR && rc != -EAGAIN && rc != -EBUSY
Incremental.cs:181	same

Measured on this machine before the fix, and now committed as tests:

io_uring_setup(0, &params)   → got -1, expected -22 (EINVAL)
io_uring_enter(-1, 0, 0, 0)  → got -1, expected  -9 (EBADF)

The last two are worse than inert — they guaranteed the opposite of their purpose. They exist so the loop survives an interrupted io_uring_enter. Instead, one signal delivered to a reactor thread returned -1, matched none of the exclusions, and ended that reactor: the process carried on with one fewer, nothing thrown, and a single stray line of output. There is no health or liveness surface anywhere in src/, and Run() returning normally is indistinguishable from a clean Stop().

Which signals actually do it (measured, and narrower than the issue claims): process-directed signals are not a trigger — 50 × kill(pid, SIGHUP) were all delivered to the main thread and both reactors survived. The triggers are thread-directed: tgkill/pthread_kill from tooling, and CPU-timer signals. SIGPROF is the one that will find you in production — an itimer-based profiler at 1 ms killed a reactor inside a 5 s window while it served 2.55 M requests.

One signal is not always enough, which is why this is intermittent rather than immediate: io_uring_enter returns the submit count when to_submit > 0, masking the EINTR, so the 250 ms ticker leaves an idle reactor alternating between vulnerable and not. Two signals ≥200 ms apart killed it every time. Verified by strace:

io_uring_enter(37, 0, 1, IORING_ENTER_GETEVENTS, NULL, 0) = -1 EINTR (Interrupted system call)
--- SIGHUP {si_code=SI_TKILL} ---
+++ exited with 0 +++

Three things the same investigation turned up

Run() had no try/finally. Anything thrown out of the loop skipped Teardown() and leaked the ring fd, both mmaps, the eventfd and the buffer slab. Since ring memory is charged against RLIMIT_MEMLOCK, leaking rings is exactly how a long-lived host ends up unable to create one.

A fatal errno now throws rather than breaking quietly. A reactor that vanishes while the process reports healthy is the worst of the options. The finally tears the ring down and the exception reaches whoever started the thread. Transients still carry on — and now genuinely do, which is the point of the fix.

Ring.Create retries ENOMEM, and this is the one with a live victim. Ring memory is released asynchronously after close(), so a host that creates and drops reactors faster than the kernel reclaims them gets ENOMEM while nothing is leaking. Reproduced standalone: 30,000 create-and-close cycles failed 14,788 times, and every failure cleared on a 5 ms retry.

This is what the GenHTTP acceptance suite hit — it reported only io_uring_setup failed: -1, which is why it looked like a mystery.

I over-attributed that in an earlier draft. Re-measured: the suite on its own peaks at 2.2 GB of the 8 GB budget across 41,728 ring setups with zero failures. locked_vm is a per-uid counter shared by every process of the same user, so the aborting run needed roughly 6 GB more charged from elsewhere — most likely the ring-churning probes I had running concurrently. The retry is still the right fix, and the diagnostic more so, but the suite is not by itself the cause.

Measured cost per reactor: 772 KB for the ring at the default RingEntries plus 64 KB for a 4096-slot buffer ring. The common 8 MB ulimit -l fits about ten. That is now in the README, along with the per-uid part, and the failure message names the errno and the cost.

Bench

Tcp/Raw, 4 reactors, four alternating pairs — SetLastError lands on io_uring_enter, the hottest syscall in the runtime:

-4.6%   +3.3%   -0.5%   +2.2%      mean +0.15%, two up two down

Independently measured at ~10 ns per enter, which at a realistic completion batch is ~0.6 ns per request — about 0.09% of the per-request budget, an order of magnitude under this box's noise floor.

The cheaper alternative (no SetLastError, read Marshal.GetLastSystemError() only on the failure branch) benchmarks free and survived 70,442 adversarial trials without a wrong read — but it reads errno after the GC-mode transition, so it depends on CoreCLR's current epilogue rather than on a contract. Not worth 0.09%.

Not addressed

The NO_SQARRAY fallback is now reachable for the first time, which means Ring.cs's _sqArray path has never executed — anywhere, ever, including CI. It reads correctly, but 6.1–6.5 has no coverage and nothing in the test suite can force the fallback. Worth a Ring.Create flags overload so CI can exercise it; I have not added one.

Also unaddressed: Reactor has no fault callback, so a host that wants to restart a dead reactor rather than take the exception has nowhere to hook. An OnFault sibling to OnStart would be the seam.

Behaviour change to note in release notes: a fatal io_uring_enter errno now propagates. A host calling Run() on a bare Thread will see the process terminate where it previously degraded silently. That is the intent, but it is a change.