quic: a UDP port whose streams are pipes, on ngtcp2 over OpenSSL 3.5

[MDA2AV]

Summary

ioxd_bind_quic(port, certs, alpn): a QUIC port on every worker, ngtcp2 for the transport, OpenSSL 3.5's QUIC TLS API through ngtcp2's ossl backend for TLS 1.3, from the same certificate store a TLS port uses (SNI as before, ALPN from the port's list). Every stream a peer opens is served by the handler of ioxd_run_pipes as a pipe of its own, on a coroutine of its own: what the peer sent is what the pipe reads, what the handler writes goes back on the stream, returning ends it. The line-echo example serves TCP and QUIC with one function.

How it follows ioxide, and where it differs

ioxide's layering is kept: the core binds the UDP port and routes datagrams by connection id, the engine (ngtcp2) owns the handshake, packet protection, loss recovery and flow control, and a stream's bytes are copied out of the engine's callbacks into the reader's queue and retained for the writer until acknowledged. Two differences:

• One TLS stack. ioxide bundles ngtcp2 with picotls. libioxd already depends on OpenSSL, and OpenSSL 3.5 has the QUIC TLS API ngtcp2 needs, so QUIC here is libngtcp2_crypto_ossl over the same OpenSSL, and certs.c builds a QUIC context beside each host's. One store, one reload, one SNI table.
• No cross-worker forwarding yet. The ids this server mints carry the worker in their first byte (cid[0] mod workers), as ioxide's do, so a reuseport filter can steer a peer that changed address back to its worker; the filter and the forwarding fallback are not in this PR. A moved peer lands on whichever worker the 4-tuple hashes to and is dropped there as a stale id.

The port on a worker (lib/quic/quic.c)

• the worker's own UDP socket (SO_REUSEPORT, no fragmentation), one multishot recvmsg into the provided buffers with the peer's address in front
• connections in an open-addressing table keyed by connection id; unknown ids get a stateless reset, unknown versions a Version Negotiation, an Initial a connection
• one kernel timeout per worker, armed at the earliest expiry of a heap of connections
• every send is what ngtcp2_conn_write_aggregate_pkt2 wrote, as a GSO train; a kernel that refuses UDP_SEGMENT turns it off for the port
• callbacks never resume a coroutine: they queue the stream, the loop resumes it after the cycle

The stream (lib/quic/stream.c)

Received bytes go into chunks the pipe reader pops and gives back; giving one back opens the peer's window by as much. Written bytes go into chunks retained until acknowledged, since ngtcp2 keeps pointers into them; a writer parks past 256 KB unacknowledged. Streams with bytes to send take turns in the connection's pump, several sharing a packet.

Underneath

The pipe reads from and writes to a link (ioxd_pipe_link): the TCP connection's ops in conn.c, the stream's in stream.c. The loop has one more tag, TAG_CALL, a completion that calls its operation back, which is what the QUIC recv, sends and timer use.

Build

QUIC is in when pkg-config finds libngtcp2 and libngtcp2_crypto_ossl (make QUIC=1 insists, QUIC=0 leaves it out; CMake IOXD_QUIC AUTO/ON/OFF). OpenSSL is found through pkg-config too, so a private OpenSSL 3.5 is a PKG_CONFIG_PATH away. Without QUIC, ioxd_bind_quic says so and returns -1.

Tests

tests/quic.py (aioquic) against the pipe fixture, which serves the echo on QUIC streams beside TCP when IOXD_CERTS is set: handshake and ALPN, a line echoed, 50 streams at once, a stream held open, 2 MB through one stream read back while it arrives, a client reset, a one-way stream, 40 connections at once, a refused protocol, an idle close. make check runs it in a QUIC build and skips it otherwise.

Gate on this branch: unit, router, smoke, conformance, stress, tls_early, pipes, quic, check-tiny, tidy, QUIC=0 and TLS=0 builds, ctest, tlsfuzzer.

Not here yet

HTTP/3 (nghttp3 on these streams), and the reuseport steering above.

🤖 Generated with Claude Code

https://claude.ai/code/session_013wYnJvEFUjKEpGLkyTLt9P