Skip to content

Fall back to userspace WireGuard when the kernel lacks it [murmur:murmuration/vprox-oidc-claim-allowlists] - #5

Merged
maerlyn5 merged 1 commit into
mainfrom
userspace-wg
Oct 11, 2026
Merged

maerlyn5 merged 1 commit into
mainfrom
userspace-wg

Conversation

@maerlyn5

@maerlyn5 maerlyn5 commented Oct 11, 2026 •

Copy link
Copy Markdown

Modal VM sandboxes (runtime="vm", kernel 7.2.9) have no WireGuard kernel module. The published release sha-9ad4489… therefore fails before registration with error creating vprox interface: operation not supported. This PR adds a userspace WireGuard fallback to the client and keeps verified control-plane TLS, Modal OIDC authentication, reconnects and cleanup unchanged. The gateway still uses kernel WireGuard.

Change

  • CreateInterface: tries kernel WireGuard first, as before. If netlink returns EOPNOTSUPP (what ip link add … type wireguard reports as Unknown device type), the client logs that it is falling back. It then runs wireguard-go on a TUN interface with the same name and the default 1420 MTU.
    • The userspace device serves the standard UAPI socket at /var/run/wireguard/<ifname>.sock.
    • The existing wgctrl configuration path (peer, endpoint, keepalive, AllowedIPs), netlink address/link-up handling, the /connect and /disconnect requests, and the reconnect loop all work unchanged. wg show also works with it.
    • Any other interface-creation error still fails as before.
  • DeleteInterface: closes the userspace device. That removes the TUN interface and unlinks the UAPI socket. If the process is killed hard, the kernel destroys the non-persistent TUN anyway, and the next run's UAPIOpen replaces the stale socket.
  • Health check: if creating an unprivileged ICMP socket fails with EACCES, it retries with a raw socket. The kernel's default net.ipv4.ping_group_range (1 0) forbids unprivileged ICMP sockets, and minimal VMs without distribution sysctl defaults keep it. In that state every health check would fail and the client would loop on reconnect. I hit exactly this on the test VM. I haven't confirmed the value inside Modal VM sandboxes, but the fallback is a no-op where unprivileged ping already works.
  • Dependencies: golang.zx2c4.com/wireguard moves from an indirect to a direct dependency at the same version. The static binary grows by about 0.75 MB (20.9 MB → 21.7 MB). The existing release publisher builds amd64 and arm64 as before.

Tests

TestUserspaceClient_TunnelsReconnectsAndCleansUp (root only) runs this sequence:

  1. Creates the userspace device.
  2. Registers through the real TLS control-plane client against a fake gateway whose certificate is verified by hostname.
  3. Completes a real WireGuard handshake with an in-memory wireguard-go gateway that answers ICMP.
  4. Passes CheckConnection over the tunnel.
  5. Asserts that wgctrl sees a userspace device with the gateway peer and a recent handshake.
  6. Reconnects, health-checks again and sends /disconnect.
  7. Asserts that the interface and the UAPI socket are gone.

CI: a new step in the go test -race job compiles the lib tests and runs them under sudo. Tests that need root skip themselves elsewhere.

Regression check: before I added the raw-socket ping fallback, this test failed on the test VM (ping_group_range = 1 0) with socket: permission denied. With the fallback it passes.

Test evidence

  • Kernel without WireGuard, reproduced locally: I blocked and unloaded the wireguard module on a Debian 6.1 VM, and ip link add wgprobe type wireguard printed Error: Unknown device type., matching Modal. In that state, Client.CreateInterface logged kernel WireGuard is unavailable; using userspace WireGuard for vproxfb0, and wgctrl reported device type: userspace. Deleting it left no interface and no socket. With the module reloaded, the same call created a Linux kernel device. This was a one-off check, not a committed test.

  • Local checks: go test -race ./... passes, the lib suite passes as root, gofmt and actionlint are clean, and the linux/arm64 build succeeds.

  • Earlier live tests (from the request, not run against this PR): a standalone wireguard-go client in two fresh nonprod Modal VM sandboxes did all of the following:

    • registered with OIDC using the sandbox's MODAL_IDENTITY_TOKEN;
    • completed the WireGuard handshake with 104.197.117.12:50227, TLS name murmur-vprox.nonprod.prasso.ai;
    • passed a tunnel health ping, resolved private DNS, and reached the private API over verified HTTPS;
    • had a forbidden destination rejected;
    • got 401 for missing, invalid or tampered credentials, and had a wrong TLS hostname rejected.

    This PR builds the same wireguard-go client into vprox itself. Those live handshakes also needed the gateway's UDP DNAT reply fix, which is tracked separately in prassoai/murmuration#12043.

Not done

  • I deployed nothing and applied no infrastructure.
  • Live validation of this binary in a Modal VM sandbox is still outstanding. After merge, the publish job will produce a sha-<merge commit> release to pin and test.

Created via Murmur

🤖 Generated with Claude Code

Note

Fall back to userspace WireGuard (wireguard-go) when kernel lacks it

  • Client.CreateInterface now detects EOPNOTSUPP from netlink and builds a userspace WireGuard device on a TUN interface instead, with a UAPI socket at the standard per-interface location. DeleteInterface closes the retained userspace device rather than calling netlink. See client.go and wg_userspace.go.
  • CheckConnection retries its three-packet ICMP health check with a privileged socket when the unprivileged attempt fails with EACCES, via a new Client.ping helper.
  • Adds a root-only integration test in wg_userspace_test.go that runs a full connect, handshake, health-check, reconnect, and cleanup cycle against an in-memory WireGuard gateway; CI runs it under sudo (ci.yml).
  • Behavioral Change: on systems with kernel WireGuard, nothing changes. Userspace fallback requires /dev/net/tun and leaves a TUN interface plus UAPI socket in $RUNDIR/wireguard/<iface>.sock until DeleteInterface runs; the server side still uses kernel WireGuard.
📊 Macroscope summarized 66f15c9. 4 files reviewed, 1 issue evaluated, 1 issue filtered, 0 comments posted

🗂️ Filtered Issues

lib/wg_userspace.go — 0 comments posted, 1 evaluated, 1 filtered
  • line 36: newUserspaceDevice permanently exits its UAPI accept loop on any Accept error, but leaves the WireGuard device and Client running. In the pinned ipc.UAPIListen, deleting or replacing /var/run/wireguard/<ifname>.sock triggers an inotify error delivered by Accept; subsequent wgctrl.ConfigureDevice calls in Client.Connect then cannot access the device, so a live client cannot reconnect until it is restarted. Propagate the listener failure or recreate the socket rather than silently abandoning UAPI service. [ Failed validation ]
🐞

Summarization
• Workflow

Modal VM sandboxes run kernels without the WireGuard module, so creating
the client interface fails with EOPNOTSUPP before registration. On that
error the client now runs wireguard-go on a TUN interface of the same
name and serves the standard UAPI socket, so the existing wgctrl
configuration, health checks, reconnects, and /disconnect work
unchanged. Closing the device removes the interface and the socket.

The health check also falls back to a raw ICMP socket when unprivileged
ping sockets are forbidden by the kernel's default ping_group_range.

CI runs the lib tests as root too, covering an end-to-end test that
tunnels through a userspace client to an in-memory WireGuard gateway.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@macroscopeapp-nonprod

Copy link
Copy Markdown

Approvability

Verdict: Approved at 66f15c9

Macroscope's review found this PR approvable — This adds a per-client userspace WireGuard fallback for kernels without WireGuard support, plus coverage and documentation for that path. The change is bounded to local tunnel setup and does not introduce a shared production, billing, access, data durability, or external-action concern.

You can add or adjust custom eligibility rules. Learn more.

@maerlyn5
maerlyn5 merged commit 58e7636 into main Oct 11, 2026
9 checks passed
@maerlyn5
maerlyn5 deleted the userspace-wg branch October 11, 2026 22:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant