diff --git a/docs/arch/windows-egress-wfp-investigation.md b/docs/arch/windows-egress-wfp-investigation.md new file mode 100644 index 0000000..b790e28 --- /dev/null +++ b/docs/arch/windows-egress-wfp-investigation.md @@ -0,0 +1,262 @@ +# Windows container egress: WFP IPFORWARD investigation (negative result) + +## TL;DR + +On Windows Server 2025 (build 26100) with a Hyper-V-isolated job container on an +HNS **NAT** network, **host-side Windows Filtering Platform (WFP) filters cannot +enforce container egress.** The container's forwarded + SNAT'd traffic does not +traverse the host WFP engine at any inspectable IPv4 layer — not +`FWPM_LAYER_IPFORWARD_V4`, not `FWPM_LAYER_INBOUND_IPPACKET_V4`, and not +`FWPM_LAYER_OUTBOUND_IPPACKET_V4`. WFP is provably functional on the host (it +blocks the host's own traffic), so this is not a "filter didn't apply" problem; +the packets are handled inside the Hyper-V virtual-switch / HNS datapath, which +is beside the host's `tcpip.sys` WFP hooks. + +This closes a fifth mechanism after four prior failures (see below). The +conclusion for the fleet is that **Windows container egress restriction requires +network-level isolation (a VLAN / managed switch), not a host-side software +filter.** This is the one platform where ephemerd cannot reach Linux parity +(nftables `FORWARD`) purely in the daemon. + +This document exists so nobody spends a sixth night rediscovering it. It records +what was tried, the exact evidence, and a reproducer. + +## Why WFP IPFORWARD looked like the answer + +ephemerd blocks container→RFC1918 on Linux in the `FORWARD` chain +(`pkg/networking/firewall_linux.go`): the host routes+NATs for the container, so +the FORWARD hook is where container-originated egress is catchable. The Windows +analogue of the Linux FORWARD chain is `FWPM_LAYER_IPFORWARD_V4`. Every prior +Windows attempt sat at a layer this forwarded/NAT'd path skips: + +| # | Mechanism | Why it failed on metal | +|---|-----------|------------------------| +| 1 | Host Windows Firewall, source-scoped `localip=10.88/16` (PR #136) | WinNAT rewrites the source before the host ALE layer; a source-scoped rule matches nothing. Also disqualified: a host-level rule would block the node's own RFC1918 (Prometheus scrapes from 192.168.10.45). | +| 2 | `New-NetFirewallHyperVRule` scoped by container `VMCreatorId` (PR #140) | No container VMCreator exists on this host, even with a live container. | +| 3 | HNS endpoint ACLs (`RuleType=Switch`) | Applied but **not enforced**: VFP is not managing the ports on this NAT switch (all `vfpctrl` ops error). | +| 4 | Route blackhole / in-container firewall | Defeated by proxy-ARP; `MpsSvc` cannot start inside the container. | + +The theory for attempt #5: put a **BLOCK filter** (no callout driver, no kernel +signing) at `FWPM_LAYER_IPFORWARD_V4`, scoped to the ephemerd NAT vSwitch +arrival interface, denying RFC1918 while leaving the container subnet and the +internet alone. Zero-config, removable, Linux-parity. + +## The stack under test + +- Node: `mfl-win-amd64-101`, Windows Server 2025 build 26100, Hyper-V-isolated + job container on HNS NAT network `ephemerd` (10.88.0.0/16, gw 10.88.0.1). +- NAT vSwitch host adapter: `vEthernet (ephemerd)`, **ifIndex 9**, + InterfaceGuid `{A99E8F02-DC01-4487-BDFA-564474AD6B91}`, NET_LUID + `0x6008003000000`. +- LAN adapter: `Ethernet`, **ifIndex 7**. The host reaches the management LAN + (e.g. 192.168.10.45) via ifIndex 7 with source **192.168.14.249** — i.e. the + container's egress is SNAT'd to the host's LAN address. + +Two facts about the datapath that already hint at the outcome: + +``` +PS> Get-NetNat + (no output — the HNS NAT network is NOT a NetNat object) + +PS> Get-NetIPInterface -AddressFamily IPv4 | ? ifIndex -in 7,9 + ifIndex InterfaceAlias Forwarding + ------- -------------- ---------- + 9 vEthernet (ephemerd) Disabled + 7 Ethernet Disabled +``` + +IP forwarding is **Disabled** on both interfaces, and there is no `NetNat` +object — the classic host IP-forwarding/WinNAT path is not what's moving these +packets. The Hyper-V virtual switch datapath is. + +## What `FWPM_LAYER_IPFORWARD_V4` actually exposes here + +The tailscale/wf binding enumerates the live layer's fields. On this host: + +``` +Layer IP Forward v4 Layer (IPFORWARD_V4) fields=15 + IP_SOURCE_ADDRESS netip.Addr + IP_DESTINATION_ADDRESS netip.Addr + IP_LOCAL_INTERFACE uint64 <- arrival interface at this layer + IP_FORWARD_INTERFACE uint64 <- outbound/nexthop interface + SOURCE_INTERFACE_INDEX uint32 + DESTINATION_INTERFACE_INDEX uint32 + IP_PHYSICAL_ARRIVAL_INTERFACE uint64 + IP_PHYSICAL_NEXTHOP_INTERFACE uint64 + ... (FLAGS, COMPARTMENT_ID, profile IDs) +``` + +Note there is **no `IP_ARRIVAL_INTERFACE`** field at this layer on this build — +the arrival interface is `IP_LOCAL_INTERFACE`, and the destination is +`IP_DESTINATION_ADDRESS` (not `IP_REMOTE_ADDRESS`). Both scoping conditions the +original plan named therefore had to be remapped before any test was valid. + +## Experiments and results + +Each experiment installs a persistent BLOCK filter (or a matrix of them) under +an ephemerd WFP provider/sublayer, then a real containment run +(`.github/workflows/containment.yml`, branch `test/containment-suite`) probes the +four fleet management planes from inside a Hyper-V-isolated job. To avoid cutting +the runner's own control channel, single-IP experiments target **Grafana +(192.168.10.45:3000)** only — blocking it does not disconnect the runner, so a +"Grafana blocked, other three reachable" result would have been the clean, +scoped success signal. + +The containment "Fleet management planes" step prints `FAIL: reached ` for +every plane it **could** reach. A working scoped block would drop the Grafana +line and keep the other three. + +| # | Layer | Conditions | Grafana result | +|---|-------|-----------|----------------| +| 1 | `IPFORWARD_V4` | `IP_DESTINATION_ADDRESS=192.168.10.45` **AND** `IP_LOCAL_INTERFACE=if9` | reached (not blocked) | +| 2 | `IPFORWARD_V4` | `IP_DESTINATION_ADDRESS=192.168.10.45` only | reached (not blocked) | +| 3 | `INBOUND_IPPACKET_V4` | `IP_LOCAL_ADDRESS=192.168.10.45` **AND** `IP_LOCAL_INTERFACE=if9` | reached (not blocked) | +| 4 | **matrix** (7 filters at once, all blocking Grafana): inbound local/remote × if9/any, ipforward dest × if9/any, **outbound remote (post-NAT, any iface)** | reached (not blocked) | + +Matrix run (id `31354660618`) installed all seven filters: + +``` + in-local-if installed (INBOUND_IPPACKET_V4, IP_LOCAL_ADDRESS + if9) + in-remote-if installed (INBOUND_IPPACKET_V4, IP_REMOTE_ADDRESS + if9) + in-local-any installed (INBOUND_IPPACKET_V4, IP_LOCAL_ADDRESS) + in-remote-any installed (INBOUND_IPPACKET_V4, IP_REMOTE_ADDRESS) + fwd-dest-if installed (IPFORWARD_V4, IP_DESTINATION_ADDRESS + if9) + fwd-dest-any installed (IPFORWARD_V4, IP_DESTINATION_ADDRESS) + out-remote-any installed (OUTBOUND_IPPACKET_V4, IP_REMOTE_ADDRESS) +ephemerd filters present: 7 +``` + +Containment Windows egress step, that same run: + +``` +FAIL: reached Proxmox coyotes at 192.168.5.1:8006 +FAIL: reached Proxmox kings at 192.168.5.2:8006 +FAIL: reached Incus daemon at 192.168.12.113:8443 +FAIL: reached Grafana at 192.168.10.45:3000 +``` + +All four reached — none of the seven filters caught the container. The +load-bearing one is **`out-remote-any`**: after SNAT the container's packet +*must* leave the host on ifIndex 7 with destination 192.168.10.45. A dest-only +block at `OUTBOUND_IPPACKET_V4` did not stop it. + +## Positive controls — WFP *is* enforced on this host + +To rule out "the filters are silently inert" (the failure mode of the HNS ACLs +in attempt #3), the identical layer + field + value was pointed at the **host's +own** traffic: + +``` +# ALE_AUTH_CONNECT_V4, block IP_REMOTE_ADDRESS=1.1.1.1 +host->1.1.1.1:443 reachable=False <- host lost 1.1.1.1 + +# OUTBOUND_IPPACKET_V4, block IP_REMOTE_ADDRESS=1.1.1.1 (same layer+field as out-remote-any) +host->1.1.1.1:443 reachable=False <- host lost 1.1.1.1 + +# after removing the ephemerd provider +host->1.1.1.1 restored=True +``` + +So `OUTBOUND_IPPACKET_V4` with `IP_REMOTE_ADDRESS` **does** block traffic that +enters the host WFP engine — it blocked the host. It just never sees the +container's SNAT'd packet. WFP is functional; the container's egress is simply +off the host's WFP-filtered path. + +## Conclusion + +On this WinNAT/HNS + Hyper-V-isolated stack: + +1. WFP filters are applied and enforced (host traffic is blockable at ALE and + OUTBOUND layers). +2. The container's forwarded + SNAT'd egress is **not classified** at any host + IPv4 WFP layer that is either (a) container-distinguishable (arrival interface + ifIndex 9, or pre-NAT source 10.88/16 at inbound) or (b) even visible + post-NAT by destination at outbound. +3. Therefore the NAT + forward happens inside the Hyper-V vSwitch / HNS + datapath, out-of-band of the host `tcpip.sys` WFP hooks. This is consistent + with `Get-NetNat` being empty, per-interface Forwarding being Disabled, and + the attempt-#3 finding that VFP is not managing the switch ports. + +**Host-side software (WFP included) cannot enforce Windows container egress on +this stack.** The remaining enforcement point is the network itself: put the +Windows runner on an isolated VLAN / managed switch whose uplink denies the +RFC1918 management ranges. That is the recommended and only proven fix for the +Windows platform. + +`pkg/networking/network_windows.go` keeps its per-endpoint HNS block ACLs as a +best-effort layer (they are cheap and correct where VFP *does* manage ports on +other Windows builds), but on this build they are inert and must not be relied +on — the VLAN is the control. + +## Reproducer + +A standalone Windows spike drove every experiment above via the pure-Go +`github.com/tailscale/wf` binding (no callout driver, no kernel signing). It was +intentionally **not** merged into the daemon or `go.mod` — there is no working +mechanism to integrate, and adding a WFP dependency for a negative result is +noise. The essential shape is small enough to reproduce inline: + +```go +//go:build windows +package main + +// go get github.com/tailscale/wf golang.org/x/sys/windows +// +// Resolve the NAT vSwitch interface LUID (arrival interface) and install a +// BLOCK filter at the chosen layer scoped to it + a destination. Change `layer`, +// the destination field, and the arrival field to reproduce each row above. + +import ( + "net/netip" + "unsafe" + + "github.com/tailscale/wf" + "golang.org/x/sys/windows" +) + +var ( + iphlp = windows.NewLazySystemDLL("iphlpapi.dll") + toLuid = iphlp.NewProc("ConvertInterfaceIndexToLuid") + provID = wf.ProviderID{Data1: 0x1d6d3d8e, Data2: 0x6c9a, Data3: 0x4e7b, Data4: [8]byte{0x9f, 0x2a, 0x3b, 0x7c, 0x1e, 0x5d, 0x9a, 0x40}} + subID = wf.SublayerID{Data1: 0x2e7e4e9f, Data2: 0x7dab, Data3: 0x4f8c, Data4: [8]byte{0xa0, 0x3b, 0x4c, 0x8d, 0x2f, 0x6e, 0xab, 0x51}} + ruleID = wf.RuleID{Data1: 0x3f8f5fa0, Data2: 0x8ebc, Data3: 0x40fd, Data4: [8]byte{0xb1, 0x4c, 0x5d, 0x9e, 0x30, 0x7f, 0xbc, 0x62}} +) + +func luid(ifindex uint32) uint64 { + var l uint64 + toLuid.Call(uintptr(ifindex), uintptr(unsafe.Pointer(&l))) + return l +} + +func main() { + s, _ := wf.New(&wf.Options{Name: "ephemerd-wfp-spike", Dynamic: false}) + defer s.Close() + s.AddProvider(&wf.Provider{ID: provID, Name: "ephemerd", Persistent: true}) + s.AddSublayer(&wf.Sublayer{ID: subID, Name: "ephemerd-egress", Provider: provID, Persistent: true, Weight: 0xffff}) + + // Row 1: IPFORWARD_V4, arrival = NAT vSwitch (ifIndex 9), dest = Grafana. + s.AddRule(&wf.Rule{ + ID: ruleID, Name: "ephemerd-spike-block", Layer: wf.LayerIPForwardV4, + Sublayer: subID, Provider: provID, Weight: 1000, Action: wf.ActionBlock, + Persistent: true, + Conditions: []*wf.Match{ + {Field: wf.FieldIPDestinationAddress, Op: wf.MatchTypeEqual, Value: netip.MustParsePrefix("192.168.10.45/32")}, + {Field: wf.FieldIPLocalInterface, Op: wf.MatchTypeEqual, Value: luid(9)}, + }, + }) + // Then dispatch the containment suite and read the Windows egress step. + // Positive control: swap Layer -> wf.LayerOutboundIPPacketV4, + // Field -> wf.FieldIPRemoteAddress, dest -> 1.1.1.1/32, drop the interface + // condition, and Test-NetConnection 1.1.1.1 from the host: it goes False, + // proving the engine enforces — the container just never reaches it. + // + // Cleanup: s.DeleteRule(ruleID); s.DeleteSublayer(subID); s.DeleteProvider(provID) +} +``` + +Field mapping that matters (this build): at `IPFORWARD_V4` the destination is +`IP_DESTINATION_ADDRESS` and the arrival interface is `IP_LOCAL_INTERFACE` +(there is no `IP_ARRIVAL_INTERFACE`); at `INBOUND_IPPACKET_V4` the destination of +a forwarded packet is `IP_LOCAL_ADDRESS` (REMOTE is the source) and the arrival +interface is again `IP_LOCAL_INTERFACE`; at `OUTBOUND_IPPACKET_V4` the +destination is `IP_REMOTE_ADDRESS`. diff --git a/pkg/networking/firewall_windows.go b/pkg/networking/firewall_windows.go index 01ea215..5a5f3db 100644 --- a/pkg/networking/firewall_windows.go +++ b/pkg/networking/firewall_windows.go @@ -44,6 +44,20 @@ import ( // Hyper-V firewall at all; it governs container-class workloads — WSL, Windows // Sandbox, and Windows containers). // +// Why NOT a host-side WFP filter at IPFORWARD_V4 (the Linux FORWARD analogue). +// Tempting, since firewall_linux.go enforces in FORWARD. It was built and driven +// on a live Server 2025 node (a pure-Go tailscale/wf spike, no callout driver) +// and does not work: the container's forwarded + SNAT'd egress is never +// classified at ANY host IPv4 WFP layer — not IPFORWARD_V4, not +// INBOUND_IPPACKET_V4 (arrival-interface scoped or not), not OUTBOUND_IPPACKET_V4 +// (dest-only, post-NAT). Positive controls prove WFP itself is enforced on the +// host (blocking the host's own 1.1.1.1 at ALE and OUTBOUND both worked), so the +// packets simply travel the Hyper-V vSwitch/HNS datapath, out-of-band of the +// host tcpip.sys WFP hooks (Get-NetNat is empty and per-interface Forwarding is +// Disabled). Full evidence: docs/arch/windows-egress-wfp-investigation.md. +// The definitive fix for this stack is network-level: an isolated VLAN whose +// uplink denies RFC1918, not a host-side software filter. +// // Gateway safety (identical reasoning to #136, and to firewall_linux.go). // The container subnet contains the NAT gateway (DNS, the default route, // module-proxy GatewayPorts) and the other containers. Blocking the gateway