diff --git a/README.md b/README.md index 0b50976..5a82fbc 100644 --- a/README.md +++ b/README.md @@ -367,4 +367,8 @@ sudo tailscale up --ssh --hostname=tahoe-vanilla Note that, without additional setup, this VM can only receive Tailscale SSH connections, and cannot SSH into other VMs on the tailnet. +## [Windows](windows) + +A Windows 11 guest that runs *inside* the `x86_64-linux` VM via [dockur/windows](https://github.com/dockur/windows) and nested KVM, reachable over the tailnet by RDP or by SSH into a PowerShell prompt (the latter mirroring the Linux nested-SSH hops). See [`windows/`](windows) for setup, access, and why the ARM equivalent doesn't work. + [flakes]: https://wiki.nixos.org/wiki/Flakes#Other_Distros,_without_Home-Manager diff --git a/bin/tailnet.py b/bin/tailnet.py index 059f1c0..3589703 100644 --- a/bin/tailnet.py +++ b/bin/tailnet.py @@ -18,7 +18,11 @@ def main(): suffix = json.loads(out)["MagicDNSSuffix"] path = Path.home() / ".ssh" / "tailnet" path.parent.mkdir(parents=True, exist_ok=True) - path.write_text(f"Host {' '.join(HOSTS)}\n HostName %h.{suffix}\n") + tailnet = f"Host {' '.join(HOSTS)}\n HostName %h.{suffix}\n" + # The `win-amd64` alias (see ssh/config) reaches the Windows guest via + # sandbox-amd64, so it needs that host's FQDN. + tailnet += f"\nHost win-amd64\n HostName sandbox-amd64.{suffix}\n" + path.write_text(tailnet) if __name__ == "__main__": diff --git a/ssh/config b/ssh/config index c0f093d..a0b4635 100644 --- a/ssh/config +++ b/ssh/config @@ -8,3 +8,12 @@ Host sandbox-arm64 Host tahoe-vanilla User admin + +# The Windows guest inside sandbox-amd64 (see windows/). RemoteCommand runs the +# inner ssh *on* sandbox-amd64 so it authenticates with that host's key (which is +# in the guest's authorized_keys); a plain ProxyJump would auth end-to-end and +# prompt for a password. `ssh win-amd64` drops straight into PowerShell. +Host win-amd64 + User agent-amd64 + RequestTTY force + RemoteCommand ssh -tt -o StrictHostKeyChecking=accept-new -p 2222 agent@localhost diff --git a/windows/.gitignore b/windows/.gitignore new file mode 100644 index 0000000..34ff768 --- /dev/null +++ b/windows/.gitignore @@ -0,0 +1 @@ +storage/ diff --git a/windows/README.md b/windows/README.md new file mode 100644 index 0000000..81eb569 --- /dev/null +++ b/windows/README.md @@ -0,0 +1,205 @@ +# Windows + +A Windows 11 guest that runs **inside** the `x86_64-linux` VM (`sandbox-amd64`), using +[dockur/windows](https://github.com/dockur/windows) — a container that runs Windows in +QEMU/KVM and fully automates the download + unattended install. Because it's nested +virtualization (Windows-in-KVM-in-the-Linux-VM), the Linux VM must expose `/dev/kvm`; +both of this repo's Linux VMs already do (the libvirt guest has nested KVM, and the Lima +guest is created with `--set '.nestedVirtualization = true'`). + +> [!NOTE] +> This works on the **x86_64** Linux VM. It does **not** work on the `aarch64` Linux VM — +> see [ARM64](#arm64-doesnt-work) at the bottom for why. + +## Setup + +The [`compose.yml`](compose.yml) here creates a Windows 11 Pro guest with a local admin +account `agent` / `password`, and publishes three ports: the web viewer (`8006`), RDP +(`3389`), and SSH (`2222`, once you set up OpenSSH — see below). The `storage/` directory +holds the VM disk and is git-ignored. + +```sh +mkdir -p ~/windows && cp compose.yml ~/windows/ && cd ~/windows +docker compose up -d +``` + +The container downloads the official Microsoft ISO and installs Windows completely +unattended (partition, skip the Microsoft-account/OOBE prompts, create the local admin, +enable RDP). It takes roughly 30–60 minutes; watch progress at + (tunnel it with `ssh -L 8006:localhost:8006 sandbox-amd64`). + +## Access + +Once installed, from the host machine over the tailnet: + +- **Web viewer** (works during install too): tunnel `8006` and open the browser. +- **RDP**: tunnel `3389` (`ssh -L 3389:localhost:3389 sandbox-amd64`) and connect with any + RDP client to `localhost:3389` as `agent` / `password`. +- **SSH → PowerShell**: once OpenSSH is set up (below), it behaves exactly like the Linux + nested-SSH hops: + + ```sh + ssh sandbox-amd64 # tailnet + ssh -p 2222 agent@localhost # into the Windows guest -> PowerShell + ``` + + or, as a single command, `ssh win-amd64` (alias in [`ssh/config`](../ssh/config); run + `tailnet` once so `~/.ssh/tailnet` picks up its hostname). The alias uses `RemoteCommand` + to run the inner `ssh` *on* `sandbox-amd64`, so it authenticates with that host's key. + Note that `ssh -J sandbox-amd64 -p 2222 agent@localhost` does **not** work passwordlessly: + `-J` authenticates end-to-end, so the guest would need your local machine's key installed, + not sandbox-amd64's. (Because of `RemoteCommand`, `ssh win-amd64` is interactive-only — it + can't also take an ad-hoc command or be used for `scp`.) + +## Enabling SSH → PowerShell + +dockur's Windows image doesn't ship an SSH server, so this is a manual one-time setup. +Run everything below in an **elevated PowerShell** inside the guest (open one via RDP, or +Win+X → "Terminal (Admin)"). + +Installing OpenSSH via the Windows "Feature on Demand" (`Add-WindowsCapability`) is +unreliable here — it pulls from Windows Update over the guest's slow NAT and often stalls. +Install it straight from the official release instead: + +```powershell +[Net.ServicePointManager]::SecurityProtocol = 'Tls12' +iwr 'https://github.com/PowerShell/Win32-OpenSSH/releases/latest/download/OpenSSH-Win64.zip' -OutFile C:\o.zip -UseBasicParsing +Expand-Archive C:\o.zip C:\ -Force +powershell -ExecutionPolicy Bypass -File C:\OpenSSH-Win64\install-sshd.ps1 +& C:\OpenSSH-Win64\ssh-keygen.exe -A # generate host keys +# host keys must not be world-readable or sshd refuses to start: +Get-ChildItem C:\ProgramData\ssh\ssh_host_*_key | ForEach-Object { + icacls $_.FullName /inheritance:r /grant SYSTEM:F /grant BUILTIN\Administrators:F +} +# make PowerShell the default shell for SSH sessions: +New-ItemProperty 'HKLM:\SOFTWARE\OpenSSH' -Name DefaultShell ` + -Value 'C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe' -PropertyType String -Force +``` + +### Running sshd (the tricky part) + +On this image the normal `sshd` **Windows service crashes** when run as `LocalSystem` +(`get_passwd: lookup_sid() failed: 1332` — an account/SID-mapping quirk of the minimal +dockur image). Running `sshd.exe` as the `agent` user works instead, so run it from a +**scheduled task at startup**, with an `S4U` principal so it runs whether or not anyone is +logged on: + +```powershell +Set-Service sshd -StartupType Disabled # don't let the broken service fight for :22 +$a = New-ScheduledTaskAction -Execute C:\OpenSSH-Win64\sshd.exe +$t = New-ScheduledTaskTrigger -AtStartup +$p = New-ScheduledTaskPrincipal -UserId agent -LogonType S4U -RunLevel Highest +$s = New-ScheduledTaskSettingsSet -ExecutionTimeLimit ([TimeSpan]::Zero) ` + -RestartCount 3 -RestartInterval (New-TimeSpan -Minutes 1) +Register-ScheduledTask sshd-boot -Action $a -Trigger $t -Principal $p -Settings $s -Force +``` + +> [!IMPORTANT] +> Use `-AtStartup` with `-LogonType S4U`, **not** `-AtLogOn` with the default `Interactive` +> logon type. An interactive task inherits a console, and since Windows Terminal is the +> default console host, `sshd` ends up parented to a `WindowsTerminal.exe` that then sits +> there rendering forever — about 8% of a host CPU core, for nothing. Worse, it couples the +> two: closing that Terminal window kills `sshd` and locks you out of the guest. With `S4U`, +> `sshd` runs in session 0 with no console and no Terminal, and it no longer needs anyone +> logged in — which is what makes [Idle CPU cost](#idle-cpu-cost) below possible. + +But a non-`SYSTEM` sshd authenticates fine and then **can't create the login session** +(the SSH channel closes right after "Authenticated") because a normal admin lacks the +privilege to create a primary token. Grant it to `agent` (takes effect next logon): + +```powershell +secedit /export /cfg C:\s.inf /areas USER_RIGHTS | Out-Null +(gc C:\s.inf) ` + -replace 'SeAssignPrimaryTokenPrivilege = ','SeAssignPrimaryTokenPrivilege = agent,' ` + -replace 'SeIncreaseQuotaPrivilege = ','SeIncreaseQuotaPrivilege = agent,' | Set-Content C:\s2.inf +secedit /configure /db C:\s.sdb /cfg C:\s2.inf /areas USER_RIGHTS +Restart-Computer -Force +``` + +After the reboot, `ssh -p 2222 agent@localhost` drops you into PowerShell. + +### Passwordless (key-based) login + +`agent` is an administrator, so Win32-OpenSSH reads authorized keys from +`C:\ProgramData\ssh\administrators_authorized_keys` (**not** `~/.ssh/authorized_keys`), +and that file must be owned/readable only by SYSTEM and Administrators: + +```powershell +# paste your public key as the content: +Set-Content C:\ProgramData\ssh\administrators_authorized_keys -Value 'ssh-ed25519 AAAA... you@host' -Encoding ascii +icacls C:\ProgramData\ssh\administrators_authorized_keys /inheritance:r /grant SYSTEM:F /grant BUILTIN\Administrators:F +``` + +With `sandbox-amd64`'s key installed this way, the whole chain +(you → `sandbox-amd64` via Tailscale SSH → guest via key) is passwordless. + +## Idle CPU cost + +This guest is meant to stay running continuously, so its **idle** cost matters. An idle +Windows guest is never free here: it's doubly nested (bare metal → Linux VM → Windows), so +every VM exit is handled by a KVM that is itself running in a VM, and there's no invariant +TSC to lean on — `/proc/cpuinfo` on the Linux VM has neither `constant_tsc` nor +`nonstop_tsc`, and QEMU warns `host doesn't support requested feature: ... invtsc` once per +vCPU at every boot. The vCPUs sit halted ~80% of the time and still cost real host cycles. + +Measured on `sandbox-amd64` (steady state, nobody connected, as a percentage of one core of +the host's Ryzen 9 9950X): + +| Configuration | Host CPU | +| --- | --- | +| 4 vCPUs, autologon + Terminal window left open | 71% | +| 4 vCPUs, `sshd` decoupled from Windows Terminal | 60% | +| **2 vCPUs, `sshd` via S4U task, no interactive logon** | **32%** | + +Three things get you from 71% to 32%: + +1. **`CPU_CORES: "2"`** in [`compose.yml`](compose.yml). Most of the idle cost is per-vCPU + ticking, so fewer vCPUs is the single biggest lever. (It doesn't halve cleanly — a + chunk of the work is fixed per-VM and just redistributes onto the remaining vCPUs.) +2. **The S4U `sshd` task** above, which eliminates the always-on `WindowsTerminal.exe`. +3. **Turning off autologon**, so no interactive session exists at all — no `explorer`, no + `dwm` compositing a real desktop, and no taskbar weather widget quietly running an + `msedgewebview2` process. This is only safe *because* of the S4U task; with the old + `-AtLogOn` task, no logon meant no `sshd`. + + ```powershell + Set-ItemProperty 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon' ` + -Name AutoAdminLogon -Value '0' -Type String + ``` + + You still get a desktop on demand — just log in as `agent` through RDP or the web viewer. + +Two things that are **not** worth chasing: Hyper-V enlightenments are already fully enabled +(`hv_passthrough` resolves correctly even nested — `hv-synic`, `hv-stimer`, +`hv-stimer-direct`, `hv-tlbflush`, `hv-ipi` et al. all come back `true` via `qom-get`), and +KVM halt-polling is negligible here (under 0.1% of a core). The residual ~32% is the +nesting itself; the only way to get to zero is `docker stop windows`. + +### Poking at a running guest without SSH + +If you break SSH (easy to do — see the warning above), you can still drive the guest from +the Linux VM through QEMU's monitor socket, which dockur leaves at `/dev/shm/monitor.sock` +inside the container. From the host side that's `/proc//root/dev/shm/monitor.sock`, +and it speaks HMP: `screendump /storage/shot.ppm` writes a screenshot into `storage/`, and +`sendkey` types for you (`sendkey meta_l-r` for Win+R, then `sendkey s`, `sendkey c`, …, +`sendkey ret`). That's enough to log in and re-run a scheduled task blind. + +## ARM64 doesn't work + +Running the equivalent (`dockurr/windows-arm`) inside the `aarch64` Linux VM +**hangs at the EDK2 → Windows-ARM bootloader handoff** ("Start boot option") and never +reaches the installer, under the doubly-nested Apple-Silicon → Lima-Ubuntu → KVM stack. +Ruled out exhaustively — all of these hang identically: + +- dockur's EDK2 firmware, and stock TianoCore/AAVMF firmware +- `gic-version=max` and `gic-version=3` +- USB vs. virtio-SCSI CD-ROM install media +- dockur, and a hand-rolled `qemu-system-aarch64` invocation + +KVM acceleration is confirmed active (not TCG), and it's not a reboot loop — the guest +simply wedges the moment the Windows-ARM UEFI bootloader starts executing. Nested *Linux* +guests and nested *x86* Windows both work fine on the same hardware, so this appears to be +a genuine nested-virtualization limitation specific to Windows-on-ARM at this nesting +depth. See [dockur/windows#1548](https://github.com/dockur/windows/issues/1548) for others +hitting the same wall. If you actually need ARM Windows, run it un-nested (directly on the +Mac host via UTM/Parallels) instead. diff --git a/windows/compose.yml b/windows/compose.yml new file mode 100644 index 0000000..ec750a0 --- /dev/null +++ b/windows/compose.yml @@ -0,0 +1,25 @@ +services: + windows: + image: dockurr/windows + container_name: windows + environment: + VERSION: "11" + RAM_SIZE: "8G" + CPU_CORES: "2" + DISK_SIZE: "64G" + USERNAME: "agent" + PASSWORD: "password" + devices: + - /dev/kvm + - /dev/net/tun + cap_add: + - NET_ADMIN + ports: + - 8006:8006 # web viewer (noVNC) + - 3389:3389/tcp # RDP + - 3389:3389/udp + - 2222:22 # SSH (OpenSSH Server in the guest; see README) + volumes: + - ./storage:/storage + restart: always + stop_grace_period: 2m