os-image: add the desktop flavor (GNOME + OSWorld guest server)

README.md

@@ -57,7 +57,7 @@ performance) — source in
   cocoon+guest) plus bare-metal acceptance drivers under `cmd/`: `demo`,
   `smoke`, `meshsmoke`, `crossnode`, `coldproof`, `egresssmoke`,
   `interceptsmoke`, `volumesmoke`, `lifecycle` (idle→hibernate→archive),
-  `androidsmoke`, `browsersmoke`, and the `pullbench`/`pushbench`/`rpcbench`
+  `androidsmoke`, `browsersmoke`, `desktopsmoke`, and the `pullbench`/`pushbench`/`rpcbench`
   perf drivers
 - `boot/kernel/` — kernel version pin (`VERSION` + matching tarball `SHA256`,
   bump both together) + config fragment (amd64: over `x86_64_defconfig` +
@@ -68,8 +68,8 @@ performance) — source in
   `/boot/vmlinuz-sandbox` + `/boot/initrd.img-sandbox`
 - `os-image/` — VM images consuming the boot artifact: `base` (layered,
   for builds), `rt` (base squashed to one layer — the default template in
-  examples), `python`, `python-rt`, `node`, `node-rt`, `browser`, and
-  `android`
+  examples), `python`, `python-rt`, `node`, `node-rt`, `browser`,
+  `desktop`, and `android`
 - `scripts/` — `boot-bench.sh` (boot phase timing), `bench.sh` (the published
   benchmark procedure), `sandboxd-e2e.sh` (bare-metal e2e, below), plus the
   `archive`/`egress`/`intercept` e2e drivers

docs/desktop.md

@@ -0,0 +1,71 @@
+# Desktop sandboxes
+
+The `desktop` flavor boots a GNOME session (Ubuntu session on Xvfb, 1920x1080)
+with the OSWorld guest server on guest loopback `5000` and the OSWorld app
+set (Google Chrome, LibreOffice, GIMP, VLC). A computer-use agent or the
+[OSWorld](https://github.com/xlang-ai/OSWorld-V2) harness claims it and
+drives the desktop through the same HTTP contract the OSWorld AWS and
+docker guests speak — screenshot, AT-SPI accessibility tree, PyAutoGUI
+actions — over the existing port relay, so the guest needs no NIC.
+
+```python
+sb = client.new("ghcr.io/cocoonstack/sandbox/desktop:24.04", size="2xlarge")
+ln = sb.proxy_port("127.0.0.1:0", 5000)
+# GET http://127.0.0.1:<port>/screenshot → PNG; /accessibility → AT-SPI XML;
+# POST /execute {"command": ["python", "-c", "import pyautogui; ..."]}
+```
+
+The claim returns when silkd answers; the session and the guest server come
+up a few seconds later — poll `GET /screenshot` until it returns 200.
+
+## Claim shape
+
+- **Lane**: `net=none` for local tasks (os, office, file work); `net=egress`
+  when tasks visit the OSWorld mocked websites or the real web.
+- **Size**: `2xlarge` (8 CPU / 16G) — the t3.xlarge class the OSWorld AWS
+  image runs on; the idle session is ~0.5 GB anonymous memory with
+  gnome-shell around 290 MB RSS, and the headroom is for the apps.
+- **Template**: `ghcr.io/cocoonstack/sandbox/desktop:24.04` — `base:24.04`
+  plus GNOME on Xvfb, `osworld-server` at a pinned commit, Chrome from
+  Google's apt repo. See [`os-image/desktop/README.md`](../os-image/desktop/README.md)
+  for the guest contract.
+
+## Guest ports
+
+| port | service |
+|---|---|
+| `5000` | osworld-server (`/screenshot`, `/accessibility`, `/execute`, `/setup/*`) |
+| `9222` | CDP bridge to the Chrome OSWorld task configs launch with `--remote-debugging-port=1337` |
+
+Both bind guest loopback; reach them with `DialPort`/`ProxyPort`.
+
+## Running the OSWorld harness on sandboxd
+
+OSWorld's `DesktopEnv` drives VMs through a `Provider` whose
+`get_ip_address` may return `localhost:<server>:<chromium>:<vnc>:<vlc>`
+with per-environment ports — the shape its docker provider uses. A cocoon
+provider claims one sandbox per environment, serves the four guest ports on
+loopback listeners with `proxy_port`, and implements `revert_to_snapshot` as
+release + fresh claim, so a warm pool is the snapshot revert:
+
+```
+DesktopEnv(provider_name="cocoon") ── localhost:<p5000> ── sandboxd ── desktop VM :5000
+```
+
+Configure it with `SANDBOXD_ADDR`, `SANDBOXD_TOKEN`, `COCOON_TEMPLATE`
+(default `desktop:24.04`), `COCOON_SIZE` (default `2xlarge`) and
+`COCOON_NET` (default `none`).
+
+## What works, what differs
+
+- Everything from the Ubuntu flavors (exec, files, sessions, git, pty) plus
+  the running desktop.
+- The desktop user is `user` (uid 1000, passwordless sudo), matching the
+  OSWorld AMI; task configs that pipe `CLIENT_PASSWORD` into `sudo -S`
+  work with any password.
+- Chrome runs `--no-sandbox` (the microVM is the isolation boundary) and a
+  fixed `--user-data-dir`, because Chrome 136+ refuses remote debugging on
+  the default profile.
+- No Thunderbird or VS Code yet (snap-only on 24.04 / vendor repo); tasks
+  targeting them are out of scope for this flavor version.
+- x86_64 only.

docs/index.md

@@ -40,6 +40,9 @@ vsock-only I/O (hardened default); `net=egress` attaches a bridge/CNI NIC.
 - [Browser sandboxes](browser.md) — headless Chromium with CDP through
   the relay: Playwright/Puppeteer access, checkpoint/branch of a live
   browser
+- [Desktop sandboxes](desktop.md) — the GNOME flavor with the OSWorld
+  guest server: claim shape, the guest control contract over the relay,
+  running the OSWorld harness on sandboxd
 - [Guarded egress](egress.md) — allow-listed, audited outbound access with
   host-side credential injection, on both lanes: no NIC (none) or an
   nftables-locked NIC (egress)

e2e/cmd/desktopsmoke/main.go

@@ -0,0 +1,180 @@
+// desktopsmoke is the desktop-flavor acceptance: claim (none/2xlarge) → the
+// OSWorld guest server over the relay (screenshot, AT-SPI tree, a PyAutoGUI
+// action echoed by the cursor) → checkpoint/branch of the warmed desktop.
+package main
+
+import (
+	"bufio"
+	"bytes"
+	"context"
+	"encoding/json"
+	"flag"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"strings"
+	"time"
+
+	"github.com/cocoonstack/sandbox/e2e/internal/harness"
+	sandbox "github.com/cocoonstack/sandbox/sdk/go"
+)
+
+const (
+	serverPort = 5000
+	// The GNOME session and osworld-server start after silkd; a cold first
+	// launch on a loaded node can take tens of seconds.
+	serverWait = 3 * time.Minute
+	clickX     = 300
+	clickY     = 300
+)
+
+var pngMagic = []byte("\x89PNG\r\n\x1a\n")
+
+func main() {
+	addr := flag.String("addr", "127.0.0.1:7777", "sandboxd address")
+	token := flag.String("token", "", "node api token")
+	template := flag.String("template", "ghcr.io/cocoonstack/sandbox/desktop:24.04", "desktop template ref")
+	flag.Parse()
+
+	if err := run(*addr, *token, *template); err != nil {
+		fmt.Fprintln(os.Stderr, "desktopsmoke:", err)
+		os.Exit(1)
+	}
+	fmt.Println("DESKTOPSMOKE PASS")
+}
+
+func run(addr, token, template string) error {
+	ctx, cancel := context.WithTimeout(context.Background(), 20*time.Minute)
+	defer cancel()
+	start := time.Now()
+	_, sb, err := harness.Claim(ctx, addr, token, template,
+		sandbox.WithNetwork(sandbox.NetNone), sandbox.WithSize(sandbox.XXLarge),
+		sandbox.WithTimeout(30*time.Minute))
+	if err != nil {
+		return err
+	}
+	defer func() { _ = sb.Close() }()
+	fmt.Printf("  claim: desktop 2xlarge up in %.1fs (silkd probed)\n", time.Since(start).Seconds())
+
+	if err = waitScreenshot(ctx, sb); err != nil {
+		return err
+	}
+	if err = accessibilityTree(ctx, sb); err != nil {
+		return err
+	}
+	if err = clickAndReadCursor(ctx, sb); err != nil {
+		return err
+	}
+
+	ckpt, err := sb.Checkpoint(ctx, "desktop-warmed")
+	if err != nil {
+		return fmt.Errorf("checkpoint: %w", err)
+	}
+	defer func() { _ = ckpt.Delete(ctx) }()
+	branch, err := ckpt.New(ctx)
+	if err != nil {
+		return fmt.Errorf("branch: %w", err)
+	}
+	defer func() { _ = branch.Close() }()
+	if err := waitScreenshot(ctx, branch); err != nil {
+		return fmt.Errorf("branch: %w", err)
+	}
+	fmt.Println("  checkpoint: branch of the warmed desktop answers /screenshot without relaunch")
+	return nil
+}
+
+func waitScreenshot(ctx context.Context, sb *sandbox.Sandbox) error {
+	deadline := time.Now().Add(serverWait)
+	for {
+		body, err := serverRequest(ctx, sb, "GET", "/screenshot", nil)
+		if err == nil {
+			if bytes.HasPrefix(body, pngMagic) {
+				fmt.Printf("  server: /screenshot → PNG, %d bytes\n", len(body))
+				return nil
+			}
+			err = fmt.Errorf("not a PNG: %q", body[:min(len(body), 16)])
+		}
+		if time.Now().After(deadline) {
+			return fmt.Errorf("osworld-server never answered: %w", err)
+		}
+		time.Sleep(2 * time.Second)
+	}
+}
+
+func accessibilityTree(ctx context.Context, sb *sandbox.Sandbox) error {
+	body, err := serverRequest(ctx, sb, "GET", "/accessibility", nil)
+	if err != nil {
+		return fmt.Errorf("accessibility: %w", err)
+	}
+	var reply struct {
+		AT string `json:"AT"`
+	}
+	if err = json.Unmarshal(body, &reply); err != nil {
+		return fmt.Errorf("accessibility: %w", err)
+	}
+	if !strings.Contains(reply.AT, "<desktop-frame") || !strings.Contains(reply.AT, "<application") {
+		return fmt.Errorf("accessibility: no AT-SPI desktop in %d bytes", len(reply.AT))
+	}
+	fmt.Printf("  server: /accessibility → AT-SPI tree, %d bytes\n", len(reply.AT))
+	return nil
+}
+
+// clickAndReadCursor proves /execute reaches the X display, not just the shell.
+func clickAndReadCursor(ctx context.Context, sb *sandbox.Sandbox) error {
+	action := fmt.Sprintf("import pyautogui; pyautogui.moveTo(%d, %d); pyautogui.click()", clickX, clickY)
+	req, _ := json.Marshal(map[string]any{"command": []string{"python", "-c", action}, "shell": false, "timeout": 15})
+	body, err := serverRequest(ctx, sb, "POST", "/execute", req)
+	if err != nil {
+		return fmt.Errorf("execute: %w", err)
+	}
+	var exec struct {
+		Status     string `json:"status"`
+		ReturnCode int    `json:"returncode"`
+		Error      string `json:"error"`
+	}
+	if err = json.Unmarshal(body, &exec); err != nil {
+		return fmt.Errorf("execute: %w", err)
+	}
+	if exec.Status != "success" || exec.ReturnCode != 0 {
+		return fmt.Errorf("execute: %s rc=%d: %s", exec.Status, exec.ReturnCode, strings.TrimSpace(exec.Error))
+	}
+	body, err = serverRequest(ctx, sb, "GET", "/cursor_position", nil)
+	if err != nil {
+		return fmt.Errorf("cursor_position: %w", err)
+	}
+	var pos [2]int
+	if err = json.Unmarshal(body, &pos); err != nil {
+		return fmt.Errorf("cursor_position: %w", err)
+	}
+	if pos != [2]int{clickX, clickY} {
+		return fmt.Errorf("cursor_position: got %v, want [%d %d]", pos, clickX, clickY)
+	}
+	fmt.Printf("  server: /execute pyautogui click → /cursor_position %v\n", pos)
+	return nil
+}
+
+// serverRequest hand-rolls one HTTP request to osworld-server over the port relay.
+func serverRequest(ctx context.Context, sb *sandbox.Sandbox, method, path string, body []byte) ([]byte, error) {
+	pc, err := sb.DialPort(ctx, serverPort)
+	if err != nil {
+		return nil, err
+	}
+	defer func() { _ = pc.Close() }()
+	if _, err = fmt.Fprintf(pc, "%s %s HTTP/1.1\r\nHost: localhost\r\nConnection: close\r\nContent-Type: application/json\r\nContent-Length: %d\r\n\r\n%s", method, path, len(body), body); err != nil {
+		return nil, err
+	}
+	resp, err := http.ReadResponse(bufio.NewReader(pc), nil)
+	if err != nil {
+		return nil, err
+	}
+	defer func() { _ = resp.Body.Close() }()
+	out, err := io.ReadAll(resp.Body)
+	if err != nil {
+		return nil, err
+	}
+	if resp.StatusCode != http.StatusOK {
+		return nil, fmt.Errorf("%s %s: %s: %s", method, path, resp.Status, strings.TrimSpace(string(out)))
+	}
+	return out, nil
+}