From 72d2450b913958f1f18d140f310a7e17226d6e93 Mon Sep 17 00:00:00 2001 From: CMGS Date: Thu, 10 Sep 2026 21:22:42 +0900 Subject: [PATCH] os-image: add the desktop flavor (GNOME + OSWorld guest server) base:24.04 plus a GNOME session on Xvfb :1 at 1920x1080 (Mesa software GL, no GDM/VT/DRM), xlang-ai/osworld-server at a pinned commit on guest loopback 5000, a socat bridge 9222 -> 1337 for the Chrome CDP the OSWorld task configs launch, and the OSWorld app set (Google Chrome, LibreOffice, GIMP, VLC). The desktop user is uid 1000 like the OSWorld AMI; the google-chrome wrapper adds --no-sandbox (no userns in-guest) and a fixed --user-data-dir because Chrome 136+ refuses remote debugging on the default profile. e2e/cmd/desktopsmoke is the acceptance: claim none/2xlarge, /screenshot, /accessibility, a PyAutoGUI click echoed by /cursor_position, checkpoint + branch. Evidence (cocoon-test2, isolated sandboxd): desktopsmoke PASS in 10.9 s; OSWorld-V2 DesktopEnv with a cocoon provider on task e0df059f scored 1.0, reset 2.8 s, revert (release + warm claim) 3.5 s; cold vm run to desktop ready 4.0 s; EROFS 1.41 GiB. Gates: make go-lint 10x "0 issues.", make go-test 18 ok. --- README.md | 6 +- docs/desktop.md | 71 ++++++++++++ docs/index.md | 3 + e2e/cmd/desktopsmoke/main.go | 180 ++++++++++++++++++++++++++++++ os-image/desktop/24.04/Dockerfile | 173 ++++++++++++++++++++++++++++ os-image/desktop/README.md | 41 +++++++ os-image/desktop/platforms | 1 + 7 files changed, 472 insertions(+), 3 deletions(-) create mode 100644 docs/desktop.md create mode 100644 e2e/cmd/desktopsmoke/main.go create mode 100644 os-image/desktop/24.04/Dockerfile create mode 100644 os-image/desktop/README.md create mode 100644 os-image/desktop/platforms diff --git a/README.md b/README.md index d0a3da02..025c2cb1 100644 --- a/README.md +++ b/README.md @@ -57,7 +57,7 @@ performance) — source in cocoon+guest) plus bare-metal acceptance drivers under `cmd/`: `demo`, `smoke`, `meshsmoke`, `crossnode`, `coldproof`, `egresssmoke`, `interceptsmoke`, `volumesmoke`, `lifecycle` (idle→hibernate→archive), - `androidsmoke`, `browsersmoke`, and the `pullbench`/`pushbench`/`rpcbench` + `androidsmoke`, `browsersmoke`, `desktopsmoke`, and the `pullbench`/`pushbench`/`rpcbench` perf drivers - `boot/kernel/` — kernel version pin (`VERSION` + matching tarball `SHA256`, bump both together) + config fragment (amd64: over `x86_64_defconfig` + @@ -68,8 +68,8 @@ performance) — source in `/boot/vmlinuz-sandbox` + `/boot/initrd.img-sandbox` - `os-image/` — VM images consuming the boot artifact: `base` (layered, for builds), `rt` (base squashed to one layer — the default template in - examples), `python`, `python-rt`, `node`, `node-rt`, `browser`, and - `android` + examples), `python`, `python-rt`, `node`, `node-rt`, `browser`, + `desktop`, and `android` - `scripts/` — `boot-bench.sh` (boot phase timing), `bench.sh` (the published benchmark procedure), `sandboxd-e2e.sh` (bare-metal e2e, below), plus the `archive`/`egress`/`intercept` e2e drivers diff --git a/docs/desktop.md b/docs/desktop.md new file mode 100644 index 00000000..86e926ae --- /dev/null +++ b/docs/desktop.md @@ -0,0 +1,71 @@ +# Desktop sandboxes + +The `desktop` flavor boots a GNOME session (Ubuntu session on Xvfb, 1920x1080) +with the OSWorld guest server on guest loopback `5000` and the OSWorld app +set (Google Chrome, LibreOffice, GIMP, VLC). A computer-use agent or the +[OSWorld](https://github.com/xlang-ai/OSWorld-V2) harness claims it and +drives the desktop through the same HTTP contract the OSWorld AWS and +docker guests speak — screenshot, AT-SPI accessibility tree, PyAutoGUI +actions — over the existing port relay, so the guest needs no NIC. + +```python +sb = client.new("ghcr.io/cocoonstack/sandbox/desktop:24.04", size="2xlarge") +ln = sb.proxy_port("127.0.0.1:0", 5000) +# GET http://127.0.0.1:/screenshot → PNG; /accessibility → AT-SPI XML; +# POST /execute {"command": ["python", "-c", "import pyautogui; ..."]} +``` + +The claim returns when silkd answers; the session and the guest server come +up a few seconds later — poll `GET /screenshot` until it returns 200. + +## Claim shape + +- **Lane**: `net=none` for local tasks (os, office, file work); `net=egress` + when tasks visit the OSWorld mocked websites or the real web. +- **Size**: `2xlarge` (8 CPU / 16G) — the t3.xlarge class the OSWorld AWS + image runs on; the idle session is ~0.5 GB anonymous memory with + gnome-shell around 290 MB RSS, and the headroom is for the apps. +- **Template**: `ghcr.io/cocoonstack/sandbox/desktop:24.04` — `base:24.04` + plus GNOME on Xvfb, `osworld-server` at a pinned commit, Chrome from + Google's apt repo. See [`os-image/desktop/README.md`](../os-image/desktop/README.md) + for the guest contract. + +## Guest ports + +| port | service | +|---|---| +| `5000` | osworld-server (`/screenshot`, `/accessibility`, `/execute`, `/setup/*`) | +| `9222` | CDP bridge to the Chrome OSWorld task configs launch with `--remote-debugging-port=1337` | + +Both bind guest loopback; reach them with `DialPort`/`ProxyPort`. + +## Running the OSWorld harness on sandboxd + +OSWorld's `DesktopEnv` drives VMs through a `Provider` whose +`get_ip_address` may return `localhost::::` +with per-environment ports — the shape its docker provider uses. A cocoon +provider claims one sandbox per environment, serves the four guest ports on +loopback listeners with `proxy_port`, and implements `revert_to_snapshot` as +release + fresh claim, so a warm pool is the snapshot revert: + +``` +DesktopEnv(provider_name="cocoon") ── localhost: ── sandboxd ── desktop VM :5000 +``` + +Configure it with `SANDBOXD_ADDR`, `SANDBOXD_TOKEN`, `COCOON_TEMPLATE` +(default `desktop:24.04`), `COCOON_SIZE` (default `2xlarge`) and +`COCOON_NET` (default `none`). + +## What works, what differs + +- Everything from the Ubuntu flavors (exec, files, sessions, git, pty) plus + the running desktop. +- The desktop user is `user` (uid 1000, passwordless sudo), matching the + OSWorld AMI; task configs that pipe `CLIENT_PASSWORD` into `sudo -S` + work with any password. +- Chrome runs `--no-sandbox` (the microVM is the isolation boundary) and a + fixed `--user-data-dir`, because Chrome 136+ refuses remote debugging on + the default profile. +- No Thunderbird or VS Code yet (snap-only on 24.04 / vendor repo); tasks + targeting them are out of scope for this flavor version. +- x86_64 only. diff --git a/docs/index.md b/docs/index.md index 50d20339..f934122e 100644 --- a/docs/index.md +++ b/docs/index.md @@ -40,6 +40,9 @@ vsock-only I/O (hardened default); `net=egress` attaches a bridge/CNI NIC. - [Browser sandboxes](browser.md) — headless Chromium with CDP through the relay: Playwright/Puppeteer access, checkpoint/branch of a live browser +- [Desktop sandboxes](desktop.md) — the GNOME flavor with the OSWorld + guest server: claim shape, the guest control contract over the relay, + running the OSWorld harness on sandboxd - [Guarded egress](egress.md) — allow-listed, audited outbound access with host-side credential injection, on both lanes: no NIC (none) or an nftables-locked NIC (egress) diff --git a/e2e/cmd/desktopsmoke/main.go b/e2e/cmd/desktopsmoke/main.go new file mode 100644 index 00000000..edce6bb4 --- /dev/null +++ b/e2e/cmd/desktopsmoke/main.go @@ -0,0 +1,180 @@ +// desktopsmoke is the desktop-flavor acceptance: claim (none/2xlarge) → the +// OSWorld guest server over the relay (screenshot, AT-SPI tree, a PyAutoGUI +// action echoed by the cursor) → checkpoint/branch of the warmed desktop. +package main + +import ( + "bufio" + "bytes" + "context" + "encoding/json" + "flag" + "fmt" + "io" + "net/http" + "os" + "strings" + "time" + + "github.com/cocoonstack/sandbox/e2e/internal/harness" + sandbox "github.com/cocoonstack/sandbox/sdk/go" +) + +const ( + serverPort = 5000 + // The GNOME session and osworld-server start after silkd; a cold first + // launch on a loaded node can take tens of seconds. + serverWait = 3 * time.Minute + clickX = 300 + clickY = 300 +) + +var pngMagic = []byte("\x89PNG\r\n\x1a\n") + +func main() { + addr := flag.String("addr", "127.0.0.1:7777", "sandboxd address") + token := flag.String("token", "", "node api token") + template := flag.String("template", "ghcr.io/cocoonstack/sandbox/desktop:24.04", "desktop template ref") + flag.Parse() + + if err := run(*addr, *token, *template); err != nil { + fmt.Fprintln(os.Stderr, "desktopsmoke:", err) + os.Exit(1) + } + fmt.Println("DESKTOPSMOKE PASS") +} + +func run(addr, token, template string) error { + ctx, cancel := context.WithTimeout(context.Background(), 20*time.Minute) + defer cancel() + start := time.Now() + _, sb, err := harness.Claim(ctx, addr, token, template, + sandbox.WithNetwork(sandbox.NetNone), sandbox.WithSize(sandbox.XXLarge), + sandbox.WithTimeout(30*time.Minute)) + if err != nil { + return err + } + defer func() { _ = sb.Close() }() + fmt.Printf(" claim: desktop 2xlarge up in %.1fs (silkd probed)\n", time.Since(start).Seconds()) + + if err = waitScreenshot(ctx, sb); err != nil { + return err + } + if err = accessibilityTree(ctx, sb); err != nil { + return err + } + if err = clickAndReadCursor(ctx, sb); err != nil { + return err + } + + ckpt, err := sb.Checkpoint(ctx, "desktop-warmed") + if err != nil { + return fmt.Errorf("checkpoint: %w", err) + } + defer func() { _ = ckpt.Delete(ctx) }() + branch, err := ckpt.New(ctx) + if err != nil { + return fmt.Errorf("branch: %w", err) + } + defer func() { _ = branch.Close() }() + if err := waitScreenshot(ctx, branch); err != nil { + return fmt.Errorf("branch: %w", err) + } + fmt.Println(" checkpoint: branch of the warmed desktop answers /screenshot without relaunch") + return nil +} + +func waitScreenshot(ctx context.Context, sb *sandbox.Sandbox) error { + deadline := time.Now().Add(serverWait) + for { + body, err := serverRequest(ctx, sb, "GET", "/screenshot", nil) + if err == nil { + if bytes.HasPrefix(body, pngMagic) { + fmt.Printf(" server: /screenshot → PNG, %d bytes\n", len(body)) + return nil + } + err = fmt.Errorf("not a PNG: %q", body[:min(len(body), 16)]) + } + if time.Now().After(deadline) { + return fmt.Errorf("osworld-server never answered: %w", err) + } + time.Sleep(2 * time.Second) + } +} + +func accessibilityTree(ctx context.Context, sb *sandbox.Sandbox) error { + body, err := serverRequest(ctx, sb, "GET", "/accessibility", nil) + if err != nil { + return fmt.Errorf("accessibility: %w", err) + } + var reply struct { + AT string `json:"AT"` + } + if err = json.Unmarshal(body, &reply); err != nil { + return fmt.Errorf("accessibility: %w", err) + } + if !strings.Contains(reply.AT, " /etc/apt/sources.list.d/google-chrome.list \ + && apt-get update && apt-get install -y --no-install-recommends google-chrome-stable \ + && rm -rf /var/lib/apt/lists/* \ + && locale-gen en_US.UTF-8 && update-locale LANG=en_US.UTF-8 \ + # Chrome userns sandboxing cannot function in-guest; the microVM is the + # isolation boundary. OSWorld launches "google-chrome" by name, so the + # wrapper shadows the packaged launcher on PATH and in the .desktop entry. + # Chrome 136+ ignores --remote-debugging-port on the default profile dir, + # and OSWorld task configs pass that flag without a --user-data-dir. + && printf '%s\n' \ + '#!/bin/sh' \ + 'exec /opt/google/chrome/google-chrome --user-data-dir=/home/user/.config/google-chrome-osworld --no-sandbox --disable-dev-shm-usage --disable-gpu --test-type "$@"' \ + > /usr/local/bin/google-chrome \ + && chmod 0755 /usr/local/bin/google-chrome \ + && sed -i 's|^Exec=/usr/bin/google-chrome-stable|Exec=/usr/local/bin/google-chrome|' /usr/share/applications/google-chrome.desktop \ + # osworld-server: pinned checkout, venv on system site-packages so the + # apt pyatspi binding is importable; PyAutoGUI/python3-xlib pins mirror + # its requirements.txt. + && git clone -q https://github.com/xlang-ai/osworld-server.git /opt/osworld-server \ + && git -C /opt/osworld-server checkout -q "${OSWORLD_SERVER_REF}" \ + && rm -rf /opt/osworld-server/.git \ + && python3 -m venv --system-site-packages /opt/osworld-server/.venv \ + && /opt/osworld-server/.venv/bin/pip install --no-cache-dir \ + "python3-xlib==0.15" "PyAutoGUI==0.9.54" Pillow flask fastapi uvicorn websockets numpy lxml requests \ + # The desktop user is uid 1000 like the OSWorld AMI (units below assume + # /run/user/1000); the stock ubuntu user gives up the uid. + && if id ubuntu >/dev/null 2>&1; then userdel -r ubuntu; fi \ + && useradd -m -u 1000 -s /bin/bash -G sudo,audio,video user \ + && echo 'user:password' | chpasswd \ + && echo 'user ALL=(ALL) NOPASSWD:ALL' > /etc/sudoers.d/user \ + && chmod 0440 /etc/sudoers.d/user \ + && install -d -o user -g user /home/user/Desktop /home/user/Documents /home/user/Downloads \ + # python-xlib insists the XAUTHORITY file exists even though Xvfb -ac ignores it. + && install -o user -g user -m 0600 /dev/null /home/user/.Xauthority \ + && chown -R user:user /opt/osworld-server \ + && install -d /var/lib/systemd/linger && touch /var/lib/systemd/linger/user \ + # Session defaults: a11y on (AT-SPI tree), no lock/idle/animations. + && install -d /etc/dconf/profile /etc/dconf/db/local.d \ + && printf 'user-db:user\nsystem-db:local\n' > /etc/dconf/profile/user \ + && printf '%s\n' \ + '[org/gnome/desktop/interface]' \ + 'toolkit-accessibility=true' \ + 'enable-animations=false' \ + '[org/gnome/desktop/session]' \ + 'idle-delay=uint32 0' \ + '[org/gnome/desktop/screensaver]' \ + 'lock-enabled=false' \ + 'idle-activation-enabled=false' \ + '[org/gnome/settings-daemon/plugins/power]' \ + 'sleep-inactive-ac-type='"'"'nothing'"'"'' \ + '[org/gnome/shell]' \ + 'welcome-dialog-last-shown-version='"'"'999'"'"'' \ + > /etc/dconf/db/local.d/00-sandbox \ + && dconf update \ + && printf '%s\n' \ + '[Unit]' \ + 'Description=virtual X display for the desktop session' \ + 'After=silkd.service' \ + '' \ + '[Service]' \ + 'Type=simple' \ + "ExecStart=/usr/bin/Xvfb :1 -screen 0 ${SCREEN}x24 -ac -noreset -nolisten tcp +extension GLX +extension RANDR +render" \ + 'Restart=always' \ + 'RestartSec=1s' \ + '' \ + '[Install]' \ + 'WantedBy=multi-user.target' \ + > /etc/systemd/system/xvfb.service \ + && printf '%s\n' \ + '[Unit]' \ + 'Description=GNOME session on the virtual display' \ + 'Requires=xvfb.service user@1000.service' \ + 'After=xvfb.service user@1000.service' \ + '' \ + '[Service]' \ + 'Type=simple' \ + 'User=user' \ + 'WorkingDirectory=/home/user' \ + 'Environment=DISPLAY=:1' \ + 'Environment=HOME=/home/user' \ + 'Environment=XDG_RUNTIME_DIR=/run/user/1000' \ + 'Environment=DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus' \ + 'Environment=XDG_SESSION_TYPE=x11' \ + 'Environment=XDG_SESSION_CLASS=user' \ + 'Environment=XDG_CURRENT_DESKTOP=ubuntu:GNOME' \ + 'Environment=GNOME_SHELL_SESSION_MODE=ubuntu' \ + 'Environment=LIBGL_ALWAYS_SOFTWARE=1' \ + 'Environment=GTK_MODULES=gail:atk-bridge' \ + 'Environment=QT_ACCESSIBILITY=1' \ + 'Environment=NO_AT_BRIDGE=0' \ + 'ExecStartPre=/usr/bin/systemctl --user import-environment DISPLAY XDG_SESSION_TYPE XDG_CURRENT_DESKTOP GNOME_SHELL_SESSION_MODE LIBGL_ALWAYS_SOFTWARE GTK_MODULES QT_ACCESSIBILITY NO_AT_BRIDGE' \ + 'ExecStart=/usr/bin/gnome-session --session=ubuntu' \ + 'Restart=always' \ + 'RestartSec=2s' \ + '' \ + '[Install]' \ + 'WantedBy=multi-user.target' \ + > /etc/systemd/system/desktop-session.service \ + && printf '%s\n' \ + '[Unit]' \ + 'Description=OSWorld guest server on loopback 5000' \ + 'After=desktop-session.service' \ + 'Wants=desktop-session.service' \ + '' \ + '[Service]' \ + 'Type=simple' \ + 'User=user' \ + 'WorkingDirectory=/opt/osworld-server' \ + 'Environment=DISPLAY=:1' \ + 'Environment=HOME=/home/user' \ + 'Environment=XDG_RUNTIME_DIR=/run/user/1000' \ + 'Environment=DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus' \ + 'Environment=PATH=/opt/osworld-server/.venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' \ + 'Environment=OSWORLD_SERVER_HOST=127.0.0.1' \ + 'Environment=OSWORLD_SERVER_PORT=5000' \ + 'ExecStart=/opt/osworld-server/.venv/bin/python /opt/osworld-server/main.py' \ + 'Restart=always' \ + 'RestartSec=2s' \ + '' \ + '[Install]' \ + 'WantedBy=multi-user.target' \ + > /etc/systemd/system/osworld-server.service \ + # OSWorld task configs start Chrome with --remote-debugging-port=1337 and + # reach CDP on 9222; the bridge is what the official guest image runs. + && printf '%s\n' \ + '[Unit]' \ + 'Description=OSWorld Chrome CDP bridge 9222 -> 1337' \ + 'After=xvfb.service' \ + '' \ + '[Service]' \ + 'Type=simple' \ + 'ExecStart=/usr/bin/socat TCP-LISTEN:9222,bind=127.0.0.1,fork,reuseaddr TCP:127.0.0.1:1337' \ + 'Restart=always' \ + 'RestartSec=2s' \ + '' \ + '[Install]' \ + 'WantedBy=multi-user.target' \ + > /etc/systemd/system/cdp-bridge.service \ + && systemctl enable xvfb.service desktop-session.service osworld-server.service cdp-bridge.service diff --git a/os-image/desktop/README.md b/os-image/desktop/README.md new file mode 100644 index 00000000..4dfe522f --- /dev/null +++ b/os-image/desktop/README.md @@ -0,0 +1,41 @@ +# desktop flavor + +GNOME desktop guest for OSWorld-style computer-use sandboxes: `base:24.04` +plus a GNOME session (Ubuntu session, dock, Yaru) on an Xvfb `:1` display at +1920x1080 with Mesa software GL, the OSWorld guest server +([xlang-ai/osworld-server](https://github.com/xlang-ai/osworld-server), +pinned commit) on guest loopback `5000`, the OSWorld Chrome CDP bridge on +loopback `9222`, and the OSWorld app set: Google Chrome, LibreOffice, GIMP, +VLC. The intended pool shape is `size: 2xlarge` (8 CPU / 16G); the session +idles at ~0.5 GB anonymous memory with gnome-shell around 290 MB RSS. + +An OSWorld harness or the Sai driver claims a sandbox and reaches the guest +server through the silkd port relay (`DialPort`/`ProxyPort` 5000, 9222), so +the `none` lane works for local tasks; tasks that visit the OSWorld mocked +websites need the `egress` lane. + +## Guest contract + +- `user` (uid 1000, password `password`, passwordless sudo) owns the + session; `/run/user/1000/bus` is the session bus the AT-SPI tree hangs off. +- `/opt/osworld-server/.venv/bin/python` has PyAutoGUI + python-xlib; the + server's `PATH` puts it first so task configs that run `python -c "import + pyautogui; ..."` resolve to it. +- `google-chrome` is a wrapper adding `--no-sandbox` (no userns in-guest) and + a fixed `--user-data-dir`: Chrome 136+ ignores `--remote-debugging-port` on + the default profile, and OSWorld task configs pass only the port. The + `cdp-bridge` unit forwards loopback `9222` to Chrome's `1337`. +- dconf defaults turn on `toolkit-accessibility`, and turn off animations, + idle, screen lock, and the welcome dialog. + +## Build + +- `24.04/Dockerfile` — `FROM base:24.04`; apt the GNOME/X/AT-SPI/app set, + Google Chrome from Google's apt repo, `osworld-server` at + `OSWORLD_SERVER_REF` into a system-site-packages venv; bake the `xvfb`, + `desktop-session`, `osworld-server` and `cdp-bridge` units. +- `platforms` — `linux/amd64`; Chrome is amd64-only. + +Services are product services, not readiness gates: the claim returns on +silkd; the session and `osworld-server` come up a few seconds later and are +polled by the workload (`GET /screenshot` returning 200). diff --git a/os-image/desktop/platforms b/os-image/desktop/platforms new file mode 100644 index 00000000..303dc7a5 --- /dev/null +++ b/os-image/desktop/platforms @@ -0,0 +1 @@ +linux/amd64