Skip to content

os-image: add the desktop flavor (GNOME + OSWorld guest server) - #152

Merged
CMGS merged 1 commit into
mainfrom
feat/desktop-flavor
Sep 11, 2026
Merged

CMGS merged 1 commit into
mainfrom
feat/desktop-flavor

Conversation

@CMGS

@CMGS CMGS commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

A desktop flavor for GUI agent work: an Ubuntu GNOME session on a virtual framebuffer with the OSWorld guest server, so a sandbox can serve screenshots, an AT-SPI tree and PyAutoGUI actions over the relay. It is what the OSWorld benchmark harness talks to, and it runs on the 2xlarge tier added in #150.

What it is

os-image/desktop/24.04 builds on base:24.04 and adds:

  • GNOME on Xvfb :1 (1920x1080, Mesa llvmpipe), started by gnome-session --session=ubuntu from a system unit as uid 1000 on the user@1000 bus. No GDM, no VT, no DRM — the microVM has no display device. AT-SPI works off that bus, which is what the accessibility tree needs.
  • osworld-server (xlang-ai/osworld-server, pinned) in /opt/osworld-server/.venv on loopback 5000, with --system-site-packages so the apt pyatspi binding is importable.
  • The OSWorld app set: Chrome, LibreOffice, GIMP, VLC, plus the scripting tools (xdotool, wmctrl, scrot, ffmpeg).
  • A CDP bridge 9222 → 1337: OSWorld task configs start Chrome with --remote-debugging-port=1337 and read CDP on 9222, which is what the official guest image does.
  • dconf defaults for the session: accessibility on, no lock, no idle, no animations.

Two details that are load-bearing and easy to lose: Chrome 136+ ignores --remote-debugging-port on the default profile directory, so the google-chrome wrapper supplies a fixed --user-data-dir (task configs pass only the port); and python-xlib insists ~/.Xauthority exists even though Xvfb -ac ignores it — without the empty file, screenshots fall back to scrot and cursor capture dies.

e2e/cmd/desktopsmoke is the acceptance driver: claim on none/2xlarge/screenshot/accessibility → a PyAutoGUI click echoed by /cursor_position → checkpoint and branch. docs/desktop.md documents the guest contract and the ports.

Evidence

Built and run on a bare-metal node (Xeon 8481C, 192 threads / 503 GB, KVM), with the image imported through the usual docker exportcocoon image import path:

  • image: docker 4.02 GB → EROFS 1.414 GiB, the same size the flavor produced on an unrelated build host
  • desktopsmoke: PASS — warm claim 0.0 s, PNG 30 KB, AT-SPI tree 142 KB, the click echoed at (300, 300), and a branch of the warmed desktop answers /screenshot without relaunching the session
  • pool of 24 warm 2xlarge desktops fills in 5 s and costs 41 GB of host memory (mmap restore shares the golden's memory file); cold boot to a ready desktop is ~4 s
  • the OSWorld harness driving this flavor through a sandbox provider scores 1.0 on task e0df059f and 0.0 after a revert, and 16 of those tasks in parallel are 16/16 at 1.0 (reset p50 3.47 s, step p50 3.46 s)
  • gates: make go-lint 10 × 0 issues., make go-test 18 packages ok with -race, asl ./... and GOOS=linux asl ./... clean

Notes

  • amd64 only (platforms): the flavor pulls google-chrome-stable, which has no arm64 build.
  • The image is ~4 GB unpacked, an order of magnitude above the other flavors. It is opt-in: nothing builds or pulls it unless a pool asks for desktop:24.04.

base:24.04 plus a GNOME session on Xvfb :1 at 1920x1080 (Mesa software
GL, no GDM/VT/DRM), xlang-ai/osworld-server at a pinned commit on guest
loopback 5000, a socat bridge 9222 -> 1337 for the Chrome CDP the OSWorld
task configs launch, and the OSWorld app set (Google Chrome, LibreOffice,
GIMP, VLC). The desktop user is uid 1000 like the OSWorld AMI; the
google-chrome wrapper adds --no-sandbox (no userns in-guest) and a fixed
--user-data-dir because Chrome 136+ refuses remote debugging on the
default profile. e2e/cmd/desktopsmoke is the acceptance: claim
none/2xlarge, /screenshot, /accessibility, a PyAutoGUI click echoed by
/cursor_position, checkpoint + branch.

Evidence (cocoon-test2, isolated sandboxd): desktopsmoke PASS in 10.9 s;
OSWorld-V2 DesktopEnv with a cocoon provider on task e0df059f scored 1.0,
reset 2.8 s, revert (release + warm claim) 3.5 s; cold vm run to desktop
ready 4.0 s; EROFS 1.41 GiB. Gates: make go-lint 10x "0 issues.", make
go-test 18 ok.
@CMGS
CMGS merged commit 9be34f5 into main Sep 11, 2026
2 checks passed
@CMGS
CMGS deleted the feat/desktop-flavor branch September 11, 2026 14:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant