os-image: add the desktop flavor (GNOME + OSWorld guest server) - #152
Merged
Merged
Conversation
base:24.04 plus a GNOME session on Xvfb :1 at 1920x1080 (Mesa software GL, no GDM/VT/DRM), xlang-ai/osworld-server at a pinned commit on guest loopback 5000, a socat bridge 9222 -> 1337 for the Chrome CDP the OSWorld task configs launch, and the OSWorld app set (Google Chrome, LibreOffice, GIMP, VLC). The desktop user is uid 1000 like the OSWorld AMI; the google-chrome wrapper adds --no-sandbox (no userns in-guest) and a fixed --user-data-dir because Chrome 136+ refuses remote debugging on the default profile. e2e/cmd/desktopsmoke is the acceptance: claim none/2xlarge, /screenshot, /accessibility, a PyAutoGUI click echoed by /cursor_position, checkpoint + branch. Evidence (cocoon-test2, isolated sandboxd): desktopsmoke PASS in 10.9 s; OSWorld-V2 DesktopEnv with a cocoon provider on task e0df059f scored 1.0, reset 2.8 s, revert (release + warm claim) 3.5 s; cold vm run to desktop ready 4.0 s; EROFS 1.41 GiB. Gates: make go-lint 10x "0 issues.", make go-test 18 ok.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A desktop flavor for GUI agent work: an Ubuntu GNOME session on a virtual framebuffer with the OSWorld guest server, so a sandbox can serve screenshots, an AT-SPI tree and PyAutoGUI actions over the relay. It is what the OSWorld benchmark harness talks to, and it runs on the
2xlargetier added in #150.What it is
os-image/desktop/24.04builds onbase:24.04and adds::1(1920x1080, Mesa llvmpipe), started bygnome-session --session=ubuntufrom a system unit as uid 1000 on theuser@1000bus. No GDM, no VT, no DRM — the microVM has no display device. AT-SPI works off that bus, which is what the accessibility tree needs./opt/osworld-server/.venvon loopback 5000, with--system-site-packagesso the aptpyatspibinding is importable.--remote-debugging-port=1337and read CDP on 9222, which is what the official guest image does.Two details that are load-bearing and easy to lose: Chrome 136+ ignores
--remote-debugging-porton the default profile directory, so thegoogle-chromewrapper supplies a fixed--user-data-dir(task configs pass only the port); andpython-xlibinsists~/.Xauthorityexists even thoughXvfb -acignores it — without the empty file, screenshots fall back to scrot and cursor capture dies.e2e/cmd/desktopsmokeis the acceptance driver: claim onnone/2xlarge→/screenshot→/accessibility→ a PyAutoGUI click echoed by/cursor_position→ checkpoint and branch.docs/desktop.mddocuments the guest contract and the ports.Evidence
Built and run on a bare-metal node (Xeon 8481C, 192 threads / 503 GB, KVM), with the image imported through the usual
docker export→cocoon image importpath:desktopsmoke: PASS — warm claim 0.0 s, PNG 30 KB, AT-SPI tree 142 KB, the click echoed at (300, 300), and a branch of the warmed desktop answers/screenshotwithout relaunching the sessionmake go-lint10 ×0 issues.,make go-test18 packages ok with-race,asl ./...andGOOS=linux asl ./...cleanNotes
platforms): the flavor pullsgoogle-chrome-stable, which has no arm64 build.desktop:24.04.