feat(cli): add parameter aliasing, decoupled screenshots and semantic click by HugehardJiang · Pull Request #76 · iFurySt/open-computer-use

HugehardJiang · 2026-09-19T02:44:46Z

Summary

This PR addresses three key developer and AI Agent experience pain points identified during live GUI automation on macOS:

Parameter Aliasing Tolerance (API Drift Protection):
- AI agents frequently supply common parameter aliases such as index instead of element_index, button instead of mouse_button, or amount instead of pages, triggering errors like invalidArguments("click requires either element_index or x/y").
- The CLI launcher (renderLauncher) now normalizes these common aliases before handing off to the native dispatcher, providing seamless 100% backward-compatible execution.
Decoupled Screenshot Storage (Token Explosion & Buffer Overflow Prevention):
- In CLI and Agent environments, dumping multi-megabyte base64 PNG data into stdout often causes terminal buffer truncation, pipe stalls, and massive LLM context token consumption.
- By default, the launcher now decouples base64 images, persists the PNG to /tmp/ocu_last_screenshot.png, and outputs a lightweight image summary descriptor in JSON.
- Users/Agents can pass --raw-image or set OCU_RAW_IMAGE=1 to retain the original inline base64 behavior.
Semantic Text Click with Heuristic Role Scoring:
- click now supports passing text / title (and optional role), allowing agents to click elements by display text (e.g. {"app": "Music", "text": "播放", "role": "button"}).
- Automatically inspects the current window snapshot and ranks interactive targets (button, cell) higher than generic container wrappers (row, outline), resolving the target element_index automatically.
Documentation & Release Notes:
- Updated skills/open-computer-use/SKILL.md and skills/open-computer-use/references/usage.md with alias specifications and macOS complex control guidelines.
- Added feature release record in docs/releases/feature-release-notes.md.
- Added change history in docs/histories/2026-09/20260919-1045-cli-enhancements-aliases-decoupled-screenshots.md.

Verification

Validated parameter aliases (index: 40, button: "left") against live native macOS apps.
Validated decoupled screenshot creation at /tmp/ocu_last_screenshot.png (verified PNG headers and image validity).
Validated batch execution via --calls and MCP stdio passthrough.
Validated documentation with make check-docs (passed).

… click

hyprcat

None of this code is macOS-specific. All of it lives in the Node launcher template in scripts/npm/build-packages.mjs, which every platform shares, plus docs. The only macOS-flavoured part is the Chinese role names (按钮 / 单元格 / 文本) in the click scoring. No Swift changes.

Main has moved on in ways that overlap most of this PR:

Launcher rewrite (#80). Since #80, the generated launcher is a thin shim that imports scripts/node-repl/open-computer-use-cli.mjs (ESM). The renderLauncher body this PR edits, with spawnAndExit, installCommands and the call/mcp branches, no longer exists. This needs a port, not a rebase.
Semantic click vs query (#73). query finds controls by text/role natively, with exact-then-contains matching and stable indexes (≥ 1,000,000) that click, set_value, scroll and perform_secondary_action accept.
Screenshot decoupling vs native options. Screenshot size is now handled natively: include_screenshot: false on get_app_state (#101), OPEN_COMPUTER_USE_ACTION_READ_BACK=0 for short action results (ComputerUseService.swift:252 on main), and image.maxLongEdgePixels / image.format (2cf3eae).

Suggestion: drop semantic click and screenshot decoupling in favour of those. If parameter aliases are still wanted, add them in ComputerUseToolDispatcher as a small separate PR with tests. That is the one place where the CLI, MCP and js all pick them up.

I rendered this PR's launcher template against a fake native binary and confirmed the issues below.

Other notes

No tests. Main has scripts/node-repl/open-computer-use-cli.test.mjs, where launcher behaviour like this is covered.

How I checked

Tests. swift test on the head: Executed 167 tests, with 1 test skipped and 0 failures. That is just its September base, since the PR has no Swift changes. make check-docs passes.

Conflicts. git merge-tree --write-tree origin/main review/pr-76 gives CONFLICT (content) in docs/releases/feature-release-notes.md, scripts/npm/build-packages.mjs and skills/open-computer-use/SKILL.md.

Launcher harness. I rendered the PR's renderLauncher() template with a stub platform table that points at a shell script, which logs its argv and echoes MCP lines. The fixed /tmp path was swapped for a scratch path.

--- MCP: one request on stdin, then EOF
{"jsonrpc":"2.0","id":1,...,"arguments":{"app":"X","element_index":3}}
launcher exit=142 (killed by the 5 s alarm)
1792698329 argv: mcp
1792698334 mcp: stdin EOF, exiting          <- only after the launcher died
--- click {"text":"Save"} against "1 button Save As…" / "3 button Save"
argv: snapshot X
argv: call click --args {"app":"X","element_index":1}
--- --calls with a semantic click second
argv: snapshot X
argv: call --calls [...{"element_index":1},{"element_index":1}]
both results: {"type":"image_saved","path":".../shot.png",...}; file contents: BBBB
--- no match {"text":"Delete account"}
argv: call click --args {"app":"X"}

hyprcat · 2026-10-22T19:58:24Z

+  });
+
+  let stdinBuffer = "";
+  process.stdin.on("data", (chunk) => {


When the client closes stdin, the launcher never passes that on, so the native MCP server never sees EOF and the launcher never exits. With one request piped in, the launcher was still running after 5 s. The fake native server logged mcp: stdin EOF, exiting only when the launcher was killed. On main the child inherited stdio, so EOF went straight through. This also means anything the runtime does when the connection closes, such as restoring parked windows, waits until the client kills the process. At minimum this needs process.stdin.on("end", () => child.stdin.end()).

hyprcat · 2026-10-22T19:58:24Z

+          const role = m[2];
+          const rest = m[3] || "";
+          const fullLine = role + " " + rest;
+          if (fullLine.toLowerCase().includes(targetText)) {


The match is a substring test on the whole line, and candidates are ranked only by role, so ties go to the first line. With 1 button Save As… and 3 button Save in the snapshot, {"text":"Save"} was sent as element_index: 1, which clicks Save As…. With labels like "Delete", a wrong pick is a real risk. query already handles this with exact-then-contains matching.

hyprcat · 2026-10-22T19:58:24Z

+        candidates.sort((a, b) => b.score - a.score);
+        argsObj.element_index = candidates[0].idx;
+      }
+    } catch (err) {}


When nothing matches, or the snapshot throws, the catch is empty and lines 430–433 then delete text/title/role. The native side receives {"app":"X"} and reports "click requires either element_index or x/y", which hides the actual reason (no element matched).

hyprcat · 2026-10-22T19:58:24Z

+    } catch (e) {}
+  }
+
+  const callsIdx = args.indexOf("--calls");


In --calls, every semantic click is resolved before the batch starts. The fake binary logged snapshot X and only then call --calls …. So a click that follows a navigation step in the same batch is resolved against the UI as it was before the batch.

hyprcat · 2026-10-22T19:58:24Z

+    try {
+      const { execFileSync } = require("node:child_process");
+      const nativeBin = resolveNativeExecutable();
+      const snapshotOut = execFileSync(nativeBin, ["snapshot", argsObj.app], {


The index comes from a separate snapshot process and is then used in another process. In MCP mode, the synchronous execFileSync also blocks the stdin handler for up to 5 s per click.

hyprcat · 2026-10-22T19:58:24Z

+  function handleContentItem(item) {
+    if (item && item.type === "image" && typeof item.data === "string") {
+      try {
+        const savePath = "/tmp/ocu_last_screenshot.png";


Every image is written to the same fixed path:

In a --calls batch with two images, both results pointed at one file, which held only the second image (BBBB).

Because it is a fixed name in shared /tmp, fs.writeFileSync follows a symlink planted there. I checked this: the write went through the symlink and overwrote the target file.

On main the format can be jpg or webp (2cf3eae), but the file is always named .png.

It also changes the default output of call for every existing consumer (image becomes image_saved), so the "100% backward-compatible" claim doesn't hold.

hyprcat · 2026-10-22T19:58:25Z

+
+  const toolName = args[1];
+
+  const argsIdx = args.indexOf("--args");


Only --args and --calls are normalized. --args-file and --calls-file skip the aliases.

hyprcat · 2026-10-22T19:58:25Z

+
+### Screenshot Decoupling in CLI
+
+By default, CLI tool calls decouple multi-megabyte base64 screenshot data from stdout and write the image to `/tmp/ocu_last_screenshot.png`. This prevents terminal buffer overflow and large LLM context overhead. The stdout JSON response summarizes the image reference:


This documents the fixed /tmp path. It would go with the decoupling.

feat(cli): add parameter aliasing, decoupled screenshots and semantic…

e61a1e9

… click

hyprcat suggested changes Oct 22, 2026

View reviewed changes

Navigation Menu

Repository navigation

feat(cli): add parameter aliasing, decoupled screenshots and semantic click - #76

feat(cli): add parameter aliasing, decoupled screenshots and semantic click#76
HugehardJiang wants to merge 1 commit into
iFurySt:mainfrom
HugehardJiang:feat/cli-enhancements-parameter-aliasing-and-decoupled-screenshots

HugehardJiang commented Sep 19, 2026

Uh oh!

hyprcat left a comment

Uh oh!

hyprcat Oct 22, 2026

Uh oh!

hyprcat Oct 22, 2026

Uh oh!

hyprcat Oct 22, 2026

Uh oh!

hyprcat Oct 22, 2026

Uh oh!

hyprcat Oct 22, 2026

Uh oh!

hyprcat Oct 22, 2026

Uh oh!

hyprcat Oct 22, 2026

Uh oh!

hyprcat Oct 22, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		const toolName = args[1];

		const argsIdx = args.indexOf("--args");


		### Screenshot Decoupling in CLI

		By default, CLI tool calls decouple multi-megabyte base64 screenshot data from stdout and write the image to `/tmp/ocu_last_screenshot.png`. This prevents terminal buffer overflow and large LLM context overhead. The stdout JSON response summarizes the image reference:

Conversation

HugehardJiang commented Sep 19, 2026

Summary

Verification

Uh oh!

hyprcat left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants