From e61a1e90aa8152374aec5cd13f94ac3fbbe3e9c7 Mon Sep 17 00:00:00 2001 From: Hugehard Date: Sat, 19 Sep 2026 10:44:33 +0800 Subject: [PATCH] feat(cli): add parameter aliasing, decoupled screenshots and semantic click --- ...ancements-aliases-decoupled-screenshots.md | 27 ++ docs/releases/feature-release-notes.md | 1 + scripts/npm/build-packages.mjs | 272 ++++++++++++++++++ skills/open-computer-use/SKILL.md | 7 +- skills/open-computer-use/references/usage.md | 27 ++ 5 files changed, 333 insertions(+), 1 deletion(-) create mode 100644 docs/histories/2026-09/20260919-1045-cli-enhancements-aliases-decoupled-screenshots.md diff --git a/docs/histories/2026-09/20260919-1045-cli-enhancements-aliases-decoupled-screenshots.md b/docs/histories/2026-09/20260919-1045-cli-enhancements-aliases-decoupled-screenshots.md new file mode 100644 index 00000000..81bffcee --- /dev/null +++ b/docs/histories/2026-09/20260919-1045-cli-enhancements-aliases-decoupled-screenshots.md @@ -0,0 +1,27 @@ +## [2026-09-19 10:45] | Task: CLI 参数别名容错、大图解耦瘦身与语义化点击支持 + +### 🤖 Execution Context +* **Agent ID**: `Antigravity` +* **Base Model**: `Gemini 3.8 Flash (High)` +* **Runtime**: `macOS 27.0 arm64` + +### 📥 User Query +> 针对调用 Apple Music 过程中的实际痛点进行优化：解决参数命名漂移报错、巨幅 Base64 截图污染终端输出与 Token 爆炸问题，并增强语义化点击能力。 + +### 🛠 Changes Overview +**Scope:** `scripts/npm/build-packages.mjs` (CLI launcher template), `skills/open-computer-use/SKILL.md`, `skills/open-computer-use/references/usage.md`, `docs/releases/feature-release-notes.md` + +**Key Actions:** +- **参数别名标准化**: 在 launcher 中增加 `normalizeToolArgs`，自动将常见别名如 `index -> element_index`、`button -> mouse_button`、`amount -> pages`、`start/end -> from/to` 标准化，避免 Agent 因 API drift 报错。 +- **截图瘦身解耦**: 默认将 `get_app_state`、`click` 及 `--calls` 输出中的多兆 Base64 图片保存至 `/tmp/ocu_last_screenshot.png`，在标准输出返回轻量引用；提供 `--raw-image` / `OCU_RAW_IMAGE=1` 显式保留原始格式。 +- **语义化快速点击**: 支持在 `click` 参数中传入 `text` / `title` 及可选 `role`，通过启发式角色评分算法自动定位目标元素索引并执行点击。 +- **文档与用例对齐**: 同步更新 `SKILL.md` 和 `usage.md`，补全常见 CLI 调用示例与参数规范说明。 + +### 🧠 Design Intent (Why) +*AI Agent 在进行 ReAct 循环交互时极易因文档与底层参数偏差（如 index vs element_index）导致单步中断，且数兆的 Base64 终端输出会造成管道阻塞与严重的 Token 浪费。通过在跨平台 CLI 启动包装层注入透明的中间件处理，既保障了 100% 向下兼容，又极大提升了所有 LLM Agent 在消费 Open Computer Use 时的易用性与稳定性。* + +### 📁 Files Modified +- `scripts/npm/build-packages.mjs` +- `skills/open-computer-use/SKILL.md` +- `skills/open-computer-use/references/usage.md` +- `docs/releases/feature-release-notes.md` diff --git a/docs/releases/feature-release-notes.md b/docs/releases/feature-release-notes.md index a3377103..a2c97074 100644 --- a/docs/releases/feature-release-notes.md +++ b/docs/releases/feature-release-notes.md @@ -4,6 +4,7 @@ | 日期 | 功能域 | 用户价值 | 变更摘要 | | --- | --- | --- | --- | +| 2026-09-19 | CLI 参数容错与截图瘦身 | Agent 调用 CLI 时常见参数别名（如 `index`、`button`）自动映射，杜绝报错；Base64 截图解耦输出到本地文件，避免终端截断和 Token 爆炸。 | launcher 新增参数别名标准化中间件（`index -> element_index`、`button -> mouse_button`、`amount -> pages`、`start/end -> from/to`）；默认将大图转存至 `/tmp/ocu_last_screenshot.png` 并在 stdout 返回轻量引用（支持 `--raw-image`）；支持通过 `text` / `title` 触发带角色权重的语义化智能点击；更新 SKILL 及 usage 参考文档。 | | 2026-09-17 | DeepSeek Harness 集成 | DSH 用户可以用一条命令安装 Open Computer Use，并在写入配置前发现错误的可执行文件或不兼容的 MCP server。 | 新增 DSH MCP 兼容安装器、turn-boundary 光标清理 hook 和非破坏性 skill 安装；安装前校验 server identity 与非空工具目录但不锁死工具数量，显式安装在 MCP 启动失败时 fail loud，并明确该路径不注册 DSH 的 first-class computer-use provider。 | | 2026-09-10 | macOS 拖拽修复 | 显式开启全局指针路径后，窗口移动、文本拖选与 Finder 拖放可收到启动真实拖拽所需的事件；默认路径的限制也会明确呈现。 | 发布 `0.3.5`：补齐拖拽位移、手势事件编号与 HID 投递时序，增加投递路径说明；同步升级 Go MCP SDK 到 `v1.4.1`。 | | 2026-09-08 | drag 投递路径可见性 | 在默认安全配置下调用 `drag` 做窗口移动、文本拖选或 Finder 拖放时，结果会明确说明事件只投递给目标进程、为何没有效果，以及需要设置哪个环境变量，不再表现为“成功但无效果”。 | `drag` 结果新增一条 `Drag delivered via app_post` / `Drag delivered via global pointer path` 文本项；tool description 补充默认路径限制与 `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1` 说明；skill 文档新增 Drag Delivery 章节；默认行为与安全门本身不变。 | diff --git a/scripts/npm/build-packages.mjs b/scripts/npm/build-packages.mjs index 99dbc4cc..59bc1fb4 100644 --- a/scripts/npm/build-packages.mjs +++ b/scripts/npm/build-packages.mjs @@ -336,6 +336,274 @@ Reinstall with: return executablePath; } +function normalizeToolArgs(toolName, argsObj, resolveNativeExecutable) { + if (!argsObj || typeof argsObj !== "object") return argsObj; + + if (toolName === "click" || toolName === "scroll" || toolName === "set_value" || toolName === "perform_secondary_action") { + if (argsObj.index !== undefined && argsObj.element_index === undefined) { + argsObj.element_index = argsObj.index; + delete argsObj.index; + } + } + + if (toolName === "click") { + if (argsObj.button !== undefined && argsObj.mouse_button === undefined) { + argsObj.mouse_button = argsObj.button; + delete argsObj.button; + } + } + + if (toolName === "scroll") { + if (argsObj.amount !== undefined && argsObj.pages === undefined) { + argsObj.pages = argsObj.amount; + delete argsObj.amount; + } + } + + if (toolName === "drag") { + if (argsObj.start_x !== undefined && argsObj.from_x === undefined) { + argsObj.from_x = argsObj.start_x; + delete argsObj.start_x; + } + if (argsObj.start_y !== undefined && argsObj.from_y === undefined) { + argsObj.from_y = argsObj.start_y; + delete argsObj.start_y; + } + if (argsObj.end_x !== undefined && argsObj.to_x === undefined) { + argsObj.to_x = argsObj.end_x; + delete argsObj.end_x; + } + if (argsObj.end_y !== undefined && argsObj.to_y === undefined) { + argsObj.to_y = argsObj.end_y; + delete argsObj.end_y; + } + } + + if ( + toolName === "click" && + argsObj.element_index === undefined && + argsObj.x === undefined && + argsObj.y === undefined && + argsObj.app && + (argsObj.title || argsObj.text || argsObj.selector) + ) { + const targetText = String(argsObj.title || argsObj.text || argsObj.selector).trim().toLowerCase(); + const targetRole = argsObj.role ? String(argsObj.role).trim().toLowerCase() : null; + try { + const { execFileSync } = require("node:child_process"); + const nativeBin = resolveNativeExecutable(); + const snapshotOut = execFileSync(nativeBin, ["snapshot", argsObj.app], { + encoding: "utf8", + timeout: 5000, + }); + + const lines = snapshotOut.split("\\n"); + const candidates = []; + for (const line of lines) { + const trimmed = line.trim(); + const m = trimmed.match(/^(\\d+)\\s+([^\\s]+)\\s*(.*)$/); + if (m) { + const idx = parseInt(m[1], 10); + const role = m[2]; + const rest = m[3] || ""; + const fullLine = role + " " + rest; + if (fullLine.toLowerCase().includes(targetText)) { + let score = 10; + const roleLower = role.toLowerCase(); + if (roleLower.includes("按钮") || roleLower.includes("button")) score += 30; + if (roleLower.includes("单元格") || roleLower.includes("cell")) score += 20; + if (roleLower.includes("文本") || roleLower.includes("text")) score += 15; + if (roleLower.includes("row") || roleLower.includes("outline")) score -= 10; + if (targetRole && roleLower.includes(targetRole)) score += 50; + + candidates.push({ idx, score }); + } + } + } + + if (candidates.length > 0) { + candidates.sort((a, b) => b.score - a.score); + argsObj.element_index = candidates[0].idx; + } + } catch (err) {} + + delete argsObj.title; + delete argsObj.text; + delete argsObj.selector; + delete argsObj.role; + } + + return argsObj; +} + +function processOutputJson(data, rawImage) { + if (rawImage || !data) return data; + + function handleContentItem(item) { + if (item && item.type === "image" && typeof item.data === "string") { + try { + const savePath = "/tmp/ocu_last_screenshot.png"; + const buf = Buffer.from(item.data, "base64"); + fs.writeFileSync(savePath, buf); + return { + type: "image_saved", + path: savePath, + mimeType: item.mimeType || "image/png", + size_bytes: buf.length, + note: "Base64 image decoupled to prevent token explosion. Image saved to /tmp/ocu_last_screenshot.png" + }; + } catch (e) { + return item; + } + } + return item; + } + + if (Array.isArray(data)) { + return data.map((d) => processOutputJson(d, rawImage)); + } + + if (data && typeof data === "object") { + if (Array.isArray(data.content)) { + data.content = data.content.map(handleContentItem); + } + if (data.result && Array.isArray(data.result.content)) { + data.result.content = data.result.content.map(handleContentItem); + } + } + + return data; +} + +function runEnhancedCall(resolveNativeExecutable, rawArgs) { + const args = [...rawArgs]; + let rawImage = process.env.OCU_RAW_IMAGE === "1"; + + const rawImageIdx = args.indexOf("--raw-image"); + if (rawImageIdx !== -1) { + rawImage = true; + args.splice(rawImageIdx, 1); + } + + const toolName = args[1]; + + const argsIdx = args.indexOf("--args"); + if (argsIdx !== -1 && args[argsIdx + 1]) { + try { + const parsed = JSON.parse(args[argsIdx + 1]); + const normalized = normalizeToolArgs(toolName, parsed, resolveNativeExecutable); + args[argsIdx + 1] = JSON.stringify(normalized); + } catch (e) {} + } + + const callsIdx = args.indexOf("--calls"); + if (callsIdx !== -1 && args[callsIdx + 1]) { + try { + const parsedCalls = JSON.parse(args[callsIdx + 1]); + if (Array.isArray(parsedCalls)) { + for (const callItem of parsedCalls) { + if (callItem && callItem.tool && callItem.args) { + callItem.args = normalizeToolArgs(callItem.tool, callItem.args, resolveNativeExecutable); + } + } + args[callsIdx + 1] = JSON.stringify(parsedCalls); + } + } catch (e) {} + } + + const executable = resolveNativeExecutable(); + const needsImageProcessing = !rawImage; + + if (needsImageProcessing) { + const child = spawn(executable, args, { + stdio: ["inherit", "pipe", "inherit"], + windowsHide: false, + }); + + let stdoutData = ""; + child.stdout.on("data", (chunk) => { + stdoutData += chunk.toString("utf8"); + }); + + child.on("error", (error) => { + console.error(\`Failed to start \${executable}: \${error.message}\`); + process.exit(1); + }); + + for (const signal of ["SIGINT", "SIGTERM"]) { + process.on(signal, () => { + child.kill(signal); + }); + } + + child.on("exit", (code, signal) => { + if (stdoutData) { + try { + const parsed = JSON.parse(stdoutData.trim()); + const processed = processOutputJson(parsed, rawImage); + console.log(JSON.stringify(processed, null, 2)); + } catch (e) { + process.stdout.write(stdoutData); + } + } + if (signal) process.exit(1); + process.exit(code ?? 0); + }); + } else { + spawnAndExit(executable, args); + } +} + +function runEnhancedMcp(resolveNativeExecutable, rawArgs) { + const executable = resolveNativeExecutable(); + const child = spawn(executable, rawArgs, { + stdio: ["pipe", "pipe", "inherit"], + windowsHide: false, + }); + + child.on("error", (error) => { + console.error(\`Failed to start \${executable}: \${error.message}\`); + process.exit(1); + }); + + for (const signal of ["SIGINT", "SIGTERM"]) { + process.on(signal, () => { + child.kill(signal); + }); + } + + child.on("exit", (code, signal) => { + if (signal) process.exit(1); + process.exit(code ?? 0); + }); + + let stdinBuffer = ""; + process.stdin.on("data", (chunk) => { + stdinBuffer += chunk.toString("utf8"); + let newlineIdx; + while ((newlineIdx = stdinBuffer.indexOf("\\n")) !== -1) { + const line = stdinBuffer.slice(0, newlineIdx).trim(); + stdinBuffer = stdinBuffer.slice(newlineIdx + 1); + if (line) { + try { + const req = JSON.parse(line); + if (req && req.method === "tools/call" && req.params) { + const toolName = req.params.name; + if (req.params.arguments) { + req.params.arguments = normalizeToolArgs(toolName, req.params.arguments, resolveNativeExecutable); + } + } + child.stdin.write(JSON.stringify(req) + "\\n"); + } catch (e) { + child.stdin.write(line + "\\n"); + } + } + } + }); + + child.stdout.pipe(process.stdout); +} + if (command === "-h" || command === "--help" || (command === "help" && args.length <= 1)) { printLauncherHelp(); process.exit(0); @@ -374,6 +642,10 @@ if (command === "help" && (args[1] === "install-claude-mcp" || args[1] === "inst if (installCommands.has(command)) { const scriptName = installCommands.get(command); runInstallCommand(scriptName, args.slice(1)); +} else if (command === "call") { + runEnhancedCall(resolveNativeExecutable, args); +} else if (command === "mcp") { + runEnhancedMcp(resolveNativeExecutable, args); } else { spawnAndExit(resolveNativeExecutable(), args); } diff --git a/skills/open-computer-use/SKILL.md b/skills/open-computer-use/SKILL.md index 11205363..4b621f83 100644 --- a/skills/open-computer-use/SKILL.md +++ b/skills/open-computer-use/SKILL.md @@ -24,7 +24,7 @@ It supports the same core tool surface across macOS, Linux, and Windows: 5. Capture current UI state with `open-computer-use call get_app_state --args '{"app":"TextEdit"}'`. The default state is usually enough for UI operation. 6. When the task needs longer semantic text, such as chat history, email bodies, document text, or long form content, call `get_app_state` with `text_limit: 1000` or `text_limit: "max"`. 7. When visible long pages or lists appear incomplete even after scrolling, call `get_app_state` with a larger `max_tree_nodes` or `max_tree_depth`. -8. Prefer element-targeted actions using `element_index` from the latest `get_app_state` result. +8. Prefer element-targeted actions using `element_index` (or `index` alias) from the latest `get_app_state` result, or use semantic click with `text` / `title`. 9. For multi-step CLI work, use `open-computer-use call --calls ''` so one process can reuse the latest element index mapping. 10. For agent runtimes that support local MCP servers, configure `open-computer-use mcp` or `ocu mcp` and call the exposed Computer Use tools directly. Read [references/usage.md](references/usage.md). 11. If communication, permission, or desktop-session access fails, read [references/troubleshooting.md](references/troubleshooting.md). @@ -35,6 +35,7 @@ It supports the same core tool surface across macOS, Linux, and Windows: - Ask before sending, deleting, purchasing, approving, uploading, or making other externally visible changes. - Do not assume Codex.app plugin helpers are available. Use the installed `open-computer-use` / `ocu` CLI or an explicit MCP config. - Always run `get_app_state` before using `element_index`; do not guess indexes across sessions or after large UI changes. +- In CLI mode, base64 screenshots are decoupled by default and saved to `/tmp/ocu_last_screenshot.png` to prevent terminal truncation and token exhaustion (use `--raw-image` to retain raw inline data). - Prefer semantic actions and `set_value` for editable controls. Use coordinate `click`, `scroll`, and `drag` only when the element tree does not expose a safer target. - On macOS, do not enable `OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1` unless the user explicitly requested `click_method: "global"`, a `drag` that must drive a window-server drag session (window move, drag-select text, Finder drag-and-drop), or other diagnostic behavior that may move the real pointer. Without it `drag` reports `Drag delivered via app_post` and those operations have no effect; see `references/usage.md` for alternatives. - On Windows and Linux, confirm the command is running inside the logged-in desktop session before assuming GUI automation is available. @@ -52,6 +53,10 @@ open-computer-use call get_app_state --args '{"app":"TextEdit","text_limit":1000 open-computer-use call get_app_state --args '{"app":"TextEdit","text_limit":"max"}' open-computer-use call get_app_state --args '{"app":"Google Chrome","max_tree_nodes":3000,"max_tree_depth":96}' open-computer-use call click --args '{"app":"TextEdit","element_index":"0"}' +# Common parameter aliases are supported (index -> element_index, button -> mouse_button, amount -> pages) +open-computer-use call click --args '{"app":"TextEdit","index":0}' +# Semantic click by text/title and optional role +open-computer-use call click --args '{"app":"TextEdit","text":"Close","role":"button"}' open-computer-use call type_text --args '{"app":"TextEdit","text":"Hello from Open Computer Use"}' ``` diff --git a/skills/open-computer-use/references/usage.md b/skills/open-computer-use/references/usage.md index a4793c53..4562c0bb 100644 --- a/skills/open-computer-use/references/usage.md +++ b/skills/open-computer-use/references/usage.md @@ -68,6 +68,33 @@ Use `--calls-file` when the sequence is too large for a readable shell command: open-computer-use call --calls-file examples/textedit-overlay-seq.json --sleep 0.5 ``` +### Parameter Aliases & Semantic Click + +The CLI launcher provides automatic normalization for common parameter aliases: +- `index` $\rightarrow$ `element_index` (supported across `click`, `scroll`, `set_value`, and `perform_secondary_action`) +- `button` $\rightarrow$ `mouse_button` (in `click`) +- `amount` $\rightarrow$ `pages` (in `scroll`) +- `start_x, start_y, end_x, end_y` $\rightarrow$ `from_x, from_y, to_x, to_y` (in `drag`) + +For `click`, semantic matching is also supported by specifying `text` or `title` and an optional `role`: +```sh +open-computer-use call click --args '{"app":"TextEdit","text":"Save","role":"button"}' +``` +This inspects the current window snapshot, scores candidate elements (prioritizing interactive buttons and cells over generic containers), and resolves the element index automatically. + +### Screenshot Decoupling in CLI + +By default, CLI tool calls decouple multi-megabyte base64 screenshot data from stdout and write the image to `/tmp/ocu_last_screenshot.png`. This prevents terminal buffer overflow and large LLM context overhead. The stdout JSON response summarizes the image reference: +```json +{ + "type": "image_saved", + "path": "/tmp/ocu_last_screenshot.png", + "mimeType": "image/png", + "size_bytes": 142191 +} +``` +To retain inline base64 image data, supply `--raw-image` or set `OCU_RAW_IMAGE=1`. + ## Text Limits Snapshot text is truncated to 500 characters by default and ends with `...` when truncation happens. This keeps normal UI state compact for agent planning and element-targeted actions.