Repository navigation
feat(cli): add parameter aliasing, decoupled screenshots and semantic click #76
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| ## [2026-09-19 10:45] | Task: CLI 参数别名容错、大图解耦瘦身与语义化点击支持 | ||
|
|
||
| ### 🤖 Execution Context | ||
| * **Agent ID**: `Antigravity` | ||
| * **Base Model**: `Gemini 3.8 Flash (High)` | ||
| * **Runtime**: `macOS 27.0 arm64` | ||
|
|
||
| ### 📥 User Query | ||
| > 针对调用 Apple Music 过程中的实际痛点进行优化：解决参数命名漂移报错、巨幅 Base64 截图污染终端输出与 Token 爆炸问题，并增强语义化点击能力。 | ||
|
|
||
| ### 🛠 Changes Overview | ||
| **Scope:** `scripts/npm/build-packages.mjs` (CLI launcher template), `skills/open-computer-use/SKILL.md`, `skills/open-computer-use/references/usage.md`, `docs/releases/feature-release-notes.md` | ||
|
|
||
| **Key Actions:** | ||
| - **参数别名标准化**: 在 launcher 中增加 `normalizeToolArgs`，自动将常见别名如 `index -> element_index`、`button -> mouse_button`、`amount -> pages`、`start/end -> from/to` 标准化，避免 Agent 因 API drift 报错。 | ||
| - **截图瘦身解耦**: 默认将 `get_app_state`、`click` 及 `--calls` 输出中的多兆 Base64 图片保存至 `/tmp/ocu_last_screenshot.png`，在标准输出返回轻量引用；提供 `--raw-image` / `OCU_RAW_IMAGE=1` 显式保留原始格式。 | ||
| - **语义化快速点击**: 支持在 `click` 参数中传入 `text` / `title` 及可选 `role`，通过启发式角色评分算法自动定位目标元素索引并执行点击。 | ||
| - **文档与用例对齐**: 同步更新 `SKILL.md` 和 `usage.md`，补全常见 CLI 调用示例与参数规范说明。 | ||
|
|
||
| ### 🧠 Design Intent (Why) | ||
| *AI Agent 在进行 ReAct 循环交互时极易因文档与底层参数偏差（如 index vs element_index）导致单步中断，且数兆的 Base64 终端输出会造成管道阻塞与严重的 Token 浪费。通过在跨平台 CLI 启动包装层注入透明的中间件处理，既保障了 100% 向下兼容，又极大提升了所有 LLM Agent 在消费 Open Computer Use 时的易用性与稳定性。* | ||
|
|
||
| ### 📁 Files Modified | ||
| - `scripts/npm/build-packages.mjs` | ||
| - `skills/open-computer-use/SKILL.md` | ||
| - `skills/open-computer-use/references/usage.md` | ||
| - `docs/releases/feature-release-notes.md` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -336,6 +336,274 @@ Reinstall with: | |
| return executablePath; | ||
| } | ||
|
|
||
| function normalizeToolArgs(toolName, argsObj, resolveNativeExecutable) { | ||
| if (!argsObj || typeof argsObj !== "object") return argsObj; | ||
|
|
||
| if (toolName === "click" || toolName === "scroll" || toolName === "set_value" || toolName === "perform_secondary_action") { | ||
| if (argsObj.index !== undefined && argsObj.element_index === undefined) { | ||
| argsObj.element_index = argsObj.index; | ||
| delete argsObj.index; | ||
| } | ||
| } | ||
|
|
||
| if (toolName === "click") { | ||
| if (argsObj.button !== undefined && argsObj.mouse_button === undefined) { | ||
| argsObj.mouse_button = argsObj.button; | ||
| delete argsObj.button; | ||
| } | ||
| } | ||
|
|
||
| if (toolName === "scroll") { | ||
| if (argsObj.amount !== undefined && argsObj.pages === undefined) { | ||
| argsObj.pages = argsObj.amount; | ||
| delete argsObj.amount; | ||
| } | ||
| } | ||
|
|
||
| if (toolName === "drag") { | ||
| if (argsObj.start_x !== undefined && argsObj.from_x === undefined) { | ||
| argsObj.from_x = argsObj.start_x; | ||
| delete argsObj.start_x; | ||
| } | ||
| if (argsObj.start_y !== undefined && argsObj.from_y === undefined) { | ||
| argsObj.from_y = argsObj.start_y; | ||
| delete argsObj.start_y; | ||
| } | ||
| if (argsObj.end_x !== undefined && argsObj.to_x === undefined) { | ||
| argsObj.to_x = argsObj.end_x; | ||
| delete argsObj.end_x; | ||
| } | ||
| if (argsObj.end_y !== undefined && argsObj.to_y === undefined) { | ||
| argsObj.to_y = argsObj.end_y; | ||
| delete argsObj.end_y; | ||
| } | ||
| } | ||
|
|
||
| if ( | ||
| toolName === "click" && | ||
| argsObj.element_index === undefined && | ||
| argsObj.x === undefined && | ||
| argsObj.y === undefined && | ||
| argsObj.app && | ||
| (argsObj.title || argsObj.text || argsObj.selector) | ||
| ) { | ||
| const targetText = String(argsObj.title || argsObj.text || argsObj.selector).trim().toLowerCase(); | ||
| const targetRole = argsObj.role ? String(argsObj.role).trim().toLowerCase() : null; | ||
| try { | ||
| const { execFileSync } = require("node:child_process"); | ||
| const nativeBin = resolveNativeExecutable(); | ||
| const snapshotOut = execFileSync(nativeBin, ["snapshot", argsObj.app], { | ||
| encoding: "utf8", | ||
| timeout: 5000, | ||
| }); | ||
|
|
||
| const lines = snapshotOut.split("\\n"); | ||
| const candidates = []; | ||
| for (const line of lines) { | ||
| const trimmed = line.trim(); | ||
| const m = trimmed.match(/^(\\d+)\\s+([^\\s]+)\\s*(.*)$/); | ||
| if (m) { | ||
| const idx = parseInt(m[1], 10); | ||
| const role = m[2]; | ||
| const rest = m[3] || ""; | ||
| const fullLine = role + " " + rest; | ||
| if (fullLine.toLowerCase().includes(targetText)) { | ||
|
Contributor
There was a problem hiding this comment. The match is a substring test on the whole line, and candidates are ranked only by role, so ties go to the first line. With |
||
| let score = 10; | ||
| const roleLower = role.toLowerCase(); | ||
| if (roleLower.includes("按钮") || roleLower.includes("button")) score += 30; | ||
| if (roleLower.includes("单元格") || roleLower.includes("cell")) score += 20; | ||
| if (roleLower.includes("文本") || roleLower.includes("text")) score += 15; | ||
| if (roleLower.includes("row") || roleLower.includes("outline")) score -= 10; | ||
| if (targetRole && roleLower.includes(targetRole)) score += 50; | ||
|
|
||
| candidates.push({ idx, score }); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| if (candidates.length > 0) { | ||
| candidates.sort((a, b) => b.score - a.score); | ||
| argsObj.element_index = candidates[0].idx; | ||
| } | ||
| } catch (err) {} | ||
|
Contributor
There was a problem hiding this comment. When nothing matches, or the snapshot throws, the catch is empty and lines 430–433 then delete |
||
|
|
||
| delete argsObj.title; | ||
| delete argsObj.text; | ||
| delete argsObj.selector; | ||
| delete argsObj.role; | ||
| } | ||
|
|
||
| return argsObj; | ||
| } | ||
|
|
||
| function processOutputJson(data, rawImage) { | ||
| if (rawImage || !data) return data; | ||
|
|
||
| function handleContentItem(item) { | ||
| if (item && item.type === "image" && typeof item.data === "string") { | ||
| try { | ||
| const savePath = "/tmp/ocu_last_screenshot.png"; | ||
|
Contributor
There was a problem hiding this comment. Every image is written to the same fixed path:
|
||
| const buf = Buffer.from(item.data, "base64"); | ||
| fs.writeFileSync(savePath, buf); | ||
| return { | ||
| type: "image_saved", | ||
| path: savePath, | ||
| mimeType: item.mimeType || "image/png", | ||
| size_bytes: buf.length, | ||
| note: "Base64 image decoupled to prevent token explosion. Image saved to /tmp/ocu_last_screenshot.png" | ||
| }; | ||
| } catch (e) { | ||
| return item; | ||
| } | ||
| } | ||
| return item; | ||
| } | ||
|
|
||
| if (Array.isArray(data)) { | ||
| return data.map((d) => processOutputJson(d, rawImage)); | ||
| } | ||
|
|
||
| if (data && typeof data === "object") { | ||
| if (Array.isArray(data.content)) { | ||
| data.content = data.content.map(handleContentItem); | ||
| } | ||
| if (data.result && Array.isArray(data.result.content)) { | ||
| data.result.content = data.result.content.map(handleContentItem); | ||
| } | ||
| } | ||
|
|
||
| return data; | ||
| } | ||
|
|
||
| function runEnhancedCall(resolveNativeExecutable, rawArgs) { | ||
| const args = [...rawArgs]; | ||
| let rawImage = process.env.OCU_RAW_IMAGE === "1"; | ||
|
|
||
| const rawImageIdx = args.indexOf("--raw-image"); | ||
| if (rawImageIdx !== -1) { | ||
| rawImage = true; | ||
| args.splice(rawImageIdx, 1); | ||
| } | ||
|
|
||
| const toolName = args[1]; | ||
|
|
||
| const argsIdx = args.indexOf("--args"); | ||
|
Contributor
There was a problem hiding this comment. Only |
||
| if (argsIdx !== -1 && args[argsIdx + 1]) { | ||
| try { | ||
| const parsed = JSON.parse(args[argsIdx + 1]); | ||
| const normalized = normalizeToolArgs(toolName, parsed, resolveNativeExecutable); | ||
| args[argsIdx + 1] = JSON.stringify(normalized); | ||
| } catch (e) {} | ||
| } | ||
|
|
||
| const callsIdx = args.indexOf("--calls"); | ||
|
Contributor
There was a problem hiding this comment. In |
||
| if (callsIdx !== -1 && args[callsIdx + 1]) { | ||
| try { | ||
| const parsedCalls = JSON.parse(args[callsIdx + 1]); | ||
| if (Array.isArray(parsedCalls)) { | ||
| for (const callItem of parsedCalls) { | ||
| if (callItem && callItem.tool && callItem.args) { | ||
| callItem.args = normalizeToolArgs(callItem.tool, callItem.args, resolveNativeExecutable); | ||
| } | ||
| } | ||
| args[callsIdx + 1] = JSON.stringify(parsedCalls); | ||
| } | ||
| } catch (e) {} | ||
| } | ||
|
|
||
| const executable = resolveNativeExecutable(); | ||
| const needsImageProcessing = !rawImage; | ||
|
|
||
| if (needsImageProcessing) { | ||
| const child = spawn(executable, args, { | ||
| stdio: ["inherit", "pipe", "inherit"], | ||
| windowsHide: false, | ||
| }); | ||
|
|
||
| let stdoutData = ""; | ||
| child.stdout.on("data", (chunk) => { | ||
| stdoutData += chunk.toString("utf8"); | ||
| }); | ||
|
|
||
| child.on("error", (error) => { | ||
| console.error(\`Failed to start \${executable}: \${error.message}\`); | ||
| process.exit(1); | ||
| }); | ||
|
|
||
| for (const signal of ["SIGINT", "SIGTERM"]) { | ||
| process.on(signal, () => { | ||
| child.kill(signal); | ||
| }); | ||
| } | ||
|
|
||
| child.on("exit", (code, signal) => { | ||
| if (stdoutData) { | ||
| try { | ||
| const parsed = JSON.parse(stdoutData.trim()); | ||
| const processed = processOutputJson(parsed, rawImage); | ||
| console.log(JSON.stringify(processed, null, 2)); | ||
| } catch (e) { | ||
| process.stdout.write(stdoutData); | ||
| } | ||
| } | ||
| if (signal) process.exit(1); | ||
| process.exit(code ?? 0); | ||
| }); | ||
| } else { | ||
| spawnAndExit(executable, args); | ||
| } | ||
| } | ||
|
|
||
| function runEnhancedMcp(resolveNativeExecutable, rawArgs) { | ||
| const executable = resolveNativeExecutable(); | ||
| const child = spawn(executable, rawArgs, { | ||
| stdio: ["pipe", "pipe", "inherit"], | ||
| windowsHide: false, | ||
| }); | ||
|
|
||
| child.on("error", (error) => { | ||
| console.error(\`Failed to start \${executable}: \${error.message}\`); | ||
| process.exit(1); | ||
| }); | ||
|
|
||
| for (const signal of ["SIGINT", "SIGTERM"]) { | ||
| process.on(signal, () => { | ||
| child.kill(signal); | ||
| }); | ||
| } | ||
|
|
||
| child.on("exit", (code, signal) => { | ||
| if (signal) process.exit(1); | ||
| process.exit(code ?? 0); | ||
| }); | ||
|
|
||
| let stdinBuffer = ""; | ||
| process.stdin.on("data", (chunk) => { | ||
|
Contributor
There was a problem hiding this comment. When the client closes stdin, the launcher never passes that on, so the native MCP server never sees EOF and the launcher never exits. With one request piped in, the launcher was still running after 5 s. The fake native server logged |
||
| stdinBuffer += chunk.toString("utf8"); | ||
| let newlineIdx; | ||
| while ((newlineIdx = stdinBuffer.indexOf("\\n")) !== -1) { | ||
| const line = stdinBuffer.slice(0, newlineIdx).trim(); | ||
| stdinBuffer = stdinBuffer.slice(newlineIdx + 1); | ||
| if (line) { | ||
| try { | ||
| const req = JSON.parse(line); | ||
| if (req && req.method === "tools/call" && req.params) { | ||
| const toolName = req.params.name; | ||
| if (req.params.arguments) { | ||
| req.params.arguments = normalizeToolArgs(toolName, req.params.arguments, resolveNativeExecutable); | ||
| } | ||
| } | ||
| child.stdin.write(JSON.stringify(req) + "\\n"); | ||
| } catch (e) { | ||
| child.stdin.write(line + "\\n"); | ||
| } | ||
| } | ||
| } | ||
| }); | ||
|
|
||
| child.stdout.pipe(process.stdout); | ||
| } | ||
|
|
||
| if (command === "-h" || command === "--help" || (command === "help" && args.length <= 1)) { | ||
| printLauncherHelp(); | ||
| process.exit(0); | ||
|
|
@@ -374,6 +642,10 @@ if (command === "help" && (args[1] === "install-claude-mcp" || args[1] === "inst | |
| if (installCommands.has(command)) { | ||
| const scriptName = installCommands.get(command); | ||
| runInstallCommand(scriptName, args.slice(1)); | ||
| } else if (command === "call") { | ||
| runEnhancedCall(resolveNativeExecutable, args); | ||
| } else if (command === "mcp") { | ||
| runEnhancedMcp(resolveNativeExecutable, args); | ||
| } else { | ||
| spawnAndExit(resolveNativeExecutable(), args); | ||
| } | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The index comes from a separate
snapshotprocess and is then used in another process. In MCP mode, the synchronousexecFileSyncalso blocks the stdin handler for up to 5 s per click.