Repository navigation
fix(macos-input): correct AX capabilities and hit-test coordinates - #27
Merged
Merged
Conversation
Signed-off-by: suyao <sy20010504@gmail.com>
This was referenced Oct 17, 2026
Merged
Merged
6 of 7 tasks
DeJeune
added a commit
to CherryHQ/cherry-studio
that referenced
this pull request
Oct 19, 2026
### What this PR does Before this PR: Cherry Studio pins `@cherrystudio/computer-use` and its six native runtime packages to `0.1.7`. After this PR: The SDK and all native runtime packages resolve to `0.1.9`. The exact-version release-age exceptions are updated alongside the manifest and lockfile. Fixes # N/A ### Why we need it and why it was done in this way Adopt upstream Windows/Linux installed-application launch support from `0.1.8` and macOS input capability, preflight rejection, and accessibility hit-testing fixes from `0.1.9`. The following tradeoffs were made: Keep exact SDK/runtime version alignment and the existing packaging path; no application integration changes are needed. The following alternatives were considered: Remaining on `0.1.7` would omit these upstream improvements. Links to places where the discussion took place: [Upstream application launch support](CherryHQ/cherry-computer-use#26), [upstream macOS input fixes](CherryHQ/cherry-computer-use#27). ### Agent Note / Spec Agent Note: N/A — mechanical dependency bump with no new repository architecture or ownership decision. Spec PR: N/A — no Spec implementation layer. Acceptance criteria: N/A — dependency maintenance; scoped regression evidence below. ### Verification - `pnpm install` — passed, including lockfile supply-chain checks. - `pnpm change:scope --base origin/main` — confirmed only `package.json`, `pnpm-lock.yaml`, and `pnpm-workspace.yaml` changed. - `pnpm check --plan` — inspected; dependency files trigger conservative full-gate selection. Ran focused checks locally instead. - `pnpm docs:affected --base origin/main` — inspected `docs/contrib/app-upgrade.md`; update-feed documentation is unaffected by this dependency bump. - `pnpm exec oxfmt --check package.json pnpm-workspace.yaml pnpm-lock.yaml` — passed for the two matched files (lockfile is ignored by the formatter). - `git diff --check` — passed. - `pnpm typecheck:node` — passed. - `pnpm test:scripts scripts/__tests__/computer-use-packaging.test.ts scripts/__tests__/computer-use-runtime-packaging.test.ts scripts/__tests__/package-contents.test.ts` — 3 files, 24 tests passed, including standalone ESM/CJS SDK loading and type resolution. - `pnpm test:main src/main/services/__tests__/ComputerUseService.test.ts src/main/services/__tests__/ComputerUseService.control.test.ts src/main/services/automation/__tests__/ComputerAutomationSession.test.ts src/main/ai/mcp/servers/__tests__/computerUse.test.ts src/main/ai/tools/adapters/aiSdk/builtin/__tests__/ComputerUseTools.test.ts` — 5 files, 51 tests passed. ### Breaking changes None expected. No application API or configuration changes. ### Special notes for your reviewer Only the SDK and its six platform packages changed in the lockfile. No computer-use patches exist in this repository. Actual Electron interaction and packaged desktop acceptance were not run; broader validation remains with CI and platform acceptance. ### Checklist This checklist is not enforcing, but it's a reminder of items that could be relevant to every PR. Approvers are expected to review this list. - [x] Branch: This PR targets `main` - [x] PR: The PR description is expressive enough and will help future contributors - [x] Code: [Write code that humans can understand](https://en.wikiquote.org/wiki/Martin_Fowler#code-for-humans) and [Keep it simple](https://en.wikipedia.org/wiki/KISS_principle) - [ ] Refactor: You have [left the code cleaner than you found it (Boy Scout Rule)](https://learning.oreilly.com/library/view/97-things-every/9780596809515/ch08.html) - [x] Upgrade: Impact of this change on upgrade flows was considered and addressed if required - [x] Documentation: A [user-guide update](https://docs.cherry-ai.com) was considered and is present (link) or not required. Check this only when the PR introduces or changes a user-facing feature or behavior. - [x] Self-review: I have reviewed my own code (e.g., via [`/gh-pr-review`](/.claude/skills/gh-pr-review/SKILL.md), `gh pr diff`, or GitHub UI) before requesting review from others ### Release note ```release-note Update Computer Use to 0.1.9, bringing Windows/Linux application launch support and fixes for macOS input capabilities and accessibility hit testing. ``` Signed-off-by: suyao <sy20010504@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
macOS SDK observations advertised
setValuebased only on a text role, so read-only browser address displays appeared writable. Rejecting such a write before input was incorrectly treated as a possible side effect. AX hit testing also scaled window points as screenshot pixels a second time, shifting the tested position for resized screenshots.setValueonly for writable, non-synthetic text controls.effect: none, allowing a fresh observation in the same session. Actual write failures remain uncertain.Validation
After fast-forwarding to current
origin/main(SDK 0.1.7):swift test --filter 'SDKDesktopInputTests|SDKSessionTests|OpenComputerUseKitTests.testSetValueAttributeGate|OpenComputerUseKitTests.testScreenshotPixel'— 26 tests passed.make check-docs— passed.npm run changeset:status— valid patch plan for the SDK and its fixed native package group.git diff --check— passed.Earlier local validation of these fixes: restoring the old capability and hit-test logic caused the new regression tests to fail; a Developer ID signed Dev build succeeded. A standalone AppKit fixture verified AX button input (0 → 1), scaled screenshot coordinate input (1 → 2), and Chinese
setValueinput.Dia acceptance and remaining issue
Dia's display-only address field no longer advertises
setValue. Forcing that call returnsUNSUPPORTED_CAPABILITYwitheffect: none, and the session remains observable. Navigation through the actual writable command-bar field and Chinese input into a page field were verified.Dia click delivery remains unresolved: AX and coordinate clicks can still return completed without changing the test page. This PR fixes capability, error-classification, and coordinate-conversion defects; it does not claim to fix all Dia interactions. Codex's native Computer Use succeeded on the same fixture, but its underlying delivery behavior has not been established here. No automatic action replay or global-input fallback is introduced.
History: macOS input hotfix.
Scope and release
macOS Swift runtime only, plus SDK documentation and a patch changeset. Linux/CuaDriver exploratory work is excluded. There are no protocol schema changes; downstream hosts should continue checking capabilities and observing the result of an action. Publishing the new native runtime and upgrading Cherry Studio's dependency are separate steps.