Repository navigation
fix(computer-use): warn when a Cmd shortcut goes to a background macOS app - #34778
unstablebrainiac merged 1 commit into
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS This only adds a caveat to the reply text: Notes
[DESIGN-REVIEWED] a9d89d3 Description read[DESCRIPTION-READ] d081fc3a6026673fd7d80808a4a011280a4b11a5e516131a6d149269f684d2bc That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS One note, nothing blocking: the fix adds a second way of asking "is this pid frontmost?" when the macOS driver already has one. Not checked: whether Notes
What this change shipsInventory (5 items) — 4 justifiedIntent: a model driving macOS learns that a Cmd shortcut sent to a background app may have done nothing, instead of reading a bare "sent" as success. FIX (#34772).
[FIRST-PRINCIPLES-REVIEWED] a9d89d3 Description read[DESCRIPTION-READ] 9a7179fb3a0a84193891a6363e5f709be932871d9d55e4324e1299805686a3fe That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
bolichen97
left a comment
There was a problem hiding this comment.
Thanks for measuring this on a real app. I read the whole diff, tests included, and press_key on main.
One change I would like before this merges, on _refuse_unless_frontmost.
On main, press_key posts the key with CGEventPostToPid to the target app's pid, so a Cmd shortcut never reaches a different (front) app. The failure in the issue is that the key can be ignored while the answer says "sent". That is a wrong message, not a misdelivery. This change adds a new refusal on a path that works today: every Cmd shortcut to a background app is now blocked, and so is the case where AXFrontmost cannot be read. The measurement covers Cmd+A in TextEdit, so other apps and shortcuts that do act on a background app would be blocked too.
Please keep today's behavior of sending the key, and fix the answer instead. For example, when the app is not frontmost, return ok=True with a text that says the Cmd shortcut was sent but macOS may ignore it for a background app, and name the element routes. If a hard refusal is still wanted, put it behind a setting that is off by default.
Everything else looks fine: the docs and skill table match the code, and the tests are clear. The test updates that set AXFrontmost would need to follow the change.
I did not run the tests locally.
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: fix (5 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: the macOS driver refuses a Cmd shortcut to a background app instead of reporting it sent; the diff adds that one refusal, its skill-table row and tests. Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/system-specs/modules/computer-use.md.
cc257cc to
58de489
Compare
|
Agreed. 58de489 keeps today's behaviour of sending the key and fixes the reply instead: when the app is not frontmost, or There is no hard-refusal setting: nothing needs one now, so it can come later if a case turns up. The tests, the skill table row, the spec paragraph and #34772 all describe the new behaviour. |
58de489 to
a31da4c
Compare
a31da4c to
a9d89d3
Compare
Problem / Motivation
Goal: On macOS,
computer_press_keytells the truth about a Cmd shortcut sent to an app that is not in front, instead of answering a bare "sent" for a key that may have done nothing.macOS delivers a Cmd shortcut to the frontmost app's menu bar and key window. With TextEdit in the background and iTerm2 in front,
computer_press_key(app="TextEdit", key="cmd+a")answeredsent cmd+aand the selection stayed at (0, 0). Pressing Edit > Select All throughAXPressdid nothing either. The same Cmd+A selected all 294 characters once TextEdit was frontmost.Why it matters
A model told "sent cmd+a" believes the text is selected and acts on that. The next step (a copy, a delete, a typed replacement) then runs against a selection that does not exist, with no signal that anything went wrong.
Not a goal
What changed (motivation → approach → change)
A Cmd shortcut to a background macOS app is still sent, and the reply now says macOS may have ignored it.
sent cmd+a, unchangedsent cmd+a, but 'TextEdit' is not the frontmost app and macOS may ignore ...After posting a key that carries the Cmd modifier, the macOS driver (
computer_use/macos_driver.py) reads the app'sAXFrontmostattribute. When it is nottrue, the reply keepsok=Trueand adds the caveat: check the refreshed state, and if nothing changed usecomputer_click,computer_set_valueorcomputer_perform_actionon an element, or ask the user to bring the app forward.The computer-use skill's reply table (
builtin_skills/computer-use/SKILL.md) gets a row for the caveat, and the Computer Use spec (docs/system-specs/modules/computer-use.md) records the measurement and the rule.Backwards compatibility
Compatible: every key is posted exactly as before, and every reply that was
ok=Truestill is. Only the text of a Cmd shortcut's reply to a background app gains a caveat.Removes nothing: no key, field, path or capability is taken away.
Tests
test/test_computer_use_snapshot_macos.pyadds four tests: a Cmd shortcut to a background app is posted and its reply carries the caveat; an unreadable frontmost state gets the caveat too; a Cmd shortcut to the frontmost app answers a plainsent cmd+a; a plain key to a background app is posted with no caveat.Proof: PROVEN test/test_computer_use_snapshot_macos.py::TestMacOSDriverHappyPaths::test_cmd_shortcut_to_a_background_app_is_sent_with_a_caveat, test/test_computer_use_snapshot_macos.py::TestMacOSDriverHappyPaths::test_cmd_shortcut_counts_an_unreadable_frontmost_state_as_background (head 58de489)
Manual verification
Measured on a Mac before the change, with TextEdit in the background and iTerm2 frontmost:
press_key cmd+aansweredsent cmd+aand the selection stayed (0, 0); Edit > Select All viaAXPressdid nothing; settingAXSelectedTextRangedirectly worked in the background; with TextEdit frontmost, Cmd+A selected all 294 characters. The caveat itself is covered by the unit tests above against the fake accessibility framework; a live run on a Mac is still to do.Related Issues
Closes #34772
Sibling sweep
Swept: git grep -nE "post_key(|FLAG_COMMAND" -- src/
Sibling: src/kiro_crew/computer_use/windows_driver.py -- not affected -- its press_key takes keyboard focus before sending the chord
Pattern harvest
Not generalizable: the defect is one macOS delivery rule (Cmd shortcuts go to the frontmost app), and the one driver that posts keys to a pid is fixed here.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)