Skip to content

fix(computer-use): warn when a Cmd shortcut goes to a background macOS app - #34778

Merged
unstablebrainiac merged 1 commit into
mainfrom
unstablebrainiac/fix/macos-cmd-shortcut-background-app
Nov 15, 2026
Merged

unstablebrainiac merged 1 commit into
mainfrom
unstablebrainiac/fix/macos-cmd-shortcut-background-app

Conversation

@unstablebrainiac

@unstablebrainiac unstablebrainiac commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: On macOS, computer_press_key tells the truth about a Cmd shortcut sent to an app that is not in front, instead of answering a bare "sent" for a key that may have done nothing.

macOS delivers a Cmd shortcut to the frontmost app's menu bar and key window. With TextEdit in the background and iTerm2 in front, computer_press_key(app="TextEdit", key="cmd+a") answered sent cmd+a and the selection stayed at (0, 0). Pressing Edit > Select All through AXPress did nothing either. The same Cmd+A selected all 294 characters once TextEdit was frontmost.

Why it matters

A model told "sent cmd+a" believes the text is selected and acts on that. The next step (a copy, a delete, a typed replacement) then runs against a selection that does not exist, with no signal that anything went wrong.

Not a goal

  • Refusing the key. It goes to the target app's pid, so it never lands in a different app, and only Cmd+A in TextEdit was measured: other apps and shortcuts may act on a background key.
  • Bringing the target app to the front before the key. That takes the user's keyboard focus and needs its own opt-in.
  • Plain keys (Return, Tab, letters without Cmd), and Windows, whose driver already takes keyboard focus before a key press.

What changed (motivation → approach → change)

A Cmd shortcut to a background macOS app is still sent, and the reply now says macOS may have ignored it.

Key Target app Reply
has Cmd frontmost sent cmd+a, unchanged
has Cmd background, or frontmost state unreadable sent cmd+a, but 'TextEdit' is not the frontmost app and macOS may ignore ...
no Cmd any unchanged

After posting a key that carries the Cmd modifier, the macOS driver (computer_use/macos_driver.py) reads the app's AXFrontmost attribute. When it is not true, the reply keeps ok=True and adds the caveat: check the refreshed state, and if nothing changed use computer_click, computer_set_value or computer_perform_action on an element, or ask the user to bring the app forward.

The computer-use skill's reply table (builtin_skills/computer-use/SKILL.md) gets a row for the caveat, and the Computer Use spec (docs/system-specs/modules/computer-use.md) records the measurement and the rule.

Backwards compatibility

Compatible: every key is posted exactly as before, and every reply that was ok=True still is. Only the text of a Cmd shortcut's reply to a background app gains a caveat.

Removes nothing: no key, field, path or capability is taken away.

Tests

test/test_computer_use_snapshot_macos.py adds four tests: a Cmd shortcut to a background app is posted and its reply carries the caveat; an unreadable frontmost state gets the caveat too; a Cmd shortcut to the frontmost app answers a plain sent cmd+a; a plain key to a background app is posted with no caveat.

Proof: PROVEN test/test_computer_use_snapshot_macos.py::TestMacOSDriverHappyPaths::test_cmd_shortcut_to_a_background_app_is_sent_with_a_caveat, test/test_computer_use_snapshot_macos.py::TestMacOSDriverHappyPaths::test_cmd_shortcut_counts_an_unreadable_frontmost_state_as_background (head 58de489)

Manual verification

Measured on a Mac before the change, with TextEdit in the background and iTerm2 frontmost: press_key cmd+a answered sent cmd+a and the selection stayed (0, 0); Edit > Select All via AXPress did nothing; setting AXSelectedTextRange directly worked in the background; with TextEdit frontmost, Cmd+A selected all 294 characters. The caveat itself is covered by the unit tests above against the fake accessibility framework; a live run on a Mac is still to do.

Related Issues

Closes #34772

Sibling sweep

Swept: git grep -nE "post_key(|FLAG_COMMAND" -- src/
Sibling: src/kiro_crew/computer_use/windows_driver.py -- not affected -- its press_key takes keyboard focus before sending the chord

Pattern harvest

Not generalizable: the defect is one macOS delivery rule (Cmd shortcuts go to the frontmost app), and the one driver that posts keys to a pid is fixed here.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 10, 2026
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision a9d89d34ef84c865c87e1b4903781ef404c9050f touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of a9d89d34ef84c865c87e1b4903781ef404c9050f — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

This only adds a caveat to the reply text: ok, key delivery and every other reply stay as they were, and a bad caveat is undone by a clean revert.

Notes

  • The live Mac run is still to do. Do it before merge: if AXFrontmost reads wrong on a real app (absent or never true), every Cmd shortcut gets a false "may ignore" caveat. The skill's "Do not retry the key" instruction would then push the model off keys that actually work.

[DESIGN-REVIEWED] a9d89d3

Description read

[DESCRIPTION-READ] d081fc3a6026673fd7d80808a4a011280a4b11a5e516131a6d149269f684d2bc

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of a9d89d34ef84c865c87e1b4903781ef404c9050f — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

One note, nothing blocking: the fix adds a second way of asking "is this pid frontmost?" when the macOS driver already has one.

Not checked: whether AXFrontmost and the CG window-list front owner give the same answer for every app on macOS.

Notes

  • Item 5 — duplicate of src/kiro_crew/computer_use/macos_skylight.py:633 (_is_frontmost(pid)). Grep for frontmost across computer_use/ found 1 existing pid-frontmost probe. The diff adds a second one, macos_ffi.ax_bool_opt(app_elem, AX_FRONTMOST) is True. Both treat a failed read as background, so the two can drift apart. Subtraction: delete AX_FRONTMOST and the ax_application block in _background_shortcut_note, and call the existing _is_frontmost(app.pid) instead.

What this change ships

Inventory (5 items) — 4 justified

Intent: a model driving macOS learns that a Cmd shortcut sent to a background app may have done nothing, instead of reading a bare "sent" as success. FIX (#34772).

  1. A Cmd shortcut to a background macOS app now replies "sent …, but 'X' is not the frontmost app…" — justified
  2. When the app's frontmost state can't be read, the reply carries the same caveat — justified
  3. The computer-use skill's reply table gains a row for the caveat — justified
  4. The Computer Use spec records the measurement and the rule — justified
  5. A new AXFrontmost frontmost probe in the macOS driver — duplicate of macos_skylight._is_frontmost

[FIRST-PRINCIPLES-REVIEWED] a9d89d3

Description read

[DESCRIPTION-READ] 9a7179fb3a0a84193891a6363e5f709be932871d9d55e4324e1299805686a3fe

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed a9d89d34ef84c865c87e1b4903781ef404c9050f — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] a9d89d3

Verdict parsed from the review's SHA-scoped output markers for commit a9d89d34ef84c865c87e1b4903781ef404c9050f.

False positive or not applicable? A repository writer can comment:
/ai-review override fable a9d89d34ef84c865c87e1b4903781ef404c9050f false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of a9d89d34ef84c865c87e1b4903781ef404c9050f and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] a9d89d3

False positive or not applicable? A repository writer can comment:
/ai-review override gpt a9d89d34ef84c865c87e1b4903781ef404c9050f false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 10, 2026
@unstablebrainiac
unstablebrainiac enabled auto-merge (squash) November 10, 2026 21:38

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for measuring this on a real app. I read the whole diff, tests included, and press_key on main.

One change I would like before this merges, on _refuse_unless_frontmost.

On main, press_key posts the key with CGEventPostToPid to the target app's pid, so a Cmd shortcut never reaches a different (front) app. The failure in the issue is that the key can be ignored while the answer says "sent". That is a wrong message, not a misdelivery. This change adds a new refusal on a path that works today: every Cmd shortcut to a background app is now blocked, and so is the case where AXFrontmost cannot be read. The measurement covers Cmd+A in TextEdit, so other apps and shortcuts that do act on a background app would be blocked too.

Please keep today's behavior of sending the key, and fix the answer instead. For example, when the app is not frontmost, return ok=True with a text that says the Cmd shortcut was sent but macOS may ignore it for a background app, and name the element routes. If a hard refusal is still wanted, put it behind a setting that is off by default.

Everything else looks fine: the docs and skill table match the code, and the tests are clear. The test updates that set AXFrontmost would need to follow the change.

I did not run the tests locally.

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (5 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: the macOS driver refuses a Cmd shortcut to a background app instead of reporting it sent; the diff adds that one refusal, its skill-table row and tests. Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/system-specs/modules/computer-use.md.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Nov 10, 2026
@unstablebrainiac unstablebrainiac changed the title fix(computer-use): refuse a Cmd shortcut sent to a background macOS app fix(computer-use): warn when a Cmd shortcut goes to a background macOS app Nov 11, 2026
@unstablebrainiac
unstablebrainiac force-pushed the unstablebrainiac/fix/macos-cmd-shortcut-background-app branch from cc257cc to 58de489 Compare November 11, 2026 20:36
@unstablebrainiac

Copy link
Copy Markdown
Contributor Author

Agreed. 58de489 keeps today's behaviour of sending the key and fixes the reply instead: when the app is not frontmost, or AXFrontmost cannot be read, the reply stays ok=True and says macOS may ignore a Cmd shortcut sent to a background app, then names the element routes. A frontmost app and plain keys get the same sent <key> as before.

There is no hard-refusal setting: nothing needs one now, so it can come later if a case turns up. The tests, the skill table row, the spec paragraph and #34772 all describe the new behaviour.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 11, 2026
@unstablebrainiac
unstablebrainiac force-pushed the unstablebrainiac/fix/macos-cmd-shortcut-background-app branch from 58de489 to a31da4c Compare November 15, 2026 07:33
@unstablebrainiac

Copy link
Copy Markdown
Contributor Author

Backend Tests (3.12, 6) red on a31da4c is main's: test/metrics/test_kiro_export_destination.py::TestOffOffer (3 tests) fails on a clean origin/main (c844e1e) since #37463. Fix in flight: #37545. This PR rebases onto it once it merges.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 15, 2026
@unstablebrainiac
unstablebrainiac force-pushed the unstablebrainiac/fix/macos-cmd-shortcut-background-app branch from a31da4c to a9d89d3 Compare November 15, 2026 08:33
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Nov 15, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 15, 2026
@unstablebrainiac
unstablebrainiac merged commit 5c52b2c into main Nov 15, 2026
97 checks passed
@unstablebrainiac
unstablebrainiac deleted the unstablebrainiac/fix/macos-cmd-shortcut-background-app branch November 15, 2026 10:02
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Computer Use on macOS reports a Cmd shortcut to a background app as a plain success, but it may do nothing

3 participants