Repository navigation
TD-5227: each agent has its own cursor on the real display - #115
Merged
Merged
Conversation
An agent doing computer use had no cursor of its own on the real display:
in full control it moved the user's pointer, and with several agents at
once nothing showed which was where. TD-3402 kept the agent cursor on the
Screen pane only; DECISIONS.md replaces that rule.
Two renderers, one identity per agent:
- tst-cu-mcp's macOS helper draws a click-through, non-shareable cursor
panel per agent (arrow and label in its color) at its last move, click
or acted-on element, fading 10 s after its last action. Reduced motion:
no glide, no click ring. Colors come from a machine-wide flock slot
registry; labels from the MCP client's name.
- The element tools run in one agent-desktop session with its own
multi-agent cursor, each TST Desk session enabled with the same label
and color.
Each TST Desk session is an agent ("TST Desk · <chat title>"): the
session loop sets a context variable, and the MCP driver names the acting
session with the internal overlay_agent tool before each pointer action,
under one lock. The display switches are mirrored into the shared policy
file (overlay.enabled, overlay.cursor) so every client follows them live.
Stop hides every drawn cursor and disables agent-desktop's overlay.
The helper's line protocol moves to overlay/darwin_wire.py so the helper
stays near its size limit.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HUthoCdmzg2hy89ssk2cSF
…op's windows out
Two host pieces for per-agent cursors on the real display.
ui_act frame. A successful ui_act (press, set_value, focus, raise,
show_menu) now replies with "frame": {"x", "y", "width", "height"} in
global top-left points, the same space as CGEvent, CGWindow bounds and the
display bounds, so the caller can show the action where it happened. The
frame comes from AXPosition and AXSize and is read before the action,
because a press can close or move the element. It is best effort: when
either attribute is missing, is not an AXValue of the right type, or holds
a non-finite number, the key is left out and the action still runs.
Following TD-5225, the host checks CFGetTypeID against AXValueGetTypeID
before AXValueGetValue (cu_cf::ax_point / ax_size). The Python fallback
(darwin_ax._local_act) adds the same key through pyobjc.
Captures without agent-desktop's windows. agent-desktop draws its agent
cursor in ordinary, capturable windows, so a plain on-screen capture
showed that cursor to the model, and one agent's cursor in another
agent's screenshot. The host capture (new cu_capture module) and the
Python local capture (new darwin_capture module) now list on-screen
windows (CGWindowListCreate plus CGWindowListCreateDescriptionFromArray)
and, when any is owned by a process named exactly "agent-desktop", build
the image with CGWindowListCreateImageFromArray from the rest, front to
back. With no such window the existing CGWindowListCreateImage call runs
unchanged, and a null from-array image falls back to it. The filter is a
pure function with unit tests on both sides. The new code lives in its own
module so cu_agent.rs gets smaller; capture_png now holds the image as an
Owned reference.
Class A notes: a window whose description is missing (it closed between
the two calls) is kept; owner matching is exact and case-sensitive.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HUthoCdmzg2hy89ssk2cSF
captcrock
force-pushed
the
td/5227-agent-cursors
branch
from
November 3, 2026 16:54
8d01466 to
35b7e18
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Every computer-use agent now has its own drawn cursor on the real display, with its own color and label. DECISIONS.md replaces the TD-3402 rule ("agent cursor on the Screen pane only") and spells out what "separate cursor" means in each mode.
Two renderers, one identity per agent:
NSWindowSharingNone(left out of screenshots), one level above the ring. It goes to the agent's last move, click, or background element action, and fades 10 s after its last action. Reduced Motion: no glide, no click ring. Covers every tst-cu-mcp client: TST Desk sessions, Grok's own copy, other MCP clients.desktop_actetc.): TST Desk runs one agent-desktop session per daemon run (session start --cursor --multi-agent --no-trace) and enables each TST Desk session with the same label and color.Identity and color:
overlay_agenttool before each pointer action, under one lock.clientInfo.name("Claude", "Grok", ...).flockslot registry, so live clients don't share one; a client that exits frees its color.Modes: in background the pointer never moves and the drawn cursor is the only sign of where each agent works (no ring, TD-5226). In full_control the real pointer still moves for each click (there is one system pointer); the drawn cursors keep the agents apart.
Switches and Stop: the daemon mirrors "Agent cursor" and "Show on the real display" into the shared policy file (
overlay.cursor,overlay.enabled); every client re-reads it live. Stop hides every drawn cursor within a quarter second and disables agent-desktop's overlay.Host pieces (second commit):
ui_actreplies carry the element'sframe(type-checked AXValue reads, read before the action), and captures leave agent-desktop's capturable cursor windows out viaCGWindowListCreateImageFromArray(plain capture when none is on screen). Same in the Python fallbacks.Tests (run on macOS; CI does not run tst-cu-mcp)
DarwinBackendhas nohit_test) fail the same on main. New:test_agent_cursors.py,test_overlay_rules.pycursor cases,test_ui_act_frame.py,test_darwin_capture_exclude.py. mypy: the same 4 errors as main, none added.test_cu_agent_cursors.py(identity, announce-under-lock, the agent-desktop session lifecycle, Stop, shared switches); ruff andmypy tstdclean.cargo test --lib148 passed;cargo clippy --all-targets -- -D warningsclean.Live (source build, macOS, four displays)
Two MCP clients ("Claude", "Grok") pressed Calculator buttons in background mode at the same time: two cursor windows at the two buttons' centers, blue and green, pointer unchanged, real screenshots without them, both gone within a second of Stop. The packaged-app check (two TST Desk sessions, agent-desktop's cursor, Stop shortcut) runs after merge against a local build from main.
🤖 Generated with Claude Code
https://claude.ai/code/session_01HUthoCdmzg2hy89ssk2cSF