Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 54 additions & 0 deletions DECISIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -19609,3 +19609,57 @@ process. It never removes the file.
own handler (`on_shortcut`). Quick entry used to build the plugin with a
single handler, which would have left no room for a second shortcut.

## 2026-11-03 — TD-5226: the ring marks the acted-on display, in full control only (Class B; revises TD-3402 and TD-3407)

**Decision:** The real-display ring means "an agent has the real screen".
tst-cu-mcp now applies its rules in one platform-neutral controller
(`overlay/control.py`) in front of the painters, which only light, darken
and aim.

1. **Full control only.** The ring shows only while the policy file's
`mode` is `full_control`. In `background` mode it is dark on every
display, even if a background-mode agent falls back to a coordinate
tool. The file is re-parsed when its size or mtime changes, so Settings
applies to every running client. A file that fails to parse reads as
background: a broken edit darkens the ring rather than lighting it. With
no file at all a client uses its installed config, whose default is also
background.
2. **One display.** The ring follows the display being acted on: the last
screenshot's display, or the display under the last pointer action
(move, click, and the scroll that moves first). The macOS helper keeps a
panel per screen and lights only the target's, named by its
CGDirectDisplayID (`target <id>`); an unknown id or no target yet means
the primary display. Windows and Linux X11 still ring every display;
aiming them is a follow-up.
3. **Episodes per session.** `overlay_session` takes the session id. The
ring holds while any session's episode is open, so one TST Desk session
ending no longer darkens another's.
4. **Non-bracketing clients fade.** A client that never sends
`overlay_session` lights the ring on each action, and it fades 30
seconds after the last one. A client that has bracketed once never falls
back to this rule, so TST Desk's own ring still lasts exactly as long as
the episode.
5. **Stop is watched, not waited for.** A watcher thread re-checks Stop,
the mode and the fade every quarter second. Stop fades the ring out (the
helper's normal hide, under a second) and drops a non-bracketing
client's claim, so after Resume it takes a new action to light it. TST
Desk's own episodes are closed by Stop already (TD-3407).
6. **The copy that acts paints.** Grok's own tst-cu-mcp child always paints
(with the user's `show_on_real_display` switch passed through), and Grok
sessions tag their episodes for the window only, so the daemon's sidecar
no longer lights a second ring on a display it is not acting on. Before
this the child was told not to paint when a command was configured,
because nothing ended its ring; rule 4 now does.

**Rationale:** The user works across several displays and could not tell
which one an agent was on. In background mode nothing takes the screen, so
a ring there signals something that is not happening. Level-triggered state
(work out "wanted" on every call and tick, talk to the painter only on a
change) keeps the watcher and the tool thread from fighting, under one lock.

**Rejected:** ringing every display with the acted-on one brighter (still
says "everything is taken" in background mode, and two weights of ring read
as two agents); hiding the ring after a fixed time inside an episode (the
TD-3407 reason stands: a long think between actions is still the agent
driving); a stop-file watch inside the AppKit helper (it would need the
config path and the mode as well; the sidecar already has both).
2 changes: 1 addition & 1 deletion core/tests/test_cu_kill.py
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ async def test_set_cu_kill_emits_cu_session_false(tmp_path: Path) -> None:
assert session.cu_session_active is False
tags = [e for e in session.event_log.all_events if isinstance(e, CuSession)]
assert tags[-1].active is False
assert ("overlay_session", {"active": False}) in mock.calls
assert ("overlay_session", {"active": False, "session": sid}) in mock.calls
runner = daemon.session_registry.get_runner(sid)
if runner is not None:
await runner.cancel()
Expand Down
20 changes: 17 additions & 3 deletions core/tests/test_cu_session.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ async def test_first_cu_tool_opens_once_and_overlay_is_told() -> None:
tags = [e for e in session.event_log.all_events if isinstance(e, CuSession)]
assert len(tags) == 1
assert tags[0].active is True
assert driver.calls == [("overlay_session", {"active": True})]
assert driver.calls == [("overlay_session", {"active": True, "session": session.id})]


@pytest.mark.asyncio
Expand All @@ -44,11 +44,25 @@ async def test_close_emits_false_and_is_idempotent() -> None:
tags = [e for e in session.event_log.all_events if isinstance(e, CuSession)]
assert [t.active for t in tags] == [True, False]
assert driver.calls == [
("overlay_session", {"active": True}),
("overlay_session", {"active": False}),
("overlay_session", {"active": True, "session": session.id}),
("overlay_session", {"active": False, "session": session.id}),
]


@pytest.mark.asyncio
async def test_window_only_episode_leaves_the_ring_to_the_acting_copy() -> None:
"""Grok's own tst-cu-mcp paints its ring; the sidecar must not light one too."""
session = Session("/tmp/ws")
driver = MockDesktopDriver()
session._overlay_session = driver.set_overlay_session
await session.open_cu_session("computer-use__click", ring=False)
await session.close_cu_session()

tags = [e for e in session.event_log.all_events if isinstance(e, CuSession)]
assert [t.active for t in tags] == [True, False]
assert driver.calls == []


@pytest.mark.asyncio
async def test_cancel_closes_the_episode() -> None:
session = Session("/tmp/ws")
Expand Down
28 changes: 23 additions & 5 deletions core/tests/test_grok_home.py
Original file line number Diff line number Diff line change
Expand Up @@ -125,9 +125,27 @@ def test_computer_use_mcp_from_command(tmp_path: Path, monkeypatch: pytest.Monke
assert spec is not None
assert spec["name"] == "computer-use"
assert spec["command"] == str(binary.resolve())
# A configured command is the daemon's driver too: it paints the ring
# from cu_session tags, so this child is told not to.
assert spec["env"] == [{"name": "TST_CU_MCP_OVERLAY", "value": "0"}]
# The child acts, so it paints its own ring (TD-5226), following the
# user's show_on_real_display switch.
assert spec["env"] == [{"name": "TST_CU_MCP_OVERLAY", "value": "1"}]


def test_computer_use_mcp_follows_the_real_display_switch(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
from tstd.cu_indicators import CuIndicatorPrefs, reset_cu_indicators, set_current_prefs

monkeypatch.delenv("TST_CU_AGENT_SOCK", raising=False)
binary = tmp_path / "tst-cu-mcp"
binary.write_text("#!/bin/sh\n", encoding="utf-8")
binary.chmod(0o755)
set_current_prefs(CuIndicatorPrefs(show_on_real_display=False))
try:
spec = computer_use_mcp(str(binary))
finally:
reset_cu_indicators()
assert spec is not None
assert {"name": "TST_CU_MCP_OVERLAY", "value": "0"} in spec["env"]


def test_acp_mcp_servers_merges_computer_use(
Expand Down Expand Up @@ -165,8 +183,8 @@ def test_computer_use_mcp_finds_checkout_from_workspace(
spec = computer_use_mcp("", search_from=str(tmp_path / "Documents"))
assert spec is not None
assert spec["command"] == str(binary.resolve())
# No configured driver means nobody else paints; this child keeps the ring.
assert {"name": "TST_CU_MCP_OVERLAY", "value": "0"} not in spec["env"]
# Found on disk: same rule, the child paints its own ring.
assert {"name": "TST_CU_MCP_OVERLAY", "value": "1"} in spec["env"]


def test_open_in_terminal_rejects_control_chars() -> None:
Expand Down
8 changes: 3 additions & 5 deletions core/tests/test_grok_loop.py
Original file line number Diff line number Diff line change
Expand Up @@ -195,10 +195,8 @@ async def prompt_with_cu(
assert types.index("cu_session") == types.index("tool_call", types.index("tool_call") + 1) + 1
# Closed at turn end, not at the last tool.
assert types[types.index("turn_complete") + 1] == "cu_session"
assert driver.calls == [
("overlay_session", {"active": True}),
("overlay_session", {"active": False}),
]
# The ring is Grok's own child's (TD-5226): the daemon's sidecar stays dark.
assert driver.calls == []
await session.cancel()
await asyncio.wait_for(task, timeout=2)

Expand Down Expand Up @@ -237,7 +235,7 @@ async def prompt_then_die(
assert done and done[0].failed is True
tags = [e for e in session.event_log.events_from(1) if isinstance(e, CuSession)]
assert [t.active for t in tags] == [True, False]
assert driver.calls[-1] == ("overlay_session", {"active": False})
assert driver.calls == []
await session.cancel()
await asyncio.wait_for(task, timeout=2)

Expand Down
6 changes: 3 additions & 3 deletions core/tstd/desktop/inprocess.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,15 +50,15 @@ def set_killed(self, killed: bool) -> None:
else:
os.environ.pop(_STOP_ENV, None)

async def set_overlay_session(self, active: bool) -> None:
async def set_overlay_session(self, active: bool, session: str = "") -> None:
try:
from tst_cu_mcp.overlay import get_overlay

overlay = get_overlay()
if active:
overlay.begin_session()
overlay.begin_session(session)
else:
overlay.end_session()
overlay.end_session(session)
except Exception:
return

Expand Down
8 changes: 5 additions & 3 deletions core/tstd/desktop/mcp_driver.py
Original file line number Diff line number Diff line change
Expand Up @@ -50,10 +50,12 @@ def __init__(
def set_killed(self, killed: bool) -> None:
self.killed = bool(killed)

async def set_overlay_session(self, active: bool) -> None:
"""Tell the sidecar the computer-use episode opened or closed."""
async def set_overlay_session(self, active: bool, session: str = "") -> None:
"""Tell the sidecar one session's computer-use episode opened or closed."""
try:
await self._client.call_tool("overlay_session", {"active": bool(active)})
await self._client.call_tool(
"overlay_session", {"active": bool(active), "session": session}
)
except Exception:
return

Expand Down
4 changes: 2 additions & 2 deletions core/tstd/desktop/mock.py
Original file line number Diff line number Diff line change
Expand Up @@ -60,8 +60,8 @@ def __init__(
def set_killed(self, killed: bool) -> None:
self.killed = bool(killed)

async def set_overlay_session(self, active: bool) -> None:
self.calls.append(("overlay_session", {"active": bool(active)}))
async def set_overlay_session(self, active: bool, session: str = "") -> None:
self.calls.append(("overlay_session", {"active": bool(active), "session": session}))

def _refuse_if_wayland(self) -> None:
if normalize_cu_platform(self.platform) != "linux":
Expand Down
4 changes: 2 additions & 2 deletions core/tstd/desktop/protocol.py
Original file line number Diff line number Diff line change
Expand Up @@ -79,8 +79,8 @@ class DesktopDriver(Protocol):
def set_killed(self, killed: bool) -> None:
"""Engage or clear the kill-switch. Screenshot still runs."""

async def set_overlay_session(self, active: bool) -> None:
"""Open or close the real-display computer-use ring (TD-3407)."""
async def set_overlay_session(self, active: bool, session: str = "") -> None:
"""Open or close one session's real-display ring episode (TD-3407, TD-5226)."""

async def screenshot(self, display: int | None = None) -> str:
"""Return a JSON object with ``png_base64`` (and size if known)."""
Expand Down
17 changes: 8 additions & 9 deletions core/tstd/grok_home.py
Original file line number Diff line number Diff line change
Expand Up @@ -235,9 +235,6 @@ def computer_use_mcp(
import shlex

argv = shlex.split(cu_command)
# A configured command is also the daemon's own driver, which paints
# the real-display ring from cu_session tags; this child must not.
configured = bool(argv)
if not argv:
from .desktop.factory import packaged_cu_argv

Expand All @@ -260,11 +257,11 @@ def computer_use_mcp(
"name": "computer-use",
"command": str(binary.resolve()),
"args": argv[1:],
"env": _host_env(argv, paint=not configured),
"env": _host_env(argv),
}


def _host_env(argv: list[str], *, paint: bool = True) -> list[dict[str, str]]:
def _host_env(argv: list[str]) -> list[dict[str, str]]:
env: list[dict[str, str]] = []
from .cu_host import SOCK_ENV

Expand All @@ -273,10 +270,12 @@ def _host_env(argv: list[str], *, paint: bool = True) -> list[dict[str, str]]:
env.append({"name": SOCK_ENV, "value": sock})
if argv and Path(argv[0]).name.startswith("tstd") and "--cu-mcp" in argv[1:]:
env.append({"name": "TST_CU_MCP_HOST", "value": "1"})
if not paint:
# Nothing tells this child when a turn ends, so a ring it lit would
# linger; the daemon's driver paints and closes it instead.
env.append({"name": OVERLAY_ENV, "value": "0"})
# This child acts, so it paints the ring on the display it acts on; Grok
# sessions do not light the daemon's sidecar (TD-5226). With no episode
# bracket its ring fades on idle. The user's switch still applies.
from .cu_indicators import current_prefs

env.append({"name": OVERLAY_ENV, "value": "1" if current_prefs().show_on_real_display else "0"})
return env


Expand Down
8 changes: 4 additions & 4 deletions core/tstd/grok_loop.py
Original file line number Diff line number Diff line change
Expand Up @@ -393,10 +393,10 @@ async def on_update(params: dict[str, Any]) -> None:
seq=1,
)
)
# Same episode tags as the native loop (TD-3407): the pane glow
# and the daemon's real-display ring follow cu_session, which
# is the only thing that can close them on this engine.
await session.open_cu_session(name)
# Same episode tags as the native loop (TD-3407) for the pane
# glow. The real-display ring is Grok's own tst-cu-mcp child's:
# it acts, so it knows the display (TD-5226).
await session.open_cu_session(name, ring=False)
after_tool = True
return
if kind == "tool_call_update":
Expand Down
25 changes: 18 additions & 7 deletions core/tstd/session.py
Original file line number Diff line number Diff line change
Expand Up @@ -546,7 +546,10 @@ def __init__(self, workspace_path: str) -> None:
# TD-3407: computer-use episode is open this turn. The overlay
# callback is the sidecar ring; None on a tombstone or mock-only.
self.cu_session_active = False
self._overlay_session: Callable[[bool], Awaitable[None]] | None = None
# (active, session id): the sidecar holds the ring while any
# session's episode is open (TD-5226).
self._overlay_session: Callable[[bool, str], Awaitable[None]] | None = None
self._cu_ring = False
# TD-4101: unattended runs iterate without a user message.
# Interactive sessions leave this false and wait on the queue.
self.autonomy = False
Expand Down Expand Up @@ -609,17 +612,24 @@ def mark_class_c(self, reason: str) -> None:
if self.autonomy_stop_reason is None:
self.autonomy_stop_reason = reason

async def open_cu_session(self, tool_name: str) -> None:
"""Open the computer-use episode on the first CU tool of a turn."""
async def open_cu_session(self, tool_name: str, *, ring: bool = True) -> None:
"""Open the computer-use episode on the first CU tool of a turn.

``ring=False`` tags the episode for the window only: the copy of
tst-cu-mcp that acts paints the real-display ring itself (Grok's own
child does, TD-5226), so the daemon's sidecar must not light a second
one on a display it is not acting on.
"""
from .local_worker import is_cu_tool
from .protocol import CuSession

if self.cu_session_active or not is_cu_tool(tool_name):
return
self.cu_session_active = True
await self.event_log.add(CuSession(session_id=self.id, active=True, seq=1))
if self._overlay_session is not None:
await self._overlay_session(True)
self._cu_ring = ring and self._overlay_session is not None
if self._cu_ring and self._overlay_session is not None:
await self._overlay_session(True, self.id)

async def close_cu_session(self) -> None:
"""Close the computer-use episode. Idempotent."""
Expand All @@ -629,8 +639,9 @@ async def close_cu_session(self) -> None:
return
self.cu_session_active = False
await self.event_log.add(CuSession(session_id=self.id, active=False, seq=1))
if self._overlay_session is not None:
await self._overlay_session(False)
if self._cu_ring and self._overlay_session is not None:
self._cu_ring = False
await self._overlay_session(False, self.id)

async def _observe_turn_end(self, event: DaemonEvent, _log: SessionEventLog) -> None:
"""Lower the open-turn count when the loop reports a turn complete."""
Expand Down
20 changes: 14 additions & 6 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -294,7 +294,8 @@ second (TD-5223).
Settings → Computer use (TD-4830) writes `~/.tst-cu-mcp/config.yaml`
(or `$TST_CU_MCP_CONFIG`): `actuation.enabled`, `mode` (`background` or
`full_control`), `scoping.denied_apps`, `scoping.unhide_on_finish`. The
MCP server re-reads that file on each background tool call.
MCP server re-reads that file on each background tool call, and the real-display
ring follows `mode` whenever the file changes (TD-5226).

Browser computer-use (TD-1710): `browser` selects the in-process mock
(CI, never launches Chrome) or Playwright with a persistent profile
Expand Down Expand Up @@ -3307,11 +3308,18 @@ proposals for the run summary and never accepts them either. The classifier stil
not keys in this file. Settings → Appearance persists them as `cu-indicators.yaml` in the
user data dir (glow and cursor default on; the real-display ring defaults on). Glow and
the agent cursor are drawn on the Screen pane. The rust ring on the real display
follows the same `cu_session` open/close tags (TD-3407) on macOS, Windows, and
Linux X11 — it stays up until the turn ends, you cancel, or the kill-switch
fires, not for eight seconds after the last click. Wayland has no real-display
ring. Screenshots hide the ring for the grab so it cannot appear in the frame.
This is not a second hardware pointer.
means "an agent has the real screen", so it shows only in `full_control` mode
(the `mode` in `~/.tst-cu-mcp/config.yaml`, re-read when it changes); in
`background` mode it stays off on every display (TD-5226). On macOS it lights only
the display being acted on: the one the last screenshot targeted, or the one
under the last pointer action, else the primary display. Windows and Linux X11
still ring every display. For a TST Desk session it follows the `cu_session`
open/close tags (TD-3407) and stays up while any session's episode is open; a
client that never sends them (Grok's own copy, another MCP client) lights it on
each action and it fades 30 seconds after the last one. Stop fades it out within
about a second, without waiting for the agent's next action. Wayland has no real-display ring. The ring's windows are left out
of screenshots, and a grab also hides it, so it cannot appear in the frame. This
is not a second hardware pointer.

**Background**, **Strength**, **Motion** and **Click ripples** (TD-5202) are not keys in this file either.
Settings → Appearance keeps them per device, like the theme, in the window's local storage:
Expand Down
Loading
Loading