From 61a93af588fc4097715cb73ee6ffa2c5f27d01b2 Mon Sep 17 00:00:00 2001 From: Edgar Arakelyan Date: Tue, 10 Nov 2026 21:42:18 -0800 Subject: [PATCH 1/7] feat(drivers): the browser computer serves Playwright MCP's browser_* tools, attached over CDP (FD-4955) The page driver installs Node 24.22.2 and Playwright MCP 0.0.84 from pinned downloads and serves its tools, with --caps vision, beside computer_*: a Playwright MCP it runs attached to the browser over a CDP address the browser's relay issues it, a session per caller, started again when it exits or the browser's data reset ends its address, ended with the driver, with none of the driver's environment and its file access confined to ~/Uploads. browser_evaluate and browser_run_code_unsafe are hidden by default, through ForwardedTool.hidden_by_default, which hides a forwarded tool as @tool(hidden_by_default=True) hides one of an env's own. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 2 +- .../agent_env_environment.py | 9 +- .../tests/test_tool_access.py | 29 + src/agent_env/drivers/README.md | 11 + src/agent_env/drivers/envs/page/driver.py | 200 ++- src/agent_env/drivers/envs/page/env.toml | 10 + .../envs/page/install-playwright-mcp.sh | 19 + .../envs/page/playwright-mcp-tools.json | 1131 +++++++++++++++++ .../computer/browser_computer_test.py | 99 +- tst/unit/drivers/page_driver_test.py | 106 +- 10 files changed, 1605 insertions(+), 11 deletions(-) create mode 100755 src/agent_env/drivers/envs/page/install-playwright-mcp.sh create mode 100644 src/agent_env/drivers/envs/page/playwright-mcp-tools.json diff --git a/AGENTS.md b/AGENTS.md index 8d99e82cf..2188ae130 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -139,7 +139,7 @@ and deserialized through a registry. | `agents/` | The `agents` bundle: the agents agent-env ships, one folder each under `agents/agents/`, with a smoke task each under `agents/tasks/`. Its Claude Code, `@local/agentenv-framework/agents/claude-code`, is the default agent. An agent's Python packages are its folder's `requirements.txt`, installed in its image, never in agent-env's own environment. A run that pulls one of its agents reads the whole bundle, so `tst/unit/bundle/shipped_bundles_test.py` parses and checks every shipped bundle. | | `computers/` | The `computers` bundle: the base computers agent-env ships as `computer` artifacts, `ubuntu-24.04` (XFCE on Xvfb, supervisord as the session user, settings as `AGENTENV_SETTING_`), and `omarchy-4.5`, a disk alone built on the composing machine from Omarchy's own VM image (its `[forms.disk]` `source`) by `disk/omarchy/`'s recipe, its services systemd user units of the graphical session that start through `with-seed`, booted by the launcher as a `linux-desktop` guest (`env/envs/computer/utils/omarchy_disk.py`, on `thin_disk.py`, which Windows' disks use too), aliased through `[artifacts]`. It has no tasks either. `disk/linux/` is the Linux disk form's recipe: `boot/`, whose Dockerfile makes a disk form (a GPT disk of an EFI system partition holding one UKI, Ubuntu's kernel with a busybox initramfs whose init reads a NoCloud seed and `switch_root`s into the image's entrypoint, and an ext4 root filesystem that is a container image's, plus `/etc/agentenv/vm/image.env` from its OCI config), `tools/`, the disk tools that make it unprivileged on the builder's own platform (`mkfs.ext4 -d`), and `launcher/`, the QEMU launcher that boots one under KVM. A base's `[forms.disk]` with no image is made from its container form; the composer makes each arch's disk from its composed container, keeping the base disk's EFI system partition, keyed by the container's key, the base disk and the recipe; gate 3 probes it where a provider boots disks natively (`disk_image_platforms()`), and a deploy's first boot with KVM probes one left `unverified`; a deploy naming `form = "disk"` boots it through `create_vm(image=..., env=...)`, its settings in the seed. | | `browsers/` | The `browsers` bundle: the website browsers agent-env ships, today `chromium`, a `computer_app` env (Chrome for Testing at the build Playwright pins, for amd64 and arm64, set up for unattended use by managed policies in `/etc/opt/chrome_for_testing/policies/managed`, opened at boot on its `start_urls`, its profile its data). All of it, its pin, install step, policies and command, is in `envs/chromium/chromium.py`; `tst/integration/computer/chromium_app_test.py` runs it. Like `drivers`, it has no tasks. | -| `drivers/` | The `drivers` bundle: the computer drivers agent-env ships as `computer_driver` envs, the reference driver (X11 on Linux, recording through `x11grab`) aliased `reference`, and trycua's cua-driver aliased `cua-driver` (its daemon from the pinned PyPI wheel, its own instructions, tools and skill resources served to agents through `@server_instructions`, `forwarded_tools` and `@resource`, its tools beside `computer_*`, `computer/v1` on its desktop-scope calls, its skill pack leading the computer skill; `tst/integration/computer/cua_driver_test.py` runs it). It has no tasks: a bundle with neither tasks nor evals is refused unless its `aliases.toml` names something for other bundles to use. The shipped `desktop` example composes it with the `ubuntu-24.04` base into a computer env, which `tst/integration/computer/` deploys, probes, records and has an agent use. | +| `drivers/` | The `drivers` bundle: the computer drivers agent-env ships as `computer_driver` envs, the reference driver (X11 on Linux, recording through `x11grab`) aliased `reference`, the page driver aliased `page`, a browser computer's, whose screen is the browser's active tab and which serves Playwright MCP's `browser_*` tools beside `computer_*` from a Playwright MCP it runs attached to the browser over a CDP address the relay issues it (`browser_evaluate` and `browser_run_code_unsafe` hidden by default), and trycua's cua-driver aliased `cua-driver` (its daemon from the pinned PyPI wheel, its own instructions, tools and skill resources served to agents through `@server_instructions`, `forwarded_tools` and `@resource`, its tools beside `computer_*`, `computer/v1` on its desktop-scope calls, its skill pack leading the computer skill; `tst/integration/computer/cua_driver_test.py` runs it). It has no tasks: a bundle with neither tasks nor evals is refused unless its `aliases.toml` names something for other bundles to use. The shipped `desktop` example composes it with the `ubuntu-24.04` base into a computer env, which `tst/integration/computer/` deploys, probes, records and has an agent use. | | `examples/` | The example bundles agent-env ships (`hello`, and `echo-mcp`, which needs Docker, and `desktop`, a computer env of `ubuntu-24.04` and `reference` with a probe task and an agent task). Tests run them, but they are examples for users, not fixtures. Every shipped bundle is registered under `agent_env.bundles` in `pyproject.toml`, ` = ""` shipping `src///`. | ### Training helpers diff --git a/packages/agentenv-protocol/src/agentenv_protocol/agent_env_environment.py b/packages/agentenv-protocol/src/agentenv_protocol/agent_env_environment.py index 4f5f0f70a..90d50fd51 100644 --- a/packages/agentenv-protocol/src/agentenv_protocol/agent_env_environment.py +++ b/packages/agentenv-protocol/src/agentenv_protocol/agent_env_environment.py @@ -245,12 +245,14 @@ def deco(fn: Callable) -> Callable: class ForwardedTool(NamedTuple): """A tool an environment serves on another server's behalf, as that server lists it: its descriptor, its own schema included, its MCP ``annotations`` and its ``output_schema``; ``call`` takes the call's arguments as keywords - and returns the other server's ``CallToolResult``, which reaches the caller as it is.""" + and returns the other server's ``CallToolResult``, which reaches the caller as it is. ``hidden_by_default`` hides it + as ``@tool(hidden_by_default=True)`` hides a tool of the environment's own.""" tool: EnvironmentTool call: Callable[..., Any] annotations: dict | None = None output_schema: dict | None = None + hidden_by_default: bool = False class _ResourceMarker(NamedTuple): @@ -450,7 +452,8 @@ def _register_forwarded(self, app: Any, descriptor: EnvironmentTool, fn: Callabl fn_metadata=FuncMetadata(arg_model=_passed_through(), output_schema=schema, output_model=_as_it_came() if schema is not None else None), is_async=True, - annotations=_mcp_tool_annotations(forwarded.annotations) if forwarded.annotations is not None else None) + annotations=_mcp_tool_annotations(forwarded.annotations) if forwarded.annotations is not None else None, + meta={HIDDEN_BY_DEFAULT_META_KEY: True} if forwarded.hidden_by_default else None) def _register_resource(self, app: Any, resource: _ServedResource) -> None: """A resource as it is, its description absent when it has none, or a template through FastMCP's own @@ -1283,6 +1286,8 @@ async def call(**arguments: Any) -> Any: return await forwarded.call(**arguments) setattr(call, _FORWARDED_ATTR, forwarded) + if forwarded.hidden_by_default: + setattr(call, _HIDDEN_BY_DEFAULT_ATTR, True) return call diff --git a/packages/agentenv-protocol/tests/test_tool_access.py b/packages/agentenv-protocol/tests/test_tool_access.py index c2d520e27..04ef1a378 100644 --- a/packages/agentenv-protocol/tests/test_tool_access.py +++ b/packages/agentenv-protocol/tests/test_tool_access.py @@ -17,6 +17,7 @@ from mcp.types import CallToolResult, TextContent # noqa: E402 from agentenv_protocol import ( # noqa: E402 + HIDDEN_BY_DEFAULT_META_KEY, ROLE_HEADER, ROLE_META_KEY, AgentEnvEnvironment, @@ -140,6 +141,34 @@ async def click(**arguments): assert agent_tools == ["read", "write"] and refused.isError +@pytest.mark.asyncio +async def test_a_forwarded_tool_hidden_by_default_stays_hidden_through_every_tool_rule_until_a_rule_names_it(): + class Forwarding(_Env): + def forwarded_tools(self): + async def evaluate(**arguments): + return CallToolResult(content=[TextContent(type="text", text="evaluated")]) + return [ForwardedTool(EnvironmentTool(name="evaluate", inputSchema={"type": "object"}), evaluate, + hidden_by_default=True)] + + with anyio.fail_after(30): + async with _served(Forwarding()) as (rest, session): + async with session("agent") as agent: + listed, refused = await _names(agent), await agent.call_tool("evaluate", {}) + await rest.post("/tools/enable", json={"role": "agent", "tools": "*"}) + async with session("agent") as agent: + after_every_tool = await _names(agent) + await rest.post("/tools/enable", json={"role": "agent", "tools": ["evaluate"]}) + async with session("agent") as agent: + shown = {t.name: t for t in (await agent.list_tools()).tools} + called = await agent.call_tool("evaluate", {}) + async with session() as anyone: + anyone_tools = await _names(anyone) + assert listed == after_every_tool == anyone_tools == ["read", "write"] + assert refused.isError and refused.structuredContent["code"] == "permission_denied" + assert sorted(shown) == ["evaluate", "read", "write"] and not called.isError + assert shown["evaluate"].meta == {HIDDEN_BY_DEFAULT_META_KEY: True} and called.content[0].text == "evaluated" + + @pytest.mark.asyncio async def test_everything_off_then_one_tool_on_for_one_role(): with anyio.fail_after(30): diff --git a/src/agent_env/drivers/README.md b/src/agent_env/drivers/README.md index bae467356..0d30899f6 100644 --- a/src/agent_env/drivers/README.md +++ b/src/agent_env/drivers/README.md @@ -21,6 +21,17 @@ a tab that paints nothing shows white. It serves the page group too: `computer_n it. The browser ending on its own fails the computer for good: its ready check is `final`, and its tools answer with why. Name it as `page`, or by its id, `@local/agentenv-framework/drivers/page`. +Beside those it serves Playwright MCP's `browser_*` tools (0.0.84, `--caps vision`, on Node 24), as +`playwright-mcp-tools.json` lists them. Its install lays Node and Playwright MCP's three npm packages out from pinned +downloads, and it runs Playwright MCP itself, attached to the browser over a CDP address the browser's relay issues to +the user `playwright-mcp`, so the guard applies to it as to any CDP client; it never starts a browser of its own. Each +caller gets a session of its own there. It starts on the first call, again once it has exited, and again on a fresh +address after the browser's data is reset; it ends with the driver, gets none of the driver's environment, and its file +access is `~/Uploads`. `browser_evaluate` and `browser_run_code_unsafe`, which run the agent's own code, are hidden by +default: a task's tool access shows them by naming them. To move to another version, update the downloads in +`env.toml` and regenerate the manifest from that version's `tools/list`; the browser computer's integration test checks +the two agree. + A run that names it writes it: its files, what its class declares for each platform it supports, read without `mss`, `python-xlib` or PyAV installed, and how a computer installs it. Its `env.toml`'s `[install.linux]` has a computer env's composer put the folder in `/opt/agentenv/drivers/reference/`, install `requirements.txt`, resolved and diff --git a/src/agent_env/drivers/envs/page/driver.py b/src/agent_env/drivers/envs/page/driver.py index fdbc8ae68..9b140492f 100644 --- a/src/agent_env/drivers/envs/page/driver.py +++ b/src/agent_env/drivers/envs/page/driver.py @@ -1,20 +1,31 @@ """The page driver, for a computer with no display whose one website browser runs alone: its screen is the browser's active tab. A screenshot is the page, and input reaches the page as CDP input events, both through the browser's own server on its socket (``browser/v1``'s ``page/*`` operations), since only that server holds the browser's debugging pipe. -The browser ending on its own is an infrastructure failure, since nothing here launches it again. Needs -``agentenv-framework-protocol[computer]`` where it runs.""" +The browser ending on its own is an infrastructure failure, since nothing here launches it again. Beside its own tools it +serves Playwright MCP's ``browser_*``, from a Playwright MCP it runs attached to the browser over a CDP address the +browser's relay issues to it. Needs ``agentenv-framework-protocol[computer]`` where it runs.""" from __future__ import annotations import asyncio import base64 +import contextlib +import functools +import json +import logging import math +import os +import socket import string +import tempfile from pathlib import Path -from typing import Any, Optional +from typing import Any, Awaitable, Callable, Optional import httpx +from mcp import ClientSession +from mcp.client.streamable_http import streamable_http_client +from mcp.types import CallToolResult, TextContent -from agentenv_protocol import client +from agentenv_protocol import EnvironmentTool, ForwardedTool, ToolContext, client from agentenv_protocol.computer import ( ComputerDriverEnvironment, DeviceUnavailable, @@ -80,6 +91,26 @@ BLANK = b"\xff\xff\xff" #: How long one call to the browser's server may take. CALL_TIMEOUT_S = 30.0 +#: Playwright MCP, as its install lays it out beside the driver, and its tools as 0.0.84 lists them with ``--caps +#: vision``, which an integration test checks against the one installed. +HERE = Path(__file__).resolve().parent +NODE = HERE / "node" / "bin" / "node" +PLAYWRIGHT_MCP = HERE / "node_modules" / "@playwright" / "mcp" / "cli.js" +PLAYWRIGHT_MCP_TOOLS = HERE / "playwright-mcp-tools.json" +#: Playwright MCP's tools that run the agent's own code, in the page and in Playwright MCP's own process: hidden until +#: a task's tool access names them. +HIDDEN_TOOLS = frozenset({"browser_evaluate", "browser_run_code_unsafe"}) +#: Whom the browser issues Playwright MCP's CDP address to, as its record names the connections. +PLAYWRIGHT_MCP_USER = "playwright-mcp" +#: What Playwright MCP's process keeps of the driver's environment, which holds the deploy's tokens and settings. +KEPT_ENVIRONMENT = ("PATH", "HOME", "USER", "LOGNAME", "LANG", "LC_ALL", "TZ", "TMPDIR") +#: Where Playwright MCP writes what it names itself, and its log. +PLAYWRIGHT_MCP_OUTPUT = Path(tempfile.gettempdir()) / "agentenv-playwright-mcp" +#: How long Playwright MCP may take to listen, and one of its tools to answer. +PLAYWRIGHT_MCP_START_S = 30.0 +PLAYWRIGHT_MCP_CALL_TIMEOUT_S = 300.0 + +logger = logging.getLogger(__name__) @computer_driver(name="page", supports=[{"os": {"type": "linux", "display": ["none"]}, "arch": ["amd64", "arm64"]}]) @@ -94,7 +125,8 @@ class PageDriver(ComputerDriverEnvironment): "around it. Its screen is the page's viewport. Go to a URL with computer_navigate, back, forward or " "reload with computer_history, and list, open, switch or close tabs with computer_tabs. Each screenshot " "names the page, and an open dialog, which computer_dialog answers, or file chooser, which " - "computer_upload gives files in ~/Uploads.") + "computer_upload gives files in ~/Uploads. Its browser_* tools, Playwright MCP's, act on the same " + "browser by the page's accessibility snapshot; browser_file_upload takes files in ~/Uploads too.") def __init__(self) -> None: super().__init__() @@ -102,6 +134,25 @@ def __init__(self) -> None: self._pointer = (0.0, 0.0) self._buttons = 0 self._held: list[str] = [] + self._playwright = _PlaywrightMcp(self._playwright_endpoint) + + def forwarded_tools(self) -> list[ForwardedTool]: + """Playwright MCP's tools, as its manifest lists them, each called in the caller's own session there; those + that run the agent's own code hidden until a task's tool access names them.""" + return [ForwardedTool(EnvironmentTool(name=t["name"], description=t["description"], inputSchema=t["inputSchema"]), + functools.partial(self._playwright_call, t["name"]), t.get("annotations"), + t.get("outputSchema"), hidden_by_default=t["name"] in HIDDEN_TOOLS) + for t in json.loads(PLAYWRIGHT_MCP_TOOLS.read_text(encoding="utf-8"))["tools"]] + + async def _playwright_call(self, tool: str, /, **arguments: Any) -> CallToolResult: + return await self._playwright.call(tool, **arguments) + + async def _playwright_endpoint(self) -> str: + """The CDP address the browser issues Playwright MCP, at this driver's own loopback address, where its relay is + reached as any agent reaches it: the same one until the browser's data is reset, which ends it.""" + folder, _ = await self._found() + origin = f"http://127.0.0.1:{os.environ.get('MCP_PORT', '18765')}/svc/{folder.name}" + return (await self._call("cdp", {"user": PLAYWRIGHT_MCP_USER, "origin": origin}))["ws_url"] @display async def viewport(self) -> Display: @@ -383,5 +434,144 @@ def _steps(path: list) -> list[tuple[float, float]]: return steps +class _PlaywrightMcp: + """Playwright MCP, attached to the browser over the CDP address ``endpoint`` gives, and its sessions there, one per + caller of the driver's tools, so each agent keeps its own snapshots and tab. It starts on the first call, again once + it has exited, and again on a fresh address once the browser's data reset has ended the last; it ends with the + driver, so a driver started again finds none of its predecessor's.""" + + def __init__(self, endpoint: Callable[[], Awaitable[str]]) -> None: + self._endpoint = endpoint + self._lock = asyncio.Lock() + self._process: Optional[asyncio.subprocess.Process] = None + self._running_on: Optional[str] = None + self._url = "" + self._sessions: dict[Optional[str], _Session] = {} + + async def call(self, tool: str, /, **arguments: Any) -> CallToolResult: + """``tool`` called on Playwright MCP as it answers, or refused saying why it couldn't be.""" + try: + session = await self._session(ToolContext.current().caller.session) + return await session.call_tool(tool, arguments) + except DeviceUnavailable as e: + return _refused(e.message) + except Exception as e: + return _refused(f"Playwright MCP didn't answer {tool}: {type(e).__name__}: {e}") + + async def _session(self, caller: Optional[str]) -> ClientSession: + async with self._lock: + url = await self._running() + held = self._sessions.get(caller) + if held is None or held.url != url or held.owner.done(): + if held is not None: + held.close.set() + held = self._sessions[caller] = await _Session.open(url) + return held.session + + async def _running(self) -> str: + """Playwright MCP's MCP URL, started on the browser's current address for it unless it runs there already.""" + endpoint = await self._endpoint() + if self._process is not None and self._process.returncode is None and endpoint == self._running_on: + return self._url + await self._stop() + port = _free_port() + PLAYWRIGHT_MCP_OUTPUT.mkdir(parents=True, exist_ok=True) + uploads = Path.home() / "Uploads" + uploads.mkdir(parents=True, exist_ok=True) + with (PLAYWRIGHT_MCP_OUTPUT / "playwright-mcp.log").open("ab") as log: + self._process = await asyncio.create_subprocess_exec( + *_command(endpoint, port), cwd=uploads, env=_environment(), stdin=asyncio.subprocess.DEVNULL, + stdout=asyncio.subprocess.DEVNULL, stderr=log) + self._running_on, self._url = endpoint, f"http://127.0.0.1:{port}/mcp" + await _listening(port, self._process) + return self._url + + async def _stop(self) -> None: + for held in self._sessions.values(): + held.close.set() + self._sessions.clear() + if self._process is not None and self._process.returncode is None: + self._process.terminate() + with contextlib.suppress(asyncio.TimeoutError): + await asyncio.wait_for(self._process.wait(), 10) + if self._process.returncode is None: + self._process.kill() + await self._process.wait() + self._process = None + + +class _Session: + """One MCP session on Playwright MCP, held open by a task of its own until ``close`` is set, since mcp's client + contexts are anyio task groups, which only the task that entered one can leave.""" + + def __init__(self, url: str) -> None: + self.url = url + self.close = asyncio.Event() + self.session: Optional[ClientSession] = None + self.owner: Optional[asyncio.Task] = None + + @classmethod + async def open(cls, url: str) -> _Session: + held = cls(url) + ready: asyncio.Future = asyncio.get_running_loop().create_future() + held.owner = asyncio.create_task(held._own(ready)) + held.session = await ready + return held + + async def _own(self, ready: asyncio.Future) -> None: + try: + timeout = httpx.Timeout(PLAYWRIGHT_MCP_CALL_TIMEOUT_S, connect=PLAYWRIGHT_MCP_START_S) + async with httpx.AsyncClient(timeout=timeout, trust_env=False) as http, \ + streamable_http_client(self.url, http_client=http) as (read, write, _), \ + ClientSession(read, write) as session: + await session.initialize() + ready.set_result(session) + await self.close.wait() + except BaseException as e: + if not ready.done(): + ready.set_exception(e if isinstance(e, Exception) else RuntimeError(repr(e))) + elif not isinstance(e, asyncio.CancelledError): + logger.warning(f"a session on Playwright MCP ended: {type(e).__name__}: {e}") + if not isinstance(e, Exception): + raise + + +def _command(endpoint: str, port: int) -> list[str]: + """Playwright MCP on loopback ``port``, attached to the browser at ``endpoint``, never a browser of its own, with + its file access confined to its working folder, the session user's ~/Uploads; ``setpriv`` ends it with the driver.""" + return ["setpriv", "--pdeathsig", "TERM", "--", str(NODE), str(PLAYWRIGHT_MCP), "--cdp-endpoint", endpoint, + "--host", "127.0.0.1", "--port", str(port), "--caps", "vision", "--ignore-roots", + "--output-dir", str(PLAYWRIGHT_MCP_OUTPUT)] + + +def _environment() -> dict[str, str]: + return {name: os.environ[name] for name in KEPT_ENVIRONMENT if name in os.environ} + + +def _free_port() -> int: + with socket.socket() as probe: + probe.bind(("127.0.0.1", 0)) + return probe.getsockname()[1] + + +async def _listening(port: int, process: asyncio.subprocess.Process) -> None: + """Once Playwright MCP listens on ``port``; DeviceUnavailable when it exits first or takes too long.""" + deadline = asyncio.get_running_loop().time() + PLAYWRIGHT_MCP_START_S + while asyncio.get_running_loop().time() < deadline: + if process.returncode is not None: + raise DeviceUnavailable(f"Playwright MCP exited at start with {process.returncode}; its log is in " + f"{PLAYWRIGHT_MCP_OUTPUT}") + with contextlib.suppress(OSError): + _, writer = await asyncio.open_connection("127.0.0.1", port) + writer.close() + return + await asyncio.sleep(0.1) + raise DeviceUnavailable(f"Playwright MCP didn't listen within {PLAYWRIGHT_MCP_START_S:.0f} s") + + +def _refused(message: str) -> CallToolResult: + return CallToolResult(isError=True, content=[TextContent(type="text", text=message)]) + + if __name__ == "__main__": PageDriver().serve() diff --git a/src/agent_env/drivers/envs/page/env.toml b/src/agent_env/drivers/envs/page/env.toml index 6af38616c..9fc8c6514 100644 --- a/src/agent_env/drivers/envs/page/env.toml +++ b/src/agent_env/drivers/envs/page/env.toml @@ -4,4 +4,14 @@ port = 18765 [install.linux] python = "requirements.txt" +# Node and Playwright MCP 0.0.84, whose browser_* tools the driver serves beside its own: Node 24.22.2's Linux tarball +# (sha256 from nodejs.org's SHASUMS256.txt) and Playwright MCP's three npm packages (checked against npm's integrity). +downloads = [ + { url = "https://nodejs.org/dist/v24.22.2/node-v24.22.2-linux-x64.tar.gz", sha256 = "361718c444e49af2ca5a110c83ba6a8901143f23ab285ebbef0341f391f482a9", to = "node.tar.gz", arch = "amd64" }, + { url = "https://nodejs.org/dist/v24.22.2/node-v24.22.2-linux-arm64.tar.gz", sha256 = "95c264150aceb7f3437a29fd6984924f5f8d1511ea3cbd6c508b83f6c555dae2", to = "node.tar.gz", arch = "arm64" }, + { url = "https://registry.npmjs.org/@playwright/mcp/-/mcp-0.0.84.tgz", sha256 = "4e749bdc5a19138fed51cf0f51e43dca265c1bfc0fc388220fc70756aac60c26", to = "playwright-mcp.tgz" }, + { url = "https://registry.npmjs.org/playwright/-/playwright-1.65.0-alpha-2026-10-30.tgz", sha256 = "0d140b937c384637321d1c90b63794b1f1739b82966b66038077c1493ab9bc2f", to = "playwright.tgz" }, + { url = "https://registry.npmjs.org/playwright-core/-/playwright-core-1.65.0-alpha-2026-10-30.tgz", sha256 = "6f46fa60a6ee52221ebe612a03329f8b233d20158ad37f6b240dd087f4a0f4f9", to = "playwright-core.tgz" }, +] +run = [{ script = "install-playwright-mcp.sh" }] services = [{ name = "driver", command = "venv/bin/python -m agentenv_protocol.computer.serve driver.py:PageDriver" }] diff --git a/src/agent_env/drivers/envs/page/install-playwright-mcp.sh b/src/agent_env/drivers/envs/page/install-playwright-mcp.sh new file mode 100755 index 000000000..aba7eb83f --- /dev/null +++ b/src/agent_env/drivers/envs/page/install-playwright-mcp.sh @@ -0,0 +1,19 @@ +#!/bin/sh +# Node and Playwright MCP, from the tarballs the install downloaded and checked, with no npm install, since an install +# has no network: Node's binary and licence in node/, and Playwright MCP's three npm packages in node_modules/, each +# where npm would put it. +set -eu +work=$(mktemp -d) +tar -xzf node.tar.gz -C "$work" +mkdir -p node/bin +mv "$work"/node-*/bin/node node/bin/node +mv "$work"/node-*/LICENSE node/LICENSE +rm -rf "$work" node.tar.gz +unpack() { + mkdir -p "node_modules/$2" + tar -xzf "$1" -C "node_modules/$2" --strip-components=1 + rm -f "$1" +} +unpack playwright-mcp.tgz @playwright/mcp +unpack playwright.tgz playwright +unpack playwright-core.tgz playwright-core diff --git a/src/agent_env/drivers/envs/page/playwright-mcp-tools.json b/src/agent_env/drivers/envs/page/playwright-mcp-tools.json new file mode 100644 index 000000000..c5799bc70 --- /dev/null +++ b/src/agent_env/drivers/envs/page/playwright-mcp-tools.json @@ -0,0 +1,1131 @@ +{ + "version": "0.0.84", + "caps": [ + "vision" + ], + "tools": [ + { + "name": "browser_close", + "description": "Close the page", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": {}, + "additionalProperties": false + }, + "annotations": { + "title": "Close browser", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_resize", + "description": "Resize the browser window", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "width": { + "type": "number", + "description": "Width of the browser window" + }, + "height": { + "type": "number", + "description": "Height of the browser window" + } + }, + "required": [ + "width", + "height" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Resize browser window", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_console_messages", + "description": "Returns all console messages", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "level": { + "default": "info", + "description": "Level of the console messages to return. Each level includes the messages of more severe levels. Defaults to \"info\".", + "type": "string", + "enum": [ + "error", + "warning", + "info", + "debug" + ] + }, + "all": { + "description": "Return all console messages since the beginning of the session, not just since the last navigation. Defaults to false.", + "type": "boolean" + }, + "filename": { + "description": "File name to save the console messages to. Relative file names are resolved against the workspace root. If not provided, messages are returned as text.", + "type": "string" + } + }, + "required": [ + "level" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Get console messages", + "readOnlyHint": true, + "destructiveHint": false, + "openWorldHint": true + } + }, + { + "name": "browser_handle_dialog", + "description": "Handle a dialog", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "accept": { + "type": "boolean", + "description": "Whether to accept the dialog." + }, + "promptText": { + "description": "The text of the prompt in case of a prompt dialog.", + "type": "string" + } + }, + "required": [ + "accept" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Handle a dialog", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_emulate_media", + "description": "Emulate CSS media features for the page, for example switch between the light and dark color scheme. Omitted parameters are left unchanged; null clears an override.", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "colorScheme": { + "description": "Emulates the prefers-color-scheme media feature", + "anyOf": [ + { + "type": "string", + "enum": [ + "light", + "dark" + ] + }, + { + "type": "null" + } + ] + }, + "reducedMotion": { + "description": "Emulates the prefers-reduced-motion media feature", + "anyOf": [ + { + "type": "string", + "enum": [ + "reduce", + "no-preference" + ] + }, + { + "type": "null" + } + ] + }, + "forcedColors": { + "description": "Emulates the forced-colors media feature", + "anyOf": [ + { + "type": "string", + "enum": [ + "active", + "none" + ] + }, + { + "type": "null" + } + ] + }, + "contrast": { + "description": "Emulates the prefers-contrast media feature", + "anyOf": [ + { + "type": "string", + "enum": [ + "more", + "no-preference" + ] + }, + { + "type": "null" + } + ] + }, + "media": { + "description": "Changes the CSS media type of the page", + "anyOf": [ + { + "type": "string", + "enum": [ + "screen", + "print" + ] + }, + { + "type": "null" + } + ] + } + }, + "additionalProperties": false + }, + "annotations": { + "title": "Emulate media features", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_evaluate", + "description": "Evaluate JavaScript expression on page or element", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "element": { + "description": "Human-readable element description used to obtain permission to interact with the element", + "type": "string" + }, + "target": { + "description": "Exact target element reference from the page snapshot, or a unique element selector", + "type": "string" + }, + "function": { + "type": "string", + "description": "() => { /* code */ } or (element) => { /* code */ } when element is provided" + }, + "filename": { + "description": "File name to save the result to. Relative file names are resolved against the workspace root. If not provided, result is returned as text.", + "type": "string" + } + }, + "required": [ + "function" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Evaluate JavaScript", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_file_upload", + "description": "Upload one or multiple files", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "paths": { + "description": "The absolute paths to the files to upload. Can be single file or multiple files. If omitted, file chooser is cancelled.", + "type": "array", + "items": { + "type": "string" + } + } + }, + "additionalProperties": false + }, + "annotations": { + "title": "Upload files", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_drop", + "description": "Drop files or MIME-typed data onto an element, as if dragged from outside the page. At least one of \"paths\" or \"data\" must be provided.", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "element": { + "description": "Human-readable element description used to obtain permission to interact with the element", + "type": "string" + }, + "target": { + "type": "string", + "description": "Exact target element reference from the page snapshot, or a unique element selector" + }, + "paths": { + "description": "Absolute paths to files to drop onto the element.", + "type": "array", + "items": { + "type": "string" + } + }, + "data": { + "description": "Data to drop, as a map of MIME type to string value (e.g. {\"text/plain\": \"hello\", \"text/uri-list\": \"https://example.com\"}).", + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": { + "type": "string" + } + } + }, + "required": [ + "target" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Drop files or data onto an element", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_find", + "description": "Search the accessibility snapshot of the current page for text or a regular expression. Returns matching snapshot nodes with a few lines of surrounding context (like search snippets), each shown under its path from the root of the tree, which is cheaper than capturing the whole snapshot when you only need to locate an element and its ref.", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "text": { + "description": "Plain text to search for in the page snapshot (case-insensitive substring match). Provide either text or regex, not both.", + "type": "string" + }, + "regex": { + "description": "Regular expression to search for in the page snapshot. Matching is case-sensitive by default; wrap the pattern in slashes to add flags, e.g. \"/error/i\" for case-insensitive. Provide either text or regex, not both.", + "type": "string" + }, + "maxResults": { + "description": "Maximum number of matching items to return. Defaults to returning all matches.", + "type": "number" + }, + "filename": { + "description": "Save results to a file instead of returning them in the response. Relative file names are resolved against the workspace root.", + "type": "string" + } + }, + "additionalProperties": false + }, + "annotations": { + "title": "Find in page snapshot", + "readOnlyHint": true, + "destructiveHint": false, + "openWorldHint": true + } + }, + { + "name": "browser_fill_form", + "description": "Fill multiple form fields", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "fields": { + "type": "array", + "items": { + "type": "object", + "properties": { + "element": { + "description": "Human-readable element description used to obtain permission to interact with the element", + "type": "string" + }, + "target": { + "type": "string", + "description": "Exact target element reference from the page snapshot, or a unique element selector" + }, + "name": { + "type": "string", + "description": "Human-readable field name" + }, + "type": { + "type": "string", + "enum": [ + "textbox", + "checkbox", + "radio", + "combobox", + "slider" + ], + "description": "Type of the field" + }, + "value": { + "type": "string", + "description": "Value to fill in the field. If the field is a checkbox, the value should be `true` or `false`. If the field is a combobox, the value should be the text of the option." + } + }, + "required": [ + "target", + "name", + "type", + "value" + ], + "additionalProperties": false + }, + "description": "Fields to fill in" + } + }, + "required": [ + "fields" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Fill form", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_press_key", + "description": "Press a key on the keyboard", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "key": { + "type": "string", + "description": "Name of the key to press or a character to generate, such as `ArrowLeft` or `a`" + } + }, + "required": [ + "key" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Press a key", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_type", + "description": "Type text into editable element", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "element": { + "description": "Human-readable element description used to obtain permission to interact with the element", + "type": "string" + }, + "target": { + "type": "string", + "description": "Exact target element reference from the page snapshot, or a unique element selector" + }, + "text": { + "type": "string", + "description": "Text to type into the element" + }, + "submit": { + "description": "Whether to submit entered text (press Enter after)", + "type": "boolean" + }, + "slowly": { + "description": "Whether to type one character at a time. Useful for triggering key handlers in the page. By default entire text is filled in at once.", + "type": "boolean" + } + }, + "required": [ + "target", + "text" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Type text", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_mouse_move_xy", + "description": "Move mouse to a given position", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "x": { + "type": "number", + "description": "X coordinate" + }, + "y": { + "type": "number", + "description": "Y coordinate" + } + }, + "required": [ + "x", + "y" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Move mouse", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_mouse_click_xy", + "description": "Click mouse button at a given position", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "x": { + "type": "number", + "description": "X coordinate" + }, + "y": { + "type": "number", + "description": "Y coordinate" + }, + "button": { + "description": "Button to click, defaults to left", + "type": "string", + "enum": [ + "left", + "right", + "middle" + ] + }, + "clickCount": { + "description": "Number of clicks, defaults to 1", + "type": "number" + }, + "delay": { + "description": "Time to wait between mouse down and mouse up in milliseconds, defaults to 0", + "type": "number" + } + }, + "required": [ + "x", + "y" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Click", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_mouse_drag_xy", + "description": "Drag left mouse button to a given position", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "startX": { + "type": "number", + "description": "Start X coordinate" + }, + "startY": { + "type": "number", + "description": "Start Y coordinate" + }, + "endX": { + "type": "number", + "description": "End X coordinate" + }, + "endY": { + "type": "number", + "description": "End Y coordinate" + } + }, + "required": [ + "startX", + "startY", + "endX", + "endY" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Drag mouse", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_mouse_down", + "description": "Press mouse down", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "button": { + "description": "Button to press, defaults to left", + "type": "string", + "enum": [ + "left", + "right", + "middle" + ] + } + }, + "additionalProperties": false + }, + "annotations": { + "title": "Press mouse down", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_mouse_up", + "description": "Press mouse up", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "button": { + "description": "Button to press, defaults to left", + "type": "string", + "enum": [ + "left", + "right", + "middle" + ] + } + }, + "additionalProperties": false + }, + "annotations": { + "title": "Press mouse up", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_mouse_wheel", + "description": "Scroll mouse wheel", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "deltaX": { + "default": 0, + "description": "X delta", + "type": "number" + }, + "deltaY": { + "default": 0, + "description": "Y delta", + "type": "number" + } + }, + "required": [ + "deltaX", + "deltaY" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Scroll mouse wheel", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_navigate", + "description": "Navigate to a URL", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "url": { + "type": "string", + "description": "The URL to navigate to" + } + }, + "required": [ + "url" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Navigate to a URL", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_navigate_back", + "description": "Go back to the previous page in the history", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": {}, + "additionalProperties": false + }, + "annotations": { + "title": "Go back", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_network_requests", + "description": "Returns a numbered list of network requests since loading the page. Use browser_network_request with the number to get full details.", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "static": { + "default": false, + "description": "Whether to include successful static resources like images, fonts, scripts, etc. Defaults to false.", + "type": "boolean" + }, + "filter": { + "description": "Only return requests whose URL matches this regexp (e.g. \"/api/.*user\").", + "type": "string" + }, + "all": { + "description": "Return all network requests since the beginning of the session, not just since the last navigation. Defaults to false.", + "type": "boolean" + }, + "filename": { + "description": "File name to save the network requests to. Relative file names are resolved against the workspace root. If not provided, requests are returned as text.", + "type": "string" + } + }, + "required": [ + "static" + ], + "additionalProperties": false + }, + "annotations": { + "title": "List network requests", + "readOnlyHint": true, + "destructiveHint": false, + "openWorldHint": true + } + }, + { + "name": "browser_network_request", + "description": "Returns full details (headers and body) of a single network request, or a single part if `part` is set. Use the number from browser_network_requests.", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "index": { + "type": "integer", + "minimum": 1, + "maximum": 9007199254740991, + "description": "1-based index of the request, as printed by browser_network_requests." + }, + "part": { + "description": "Return only this part of the request. Omit to return full details.", + "type": "string", + "enum": [ + "request-headers", + "request-body", + "response-headers", + "response-body" + ] + }, + "filename": { + "description": "File name to save the result to. Relative file names are resolved against the workspace root. If not provided, output is returned as text.", + "type": "string" + } + }, + "required": [ + "index" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Show network request details", + "readOnlyHint": true, + "destructiveHint": false, + "openWorldHint": true + } + }, + { + "name": "browser_run_code_unsafe", + "description": "Run a Playwright code snippet. Unsafe: executes arbitrary JavaScript in the Playwright server process and is RCE-equivalent.", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "code": { + "description": "A JavaScript function containing Playwright code to execute. It will be invoked with a single argument, page, which you can use for any page interaction. For example: `async (page) => { await page.getByRole('button', { name: 'Submit' }).click(); return await page.title(); }`", + "type": "string" + }, + "filename": { + "description": "Load code from the specified file. Relative file names are resolved against the workspace root. If both code and filename are provided, code will be ignored.", + "type": "string" + } + }, + "additionalProperties": false + }, + "annotations": { + "title": "Run Playwright code (unsafe)", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_take_screenshot", + "description": "Take a screenshot of the current page. You can't perform actions based on the screenshot, use browser_snapshot for actions.", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "element": { + "description": "Human-readable element description used to obtain permission to interact with the element", + "type": "string" + }, + "target": { + "description": "Exact target element reference from the page snapshot, or a unique element selector", + "type": "string" + }, + "type": { + "description": "Image format for the screenshot. If unset, inferred from the filename extension, otherwise png.", + "type": "string", + "enum": [ + "png", + "jpeg", + "webp" + ] + }, + "filename": { + "description": "File name to save the screenshot to. Relative file names are resolved against the workspace root. If not specified, the screenshot is saved into the output directory as `page-{timestamp}.{png|jpeg|webp}`.", + "type": "string" + }, + "fullPage": { + "description": "When true, takes a screenshot of the full scrollable page, instead of the currently visible viewport. Cannot be used with element screenshots.", + "type": "boolean" + }, + "scale": { + "default": "css", + "description": "Image resolution scale. \"css\" produces a screenshot sized in CSS pixels (smaller, consistent across devices). \"device\" produces a high-resolution screenshot using device pixels (larger, accounts for the device pixel ratio). Default is css.", + "type": "string", + "enum": [ + "css", + "device" + ] + } + }, + "required": [ + "scale" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Take a screenshot", + "readOnlyHint": true, + "destructiveHint": false, + "openWorldHint": true + } + }, + { + "name": "browser_snapshot", + "description": "Capture accessibility snapshot of the current page, this is better than screenshot", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "target": { + "description": "Exact target element reference from the page snapshot, or a unique element selector", + "type": "string" + }, + "filename": { + "description": "Save snapshot to a file instead of returning it in the response. Relative file names are resolved against the workspace root.", + "type": "string" + }, + "depth": { + "description": "Limit the depth of the snapshot tree", + "type": "number" + }, + "boxes": { + "description": "Include each element's bounding box as [box=x,y,width,height] in the snapshot. Coordinates are viewport-relative, in CSS pixels (Element.getBoundingClientRect)", + "type": "boolean" + } + }, + "additionalProperties": false + }, + "annotations": { + "title": "Page snapshot", + "readOnlyHint": true, + "destructiveHint": false, + "openWorldHint": true + } + }, + { + "name": "browser_click", + "description": "Perform click on a web page", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "element": { + "description": "Human-readable element description used to obtain permission to interact with the element", + "type": "string" + }, + "target": { + "type": "string", + "description": "Exact target element reference from the page snapshot, or a unique element selector" + }, + "doubleClick": { + "description": "Whether to perform a double click instead of a single click", + "type": "boolean" + }, + "button": { + "description": "Button to click, defaults to left", + "type": "string", + "enum": [ + "left", + "right", + "middle" + ] + }, + "modifiers": { + "description": "Modifier keys to press", + "type": "array", + "items": { + "type": "string", + "enum": [ + "Alt", + "Control", + "ControlOrMeta", + "Meta", + "Shift" + ] + } + } + }, + "required": [ + "target" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Click", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_drag", + "description": "Perform drag and drop between two elements", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "startElement": { + "description": "Human-readable source element description used to obtain the permission to interact with the element", + "type": "string" + }, + "startTarget": { + "type": "string", + "description": "Exact target element reference from the page snapshot, or a unique element selector" + }, + "endElement": { + "description": "Human-readable target element description used to obtain the permission to interact with the element", + "type": "string" + }, + "endTarget": { + "type": "string", + "description": "Exact target element reference from the page snapshot, or a unique element selector" + } + }, + "required": [ + "startTarget", + "endTarget" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Drag mouse", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_hover", + "description": "Hover over element on page", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "element": { + "description": "Human-readable element description used to obtain permission to interact with the element", + "type": "string" + }, + "target": { + "type": "string", + "description": "Exact target element reference from the page snapshot, or a unique element selector" + } + }, + "required": [ + "target" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Hover mouse", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_select_option", + "description": "Select an option in a dropdown", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "element": { + "description": "Human-readable element description used to obtain permission to interact with the element", + "type": "string" + }, + "target": { + "type": "string", + "description": "Exact target element reference from the page snapshot, or a unique element selector" + }, + "values": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Array of values to select in the dropdown. This can be a single value or multiple values." + } + }, + "required": [ + "target", + "values" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Select option", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_tabs", + "description": "List, create, close, or select a browser tab.", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "action": { + "type": "string", + "enum": [ + "list", + "new", + "close", + "select" + ], + "description": "Operation to perform" + }, + "index": { + "description": "Tab index, used for close/select. If omitted for close, current tab is closed.", + "type": "number" + }, + "url": { + "description": "URL to navigate to in the new tab, used for new.", + "type": "string" + }, + "isolatedContext": { + "description": "Name of an isolated browser context to create the new tab in, used for new. Tabs in the same isolated context share cookies and storage, tabs in different contexts are fully isolated. If omitted or \"default\", the tab is created in the default context.", + "type": "string" + } + }, + "required": [ + "action" + ], + "additionalProperties": false + }, + "annotations": { + "title": "Manage tabs", + "readOnlyHint": false, + "destructiveHint": true, + "openWorldHint": true + } + }, + { + "name": "browser_wait_for", + "description": "Wait for text to appear or disappear or a specified time to pass. When both text and textGone are provided, waits for the first one to happen", + "inputSchema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": { + "time": { + "description": "The time to wait in seconds, at most 30. When combined with text or textGone, serves as a timeout for them instead of the default action timeout", + "type": "number" + }, + "text": { + "description": "The text to wait for", + "type": "string" + }, + "textGone": { + "description": "The text to wait for to disappear", + "type": "string" + } + }, + "additionalProperties": false + }, + "annotations": { + "title": "Wait for", + "readOnlyHint": true, + "destructiveHint": false, + "openWorldHint": true + } + } + ] +} diff --git a/tst/integration/computer/browser_computer_test.py b/tst/integration/computer/browser_computer_test.py index 5e3656cca..14154b31b 100644 --- a/tst/integration/computer/browser_computer_test.py +++ b/tst/integration/computer/browser_computer_test.py @@ -1,7 +1,9 @@ """The browser as a computer: agentenv's chromium deployed alone, on the ubuntu-24.04-browser base with no desktop and the page driver, whose screen is chromium's active tab, headful on the X server only the browser uses. Gate 3 makes this machine's arch ready with it, its input checks and page tools verified on the HTML desk; deployed through the computer provider like any computer, -its computer_* tools see and drive the page, and its page tools navigate, move in history, answer a dialog, use tabs +its computer_* tools see and drive the page, Playwright MCP's browser_* tools beside them type into it and answer its +dialog, the two that run code hidden until tool access names one, as the installed Playwright MCP lists them, and +back after Playwright MCP or the driver is killed, and its page tools navigate, move in history, answer a dialog, use tabs and upload; the browser's X server runs but no desktop, and nothing listens but the driver and the site proxy; a killed app server or driver is started again by supervisord and the computer is ready again; chromium killed fails it for good, computer-ready/v1 saying why and the tools answering with it, and so does its X server killed, which takes @@ -59,8 +61,10 @@ from agentenv_protocol.client import find_extension_method, invoke_extension from agentenv_protocol.computer import COMPUTER_PAGE, COMPUTER_READY from agentenv_protocol.computer.conformance.client import image +from agentenv_protocol.tool_access import ENABLE, ENABLE_TOOL_EXTENSION_URI from agentenv_protocol.website_browser import BROWSER +import agent_env from agent_env.a2a_agent import conversation_store from agent_env.artifact import EnvironmentUniverseArtifact from agent_env.artifact.artifacts.environment import EnvironmentArtifact @@ -161,6 +165,16 @@ def log_message(self, *args): time.sleep(float(sys.argv[1])) """ SERVE_START_S = 3 +#: Playwright MCP's own tools/list in the computer, from the node and cli.js it's given, as JSON. +LISTED_TOOLS = """import json, subprocess, sys +messages = [{"jsonrpc": "2.0", "id": 1, "method": "initialize", "params": {"protocolVersion": "2025-06-18", + "capabilities": {}, "clientInfo": {"name": "test", "version": "0"}}}, + {"jsonrpc": "2.0", "method": "notifications/initialized"}, {"jsonrpc": "2.0", "id": 2, "method": "tools/list"}] +out = subprocess.run(sys.argv[1:] + ["--cdp-endpoint", "http://127.0.0.1:9/", "--caps", "vision"], capture_output=True, + input="\\n".join(map(json.dumps, messages)) + "\\n", text=True, timeout=60).stdout +print(json.dumps(next(r["result"]["tools"] for r in map(json.loads, out.splitlines()) if r.get("id") == 2))) +""" +PLAYWRIGHT_MCP_TOOLS = Path(agent_env.__file__).parent / "drivers" / "envs" / "page" / "playwright-mcp-tools.json" #: A page that needs no server, so killing the driver, whose child a served site is, leaves it to open. DATA_PAGE = "data:text/html,Home

Home

" #: A page a task's tab steps open, bring to the front and close by its URL. @@ -882,6 +896,89 @@ async def test_playwright_and_puppeteer_drive_it_through_its_cdp_address_which_a assert fresh != ada and not report.failed +@pytest.mark.asyncio +async def test_playwright_mcps_tools_drive_its_page_hide_the_code_tools_and_come_back_after_a_kill(browser_computer): + with namespace_routing(): + deployed = await Env.get(browser_computer.id).deploy( + env_params={"app_settings": {"chromium": {"start_urls": [PAGE_URL]}}}) + context = TaskStepContext(deployed_envs=[deployed]) + serving = None + try: + viewport = (await _screenshot(deployed)).size + listed = await _tool_names(deployed) + typed = await _mcp(deployed, _type_hello) + state = await _state(deployed, lambda s: s["v"] == "hello") + after_typing = (await _screenshot(deployed)).size + serving = asyncio.ensure_future(inside.run(deployed, ["python3", "-c", SERVE_SITE, "900"], timeout_s=960)) + await asyncio.sleep(SERVE_START_S) + confirmed = await _mcp(deployed, _confirm_ask) + await deployed.invoke(ENABLE_TOOL_EXTENSION_URI, ENABLE, {"role": "default", "tools": ["browser_evaluate"]}) + shown, evaluated = await _tool_names(deployed), await _mcp(deployed, _evaluate_title) + running = (await inside.run(deployed, ["sh", "-c", "ps -eo args | grep -m1 '[@]playwright/mcp/cli.js'"], + timeout_s=30))["stdout"].split() + installed = json.loads((await inside.run(deployed, ["python3", "-c", LISTED_TOOLS, *running[:2]], + timeout_s=120))["stdout"]) + await _kill(deployed, "[@]playwright/mcp/cli.js") + after_its_kill = await _mcp(deployed, _snapshot) + await _kill(deployed, "[c]omputer.serve driver.py:PageDriver") + await _ready(deployed) + after_driver_kill = await _mcp(deployed, _snapshot) + count = (await inside.run(deployed, ["sh", "-c", "pgrep -fc '[@]playwright/mcp/cli.js'"], + timeout_s=30))["stdout"].strip() + record = await _record(deployed) + finally: + if serving is not None: + serving.cancel() + report = await teardown_run(context) + + playwright = {t["name"] for t in json.loads(PLAYWRIGHT_MCP_TOOLS.read_text())["tools"]} + assert playwright - {"browser_evaluate", "browser_run_code_unsafe"} <= set(listed), listed + assert not {"browser_evaluate", "browser_run_code_unsafe"} & set(listed) and "computer_screenshot" in listed + assert "hello" in typed and state["v"] == "hello" and after_typing == viewport + assert "confirmed true" in confirmed, confirmed + assert "browser_evaluate" in shown and "browser_run_code_unsafe" not in shown and "confirmed true" in evaluated + assert installed == json.loads(PLAYWRIGHT_MCP_TOOLS.read_text())["tools"] + assert "Page URL" in after_its_kill and "Page URL" in after_driver_kill and count == "1" + assert "playwright-mcp" in {connection["user"] for connection in record["cdp_connections"]} + assert not report.failed + + +async def _mcp(deployed, work): + """What ``work`` makes of one MCP session on the computer, as a role-less agent opens one.""" + async with streamablehttp_client(deployed.mcp_url) as (read, write, _): + async with ClientSession(read, write) as session: + await session.initialize() + return await work(session) + + +async def _called(session: ClientSession, tool: str, arguments: dict) -> str: + result = await session.call_tool(tool, arguments) + text = "\n".join(part.text for part in result.content if part.type == "text") + assert not result.isError, f"{tool}: {text}" + return text + + +async def _snapshot(session: ClientSession) -> str: + return await _called(session, "browser_snapshot", {}) + + +async def _type_hello(session: ClientSession) -> str: + ref = re.search(r"textbox \[ref=(\w+)\]", await _snapshot(session)).group(1) + return await _called(session, "browser_type", {"target": ref, "text": "hello"}) + + +async def _confirm_ask(session: ClientSession) -> str: + await _called(session, "browser_navigate", {"url": SITE}) + ref = re.search(r'button "ask" \[ref=(\w+)\]', await _snapshot(session)).group(1) + await session.call_tool("browser_click", {"target": ref}) + await _called(session, "browser_handle_dialog", {"accept": True}) + return await _snapshot(session) + + +async def _evaluate_title(session: ClientSession) -> str: + return await _called(session, "browser_evaluate", {"function": "() => document.title"}) + + @pytest.mark.asyncio async def test_behind_a_front_that_ends_tls_an_address_under_the_origin_named_is_https_and_wss_and_clients_connect( browser_computer, cdp_clients): diff --git a/tst/unit/drivers/page_driver_test.py b/tst/unit/drivers/page_driver_test.py index f6ee8f1e4..13a97d4c8 100644 --- a/tst/unit/drivers/page_driver_test.py +++ b/tst/unit/drivers/page_driver_test.py @@ -1,31 +1,55 @@ """The page driver against a fake browser server on the app's socket: it passes the definition checks with a page screen, turns tool calls into the CDP input they mean, shows a tab that paints nothing as white, and once the browser -has ended on its own, its ready check fails for good and its calls say why.""" +has ended on its own, its ready check fails for good and its calls say why. Beside its own tools it serves Playwright +MCP's from their manifest, the two that run the agent's code hidden, on a Playwright MCP it runs on the CDP address the +browser issues it under the driver's loopback origin: here a stand-in, which starts on the first call, keeps a session +per caller, starts again on a new address or once killed, and never gets the driver's environment.""" from __future__ import annotations +import asyncio import base64 import importlib.util import json +import os +import signal +import sys from pathlib import Path import httpx import pytest +from pytest_socket import enable_socket import agent_env from agentenv_protocol.computer import (DeviceUnavailable, FileChooser, InvalidArgument, PageDialog, PageState, Platform, Tab) from agentenv_protocol.computer import driver as computer_driver from agentenv_protocol.computer.app_runtime import LAUNCHES +from agentenv_protocol.tool_context import ToolContext, bound from agentenv_protocol.website_browser import BROWSER DRIVER = Path(agent_env.__file__).parent / "drivers" / "envs" / "page" / "driver.py" NO_DISPLAY = Platform(os="linux", distribution="ubuntu", version="24.04", arch="amd64", display="none") PAGE_OPS = ("page/viewport", "page/capture", "page/input", "page/state", "navigate", "history", "tabs/list", - "tabs/open", "tabs/activate", "tabs/close", "dialog/answer", "files/choose") + "tabs/open", "tabs/activate", "tabs/close", "dialog/answer", "files/choose", "cdp") GETS = ("page/viewport", "page/capture", "page/state", "tabs/list") CARD = {"name": "chromium", "capabilities": {"extensions": [{"uri": BROWSER, "params": {"methods": { op: {"method": "GET" if op in GETS else "POST", "endpoint": f"/agentenv/ext/browser/{op}"} for op in PAGE_OPS}}}]}} +#: A stand-in for Playwright MCP on the port and CDP address it's started with, whose one tool says what it was started +#: with and which session called it; started on an address ending /exits, it exits at once. +FAKE_PLAYWRIGHT_MCP = """import json, os, sys +from mcp.server.fastmcp import Context, FastMCP +args = sys.argv[1:] +endpoint, port = args[args.index("--cdp-endpoint") + 1], int(args[args.index("--port") + 1]) +if endpoint.endswith("/exits"): + sys.exit(3) +app = FastMCP("fake", host="127.0.0.1", port=port) +@app.tool() +def browser_snapshot(ctx: Context) -> str: + return json.dumps({"pid": os.getpid(), "endpoint": endpoint, "session": id(ctx.session), "cwd": os.getcwd(), + "env": sorted(os.environ)}) +app.run("streamable-http") +""" #: A 2×1 PNG's first 24 bytes: its signature, then its IHDR's length, type, width and height. PNG = b"\x89PNG\r\n\x1a\n" + b"\x00\x00\x00\rIHDR" + (2).to_bytes(4, "big") + (1).to_bytes(4, "big") @@ -220,3 +244,81 @@ async def test_a_call_the_browser_refuses_as_given_wrong_is_an_invalid_argument_ with pytest.raises(InvalidArgument, match="there's no file cv.pdf in ~/Uploads"): await page.choose_files(["cv.pdf"]) + +def test_it_serves_playwright_mcps_tools_from_their_manifest_and_hides_the_two_that_run_the_agents_code(page): + manifest = json.loads((DRIVER.parent / "playwright-mcp-tools.json").read_text()) + forwarded = page.forwarded_tools() + + assert manifest["version"] == "0.0.84" and manifest["caps"] == ["vision"] + assert [f.tool.name for f in forwarded] == [t["name"] for t in manifest["tools"]] + assert [f.tool.inputSchema for f in forwarded] == [t["inputSchema"] for t in manifest["tools"]] + assert {f.tool.name for f in forwarded if f.hidden_by_default} == {"browser_evaluate", "browser_run_code_unsafe"} + + +@pytest.mark.asyncio +async def test_it_asks_the_browser_for_playwright_mcps_address_under_its_own_loopback_origin(page, monkeypatch): + monkeypatch.setenv("MCP_PORT", "18765") + page.served["cdp"] = {"http_url": "http://127.0.0.1:18765/svc/chromium/cdp/t/", + "ws_url": "ws://127.0.0.1:18765/svc/chromium/cdp/t/devtools/browser"} + + endpoint = await page._playwright_endpoint() + + assert endpoint == "ws://127.0.0.1:18765/svc/chromium/cdp/t/devtools/browser" + assert page.sent[-1] == ("cdp", {"user": "playwright-mcp", "origin": f"http://127.0.0.1:18765/svc/{page._browser[0].name}"}) + + +def test_playwright_mcp_attaches_to_the_browser_never_launches_one_and_ends_with_the_driver(): + command = _load()._command("ws://127.0.0.1:18765/svc/chromium/cdp/t/devtools/browser", 40123) + + assert command[:4] == ["setpriv", "--pdeathsig", "TERM", "--"] + assert command[command.index("--cdp-endpoint") + 1] == "ws://127.0.0.1:18765/svc/chromium/cdp/t/devtools/browser" + assert {"--caps", "vision", "--ignore-roots", "127.0.0.1", "40123"} <= set(command) + assert not {"--isolated", "--headless", "--browser", "--allow-unrestricted-file-access"} & set(command) + + +@pytest.mark.asyncio +async def test_playwright_mcp_starts_on_the_first_call_keeps_a_session_per_caller_and_starts_again_when_it_must( + monkeypatch, tmp_path): + enable_socket() + module = _load() + fake = tmp_path / "fake_playwright_mcp.py" + fake.write_text(FAKE_PLAYWRIGHT_MCP) + monkeypatch.setattr(module, "_command", lambda endpoint, port: [ + sys.executable, str(fake), "--cdp-endpoint", endpoint, "--port", str(port)]) + monkeypatch.setattr(module, "PLAYWRIGHT_MCP_OUTPUT", tmp_path / "out") + monkeypatch.setenv("HOME", str(tmp_path / "home")) + monkeypatch.setenv("AGENTENV_CONTROL_TOKEN", "secret") + endpoints = ["ws://127.0.0.1:18765/svc/chromium/cdp/one/devtools/browser"] + + async def endpoint() -> str: + return endpoints[-1] + + playwright = module._PlaywrightMcp(endpoint) + + async def called(caller: str): + with bound(ToolContext.as_caller(session=caller)): + return await playwright.call("browser_snapshot") + + async def snapshot(caller: str) -> dict: + result = await called(caller) + assert not result.isError, result.content + return json.loads(result.content[0].text) + + try: + first, again, other = await snapshot("ada"), await snapshot("ada"), await snapshot("bob") + endpoints.append("ws://127.0.0.1:18765/svc/chromium/cdp/two/devtools/browser") + moved = await snapshot("ada") + os.kill(moved["pid"], signal.SIGKILL) + await asyncio.sleep(1) + revived = await snapshot("ada") + endpoints.append("ws://127.0.0.1:18765/svc/chromium/cdp/exits") + refused = await called("ada") + finally: + await playwright._stop() + + assert first["pid"] == again["pid"] == other["pid"] and first["session"] == again["session"] != other["session"] + assert first["endpoint"].endswith("/cdp/one/devtools/browser") and first["cwd"] == str(tmp_path / "home" / "Uploads") + assert "AGENTENV_CONTROL_TOKEN" not in first["env"] and "HOME" in first["env"] + assert moved["pid"] != first["pid"] and moved["endpoint"].endswith("/cdp/two/devtools/browser") + assert revived["pid"] != moved["pid"] + assert refused.isError and "exited at start with 3" in refused.content[0].text From ada7a3f2328291ae69b029597288bce996eec084 Mon Sep 17 00:00:00 2001 From: Edgar Arakelyan Date: Wed, 11 Nov 2026 05:13:46 -0800 Subject: [PATCH 2/7] fix(drivers): a Playwright MCP call fails when it ends under it, idle sessions close, a stuck start is stopped A call races Playwright MCP's exit and its session's end, so one Playwright MCP stops answering fails then, without being sent again, since it may have acted. A caller's session left idle for an hour is closed when another caller comes, since a caller's own MCP session ends with no word to the driver. A start that never listens is stopped, and its address and URL are recorded only once it does, so the next call starts afresh. Co-Authored-By: Claude Opus 5.5 --- src/agent_env/drivers/envs/page/driver.py | 50 +++++++-- tst/unit/drivers/page_driver_test.py | 123 ++++++++++++++++++---- 2 files changed, 140 insertions(+), 33 deletions(-) diff --git a/src/agent_env/drivers/envs/page/driver.py b/src/agent_env/drivers/envs/page/driver.py index 9b140492f..952e5251e 100644 --- a/src/agent_env/drivers/envs/page/driver.py +++ b/src/agent_env/drivers/envs/page/driver.py @@ -109,6 +109,9 @@ #: How long Playwright MCP may take to listen, and one of its tools to answer. PLAYWRIGHT_MCP_START_S = 30.0 PLAYWRIGHT_MCP_CALL_TIMEOUT_S = 300.0 +#: How long a caller's session on Playwright MCP is kept with no call: a caller's own MCP session ends with no word to +#: the driver, and each holds a connection to the browser. +PLAYWRIGHT_MCP_SESSION_IDLE_S = 3600.0 logger = logging.getLogger(__name__) @@ -436,9 +439,10 @@ def _steps(path: list) -> list[tuple[float, float]]: class _PlaywrightMcp: """Playwright MCP, attached to the browser over the CDP address ``endpoint`` gives, and its sessions there, one per - caller of the driver's tools, so each agent keeps its own snapshots and tab. It starts on the first call, again once - it has exited, and again on a fresh address once the browser's data reset has ended the last; it ends with the - driver, so a driver started again finds none of its predecessor's.""" + caller of the driver's tools, so each agent keeps its own snapshots and tab, each closed once idle for + PLAYWRIGHT_MCP_SESSION_IDLE_S. It starts on the first call, again once it has exited or failed to start, and again on + a fresh address once the browser's data reset has ended the last; it ends with the driver, so a driver started again + finds none of its predecessor's.""" def __init__(self, endpoint: Callable[[], Awaitable[str]]) -> None: self._endpoint = endpoint @@ -449,24 +453,45 @@ def __init__(self, endpoint: Callable[[], Awaitable[str]]) -> None: self._sessions: dict[Optional[str], _Session] = {} async def call(self, tool: str, /, **arguments: Any) -> CallToolResult: - """``tool`` called on Playwright MCP as it answers, or refused saying why it couldn't be.""" + """``tool`` called on Playwright MCP as it answers, or refused saying why it couldn't be. A call Playwright MCP + or its session ends under fails then, and isn't sent again, since it may have acted.""" try: - session = await self._session(ToolContext.current().caller.session) - return await session.call_tool(tool, arguments) + held, process = await self._session(ToolContext.current().caller.session) except DeviceUnavailable as e: return _refused(e.message) except Exception as e: return _refused(f"Playwright MCP didn't answer {tool}: {type(e).__name__}: {e}") + called = asyncio.ensure_future(held.session.call_tool(tool, arguments)) + exited = asyncio.ensure_future(process.wait()) + try: + await asyncio.wait({called, exited, held.owner}, return_when=asyncio.FIRST_COMPLETED) + finally: + exited.cancel() + if not called.done(): + called.cancel() + held.close.set() + return _refused(f"Playwright MCP ended during {tool}, which may have acted before it did") + try: + return called.result() + except Exception as e: + return _refused(f"Playwright MCP didn't answer {tool}: {type(e).__name__}: {e}") + finally: + held.used_at = asyncio.get_running_loop().time() - async def _session(self, caller: Optional[str]) -> ClientSession: + async def _session(self, caller: Optional[str]) -> tuple[_Session, asyncio.subprocess.Process]: async with self._lock: url = await self._running() + now = asyncio.get_running_loop().time() + for idle in [key for key, held in self._sessions.items() + if key != caller and now - held.used_at > PLAYWRIGHT_MCP_SESSION_IDLE_S]: + self._sessions.pop(idle).close.set() held = self._sessions.get(caller) if held is None or held.url != url or held.owner.done(): if held is not None: held.close.set() held = self._sessions[caller] = await _Session.open(url) - return held.session + held.used_at = now + return held, self._process async def _running(self) -> str: """Playwright MCP's MCP URL, started on the browser's current address for it unless it runs there already.""" @@ -482,8 +507,12 @@ async def _running(self) -> str: self._process = await asyncio.create_subprocess_exec( *_command(endpoint, port), cwd=uploads, env=_environment(), stdin=asyncio.subprocess.DEVNULL, stdout=asyncio.subprocess.DEVNULL, stderr=log) + try: + await _listening(port, self._process) + except BaseException: + await self._stop() + raise self._running_on, self._url = endpoint, f"http://127.0.0.1:{port}/mcp" - await _listening(port, self._process) return self._url async def _stop(self) -> None: @@ -497,7 +526,7 @@ async def _stop(self) -> None: if self._process.returncode is None: self._process.kill() await self._process.wait() - self._process = None + self._process, self._running_on = None, None class _Session: @@ -509,6 +538,7 @@ def __init__(self, url: str) -> None: self.close = asyncio.Event() self.session: Optional[ClientSession] = None self.owner: Optional[asyncio.Task] = None + self.used_at = 0.0 @classmethod async def open(cls, url: str) -> _Session: diff --git a/tst/unit/drivers/page_driver_test.py b/tst/unit/drivers/page_driver_test.py index 13a97d4c8..353721cdb 100644 --- a/tst/unit/drivers/page_driver_test.py +++ b/tst/unit/drivers/page_driver_test.py @@ -3,7 +3,8 @@ has ended on its own, its ready check fails for good and its calls say why. Beside its own tools it serves Playwright MCP's from their manifest, the two that run the agent's code hidden, on a Playwright MCP it runs on the CDP address the browser issues it under the driver's loopback origin: here a stand-in, which starts on the first call, keeps a session -per caller, starts again on a new address or once killed, and never gets the driver's environment.""" +per caller, starts again on a new address or once killed, and never gets the driver's environment; a call it ends +under fails then, a caller's session left idle is closed, and a start that never listens is stopped and tried afresh.""" from __future__ import annotations import asyncio @@ -35,19 +36,28 @@ CARD = {"name": "chromium", "capabilities": {"extensions": [{"uri": BROWSER, "params": {"methods": { op: {"method": "GET" if op in GETS else "POST", "endpoint": f"/agentenv/ext/browser/{op}"} for op in PAGE_OPS}}}]}} -#: A stand-in for Playwright MCP on the port and CDP address it's started with, whose one tool says what it was started -#: with and which session called it; started on an address ending /exits, it exits at once. -FAKE_PLAYWRIGHT_MCP = """import json, os, sys +#: A stand-in for Playwright MCP on the port and CDP address it's started with: browser_snapshot says what it was +#: started with and which session called it, and browser_wait_for waits. Started on an address ending /exits, it exits +#: at once; on one ending /silent-once, the first time it writes its pid to the marker it's given and never listens. +FAKE_PLAYWRIGHT_MCP = """import asyncio, json, os, sys, time from mcp.server.fastmcp import Context, FastMCP args = sys.argv[1:] endpoint, port = args[args.index("--cdp-endpoint") + 1], int(args[args.index("--port") + 1]) if endpoint.endswith("/exits"): sys.exit(3) +marker = args[args.index("--marker") + 1] +if endpoint.endswith("/silent-once") and not os.path.exists(marker): + open(marker, "w").write(str(os.getpid())) + time.sleep(3600) app = FastMCP("fake", host="127.0.0.1", port=port) @app.tool() def browser_snapshot(ctx: Context) -> str: return json.dumps({"pid": os.getpid(), "endpoint": endpoint, "session": id(ctx.session), "cwd": os.getcwd(), "env": sorted(os.environ)}) +@app.tool() +async def browser_wait_for(time: float) -> str: + await asyncio.sleep(time) + return "waited" app.run("streamable-http") """ #: A 2×1 PNG's first 24 bytes: its signature, then its IHDR's length, type, width and height. @@ -278,26 +288,9 @@ def test_playwright_mcp_attaches_to_the_browser_never_launches_one_and_ends_with @pytest.mark.asyncio async def test_playwright_mcp_starts_on_the_first_call_keeps_a_session_per_caller_and_starts_again_when_it_must( - monkeypatch, tmp_path): - enable_socket() - module = _load() - fake = tmp_path / "fake_playwright_mcp.py" - fake.write_text(FAKE_PLAYWRIGHT_MCP) - monkeypatch.setattr(module, "_command", lambda endpoint, port: [ - sys.executable, str(fake), "--cdp-endpoint", endpoint, "--port", str(port)]) - monkeypatch.setattr(module, "PLAYWRIGHT_MCP_OUTPUT", tmp_path / "out") - monkeypatch.setenv("HOME", str(tmp_path / "home")) + stand_in, monkeypatch, tmp_path): + _, playwright, endpoints, called = stand_in monkeypatch.setenv("AGENTENV_CONTROL_TOKEN", "secret") - endpoints = ["ws://127.0.0.1:18765/svc/chromium/cdp/one/devtools/browser"] - - async def endpoint() -> str: - return endpoints[-1] - - playwright = module._PlaywrightMcp(endpoint) - - async def called(caller: str): - with bound(ToolContext.as_caller(session=caller)): - return await playwright.call("browser_snapshot") async def snapshot(caller: str) -> dict: result = await called(caller) @@ -322,3 +315,87 @@ async def snapshot(caller: str) -> dict: assert moved["pid"] != first["pid"] and moved["endpoint"].endswith("/cdp/two/devtools/browser") assert revived["pid"] != moved["pid"] assert refused.isError and "exited at start with 3" in refused.content[0].text + + +@pytest.fixture +def stand_in(monkeypatch, tmp_path): + """A ``_PlaywrightMcp`` on the stand-in, its address the last of ``endpoints``, and a way to call it as a caller.""" + enable_socket() + module = _load() + fake = tmp_path / "fake_playwright_mcp.py" + fake.write_text(FAKE_PLAYWRIGHT_MCP) + monkeypatch.setattr(module, "_command", lambda endpoint, port: [ + sys.executable, str(fake), "--cdp-endpoint", endpoint, "--port", str(port), "--marker", str(tmp_path / "m")]) + monkeypatch.setattr(module, "PLAYWRIGHT_MCP_OUTPUT", tmp_path / "out") + monkeypatch.setenv("HOME", str(tmp_path / "home")) + endpoints = ["ws://127.0.0.1:18765/svc/chromium/cdp/one/devtools/browser"] + + async def endpoint() -> str: + return endpoints[-1] + + playwright = module._PlaywrightMcp(endpoint) + + async def called(caller: str, tool: str = "browser_snapshot", arguments: dict | None = None): + with bound(ToolContext.as_caller(session=caller)): + return await playwright.call(tool, **(arguments or {})) + + return module, playwright, endpoints, called + + +@pytest.mark.asyncio +async def test_a_call_playwright_mcp_ends_under_fails_then_and_isnt_sent_again(stand_in): + _, playwright, _, called = stand_in + try: + pid = json.loads((await called("ada")).content[0].text)["pid"] + waiting = asyncio.ensure_future(called("ada", "browser_wait_for", {"time": 60})) + await asyncio.sleep(1) + os.kill(pid, signal.SIGKILL) + ended = await asyncio.wait_for(waiting, 10) + after = await called("ada") + finally: + await playwright._stop() + + assert ended.isError and "ended during browser_wait_for, which may have acted" in ended.content[0].text + assert not after.isError and json.loads(after.content[0].text)["pid"] != pid + + +@pytest.mark.asyncio +async def test_a_callers_session_left_idle_is_closed_when_another_caller_comes(stand_in, monkeypatch): + module, playwright, _, called = stand_in + monkeypatch.setattr(module, "PLAYWRIGHT_MCP_SESSION_IDLE_S", 0.5) + try: + await called("ada") + ada = playwright._sessions["ada"] + await asyncio.sleep(1) + await called("bob") + await asyncio.wait_for(ada.owner, 10) + kept = sorted(playwright._sessions) + finally: + await playwright._stop() + + assert kept == ["bob"] and ada.close.is_set() + + +@pytest.mark.asyncio +async def test_a_start_that_never_listens_is_stopped_and_the_next_call_starts_afresh(stand_in, monkeypatch, tmp_path): + module, playwright, endpoints, called = stand_in + monkeypatch.setattr(module, "PLAYWRIGHT_MCP_START_S", 2.0) + endpoints.append("ws://127.0.0.1:18765/svc/chromium/cdp/silent-once") + try: + refused = await called("ada") + hung = int((tmp_path / "m").read_text()) + gone = _gone(hung) + answered = await called("ada") + finally: + await playwright._stop() + + assert refused.isError and "didn't listen within 2 s" in refused.content[0].text + assert gone and not answered.isError + + +def _gone(pid: int) -> bool: + try: + os.kill(pid, 0) + except ProcessLookupError: + return True + return False From 14709280f2b903cc7b04bd207e77ce02a158c478 Mon Sep 17 00:00:00 2001 From: Edgar Arakelyan Date: Wed, 11 Nov 2026 06:09:59 -0800 Subject: [PATCH 3/7] fix(drivers): a Playwright MCP call its caller cancels ends there too A caller cancelled while its call waited left the call to Playwright MCP running; it's cancelled and awaited now, and not sent again. Co-Authored-By: Claude Opus 5.5 --- src/agent_env/drivers/envs/page/driver.py | 5 +++++ tst/unit/drivers/page_driver_test.py | 19 +++++++++++++++++++ 2 files changed, 24 insertions(+) diff --git a/src/agent_env/drivers/envs/page/driver.py b/src/agent_env/drivers/envs/page/driver.py index 952e5251e..d688e2b84 100644 --- a/src/agent_env/drivers/envs/page/driver.py +++ b/src/agent_env/drivers/envs/page/driver.py @@ -465,6 +465,11 @@ async def call(self, tool: str, /, **arguments: Any) -> CallToolResult: exited = asyncio.ensure_future(process.wait()) try: await asyncio.wait({called, exited, held.owner}, return_when=asyncio.FIRST_COMPLETED) + except asyncio.CancelledError: + called.cancel() + with contextlib.suppress(asyncio.CancelledError, Exception): + await called + raise finally: exited.cancel() if not called.done(): diff --git a/tst/unit/drivers/page_driver_test.py b/tst/unit/drivers/page_driver_test.py index 353721cdb..f003257d8 100644 --- a/tst/unit/drivers/page_driver_test.py +++ b/tst/unit/drivers/page_driver_test.py @@ -359,6 +359,25 @@ async def test_a_call_playwright_mcp_ends_under_fails_then_and_isnt_sent_again(s assert not after.isError and json.loads(after.content[0].text)["pid"] != pid +@pytest.mark.asyncio +async def test_a_call_its_caller_cancels_ends_its_call_on_playwright_mcp_too(stand_in): + _, playwright, _, called = stand_in + try: + await called("ada") + waiting = asyncio.ensure_future(called("ada", "browser_wait_for", {"time": 60})) + await asyncio.sleep(1) + waiting.cancel() + with pytest.raises(asyncio.CancelledError): + await waiting + await asyncio.sleep(0.5) + left = [task for task in asyncio.all_tasks() if "call_tool" in repr(task.get_coro())] + after = await called("ada") + finally: + await playwright._stop() + + assert left == [] and not after.isError + + @pytest.mark.asyncio async def test_a_callers_session_left_idle_is_closed_when_another_caller_comes(stand_in, monkeypatch): module, playwright, _, called = stand_in From d03f174e2fc9f9ab6a44d74e532442a660035e57 Mon Sep 17 00:00:00 2001 From: Edgar Arakelyan Date: Wed, 11 Nov 2026 06:43:43 -0800 Subject: [PATCH 4/7] fix(drivers): every wait on Playwright MCP is bounded A call is bounded by PLAYWRIGHT_MCP_CALL_TIMEOUT_S, and one that times out or fails in transport closes its session, so the next call opens a fresh one rather than reusing a session already closing. Opening a session is bounded by PLAYWRIGHT_MCP_START_S, and a Playwright MCP that can't open one in that time is stopped, so the next call starts it afresh instead of every caller waiting on the one that hangs. Co-Authored-By: Claude Opus 5.5 --- src/agent_env/drivers/envs/page/driver.py | 25 ++++++++++--- tst/unit/drivers/page_driver_test.py | 44 ++++++++++++++++++++++- 2 files changed, 63 insertions(+), 6 deletions(-) diff --git a/src/agent_env/drivers/envs/page/driver.py b/src/agent_env/drivers/envs/page/driver.py index d688e2b84..6178486b3 100644 --- a/src/agent_env/drivers/envs/page/driver.py +++ b/src/agent_env/drivers/envs/page/driver.py @@ -17,6 +17,7 @@ import socket import string import tempfile +from datetime import timedelta from pathlib import Path from typing import Any, Awaitable, Callable, Optional @@ -106,7 +107,7 @@ KEPT_ENVIRONMENT = ("PATH", "HOME", "USER", "LOGNAME", "LANG", "LC_ALL", "TZ", "TMPDIR") #: Where Playwright MCP writes what it names itself, and its log. PLAYWRIGHT_MCP_OUTPUT = Path(tempfile.gettempdir()) / "agentenv-playwright-mcp" -#: How long Playwright MCP may take to listen, and one of its tools to answer. +#: How long Playwright MCP may take to listen or to open a session, and one of its tools to answer. PLAYWRIGHT_MCP_START_S = 30.0 PLAYWRIGHT_MCP_CALL_TIMEOUT_S = 300.0 #: How long a caller's session on Playwright MCP is kept with no call: a caller's own MCP session ends with no word to @@ -461,7 +462,8 @@ async def call(self, tool: str, /, **arguments: Any) -> CallToolResult: return _refused(e.message) except Exception as e: return _refused(f"Playwright MCP didn't answer {tool}: {type(e).__name__}: {e}") - called = asyncio.ensure_future(held.session.call_tool(tool, arguments)) + called = asyncio.ensure_future(held.session.call_tool( + tool, arguments, read_timeout_seconds=timedelta(seconds=PLAYWRIGHT_MCP_CALL_TIMEOUT_S))) exited = asyncio.ensure_future(process.wait()) try: await asyncio.wait({called, exited, held.owner}, return_when=asyncio.FIRST_COMPLETED) @@ -479,6 +481,7 @@ async def call(self, tool: str, /, **arguments: Any) -> CallToolResult: try: return called.result() except Exception as e: + held.close.set() return _refused(f"Playwright MCP didn't answer {tool}: {type(e).__name__}: {e}") finally: held.used_at = asyncio.get_running_loop().time() @@ -491,10 +494,14 @@ async def _session(self, caller: Optional[str]) -> tuple[_Session, asyncio.subpr if key != caller and now - held.used_at > PLAYWRIGHT_MCP_SESSION_IDLE_S]: self._sessions.pop(idle).close.set() held = self._sessions.get(caller) - if held is None or held.url != url or held.owner.done(): + if held is None or held.url != url or held.owner.done() or held.close.is_set(): if held is not None: held.close.set() - held = self._sessions[caller] = await _Session.open(url) + try: + held = self._sessions[caller] = await _Session.open(url) + except DeviceUnavailable: + await self._stop() + raise held.used_at = now return held, self._process @@ -547,10 +554,16 @@ def __init__(self, url: str) -> None: @classmethod async def open(cls, url: str) -> _Session: + """A session on Playwright MCP at ``url``; DeviceUnavailable when it doesn't open one in time, wedged.""" held = cls(url) ready: asyncio.Future = asyncio.get_running_loop().create_future() held.owner = asyncio.create_task(held._own(ready)) - held.session = await ready + try: + held.session = await asyncio.wait_for(ready, PLAYWRIGHT_MCP_START_S) + except asyncio.TimeoutError: + held.close.set() + held.owner.cancel() + raise DeviceUnavailable(f"Playwright MCP didn't open a session within {PLAYWRIGHT_MCP_START_S:.0f} s") return held async def _own(self, ready: asyncio.Future) -> None: @@ -560,6 +573,8 @@ async def _own(self, ready: asyncio.Future) -> None: streamable_http_client(self.url, http_client=http) as (read, write, _), \ ClientSession(read, write) as session: await session.initialize() + if ready.done(): + return ready.set_result(session) await self.close.wait() except BaseException as e: diff --git a/tst/unit/drivers/page_driver_test.py b/tst/unit/drivers/page_driver_test.py index f003257d8..1af913a9d 100644 --- a/tst/unit/drivers/page_driver_test.py +++ b/tst/unit/drivers/page_driver_test.py @@ -38,7 +38,8 @@ for op in PAGE_OPS}}}]}} #: A stand-in for Playwright MCP on the port and CDP address it's started with: browser_snapshot says what it was #: started with and which session called it, and browser_wait_for waits. Started on an address ending /exits, it exits -#: at once; on one ending /silent-once, the first time it writes its pid to the marker it's given and never listens. +#: at once; on one ending /silent-once, the first time it writes its pid to the marker it's given and never listens; on +#: one ending /wedged-once, the first time it listens but never answers. FAKE_PLAYWRIGHT_MCP = """import asyncio, json, os, sys, time from mcp.server.fastmcp import Context, FastMCP args = sys.argv[1:] @@ -49,6 +50,13 @@ if endpoint.endswith("/silent-once") and not os.path.exists(marker): open(marker, "w").write(str(os.getpid())) time.sleep(3600) +if endpoint.endswith("/wedged-once") and not os.path.exists(marker): + import socket + open(marker, "w").write(str(os.getpid())) + held = socket.create_server(("127.0.0.1", port)) + accepted = [] + while True: + accepted.append(held.accept()) app = FastMCP("fake", host="127.0.0.1", port=port) @app.tool() def browser_snapshot(ctx: Context) -> str: @@ -418,3 +426,37 @@ def _gone(pid: int) -> bool: except ProcessLookupError: return True return False + + +@pytest.mark.asyncio +async def test_a_call_slower_than_its_bound_is_refused_and_its_session_opened_afresh(stand_in, monkeypatch): + module, playwright, _, called = stand_in + monkeypatch.setattr(module, "PLAYWRIGHT_MCP_CALL_TIMEOUT_S", 1.0) + try: + await called("ada") + first = playwright._sessions["ada"] + slow = await asyncio.wait_for(called("ada", "browser_wait_for", {"time": 30}), 10) + after = await called("ada") + finally: + await playwright._stop() + + assert slow.isError and "didn't answer browser_wait_for" in slow.content[0].text + assert first.close.is_set() and not after.isError + + +@pytest.mark.asyncio +async def test_a_playwright_mcp_that_never_opens_a_session_is_stopped_and_the_next_call_starts_afresh( + stand_in, monkeypatch, tmp_path): + module, playwright, endpoints, called = stand_in + monkeypatch.setattr(module, "PLAYWRIGHT_MCP_START_S", 2.0) + endpoints.append("ws://127.0.0.1:18765/svc/chromium/cdp/wedged-once") + try: + refused = await asyncio.wait_for(called("ada"), 20) + wedged = int((tmp_path / "m").read_text()) + gone = _gone(wedged) + answered = await called("ada") + finally: + await playwright._stop() + + assert refused.isError and "didn't open a session within 2 s" in refused.content[0].text + assert gone and not answered.isError From 63269fce1bd6ccb4c45655c1751e11cf1c6628a4 Mon Sep 17 00:00:00 2001 From: Edgar Arakelyan Date: Wed, 11 Nov 2026 06:56:30 -0800 Subject: [PATCH 5/7] refactor(drivers): name Playwright MCP's poll and stop waits Co-Authored-By: Claude Opus 5.5 --- src/agent_env/drivers/envs/page/driver.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/src/agent_env/drivers/envs/page/driver.py b/src/agent_env/drivers/envs/page/driver.py index 6178486b3..7cd88edbe 100644 --- a/src/agent_env/drivers/envs/page/driver.py +++ b/src/agent_env/drivers/envs/page/driver.py @@ -110,6 +110,10 @@ #: How long Playwright MCP may take to listen or to open a session, and one of its tools to answer. PLAYWRIGHT_MCP_START_S = 30.0 PLAYWRIGHT_MCP_CALL_TIMEOUT_S = 300.0 +#: How often a start checks whether Playwright MCP listens yet, and how long a stop waits for it to end on SIGTERM +#: before it's killed. +PLAYWRIGHT_MCP_POLL_S = 0.1 +PLAYWRIGHT_MCP_STOP_S = 10.0 #: How long a caller's session on Playwright MCP is kept with no call: a caller's own MCP session ends with no word to #: the driver, and each holds a connection to the browser. PLAYWRIGHT_MCP_SESSION_IDLE_S = 3600.0 @@ -534,7 +538,7 @@ async def _stop(self) -> None: if self._process is not None and self._process.returncode is None: self._process.terminate() with contextlib.suppress(asyncio.TimeoutError): - await asyncio.wait_for(self._process.wait(), 10) + await asyncio.wait_for(self._process.wait(), PLAYWRIGHT_MCP_STOP_S) if self._process.returncode is None: self._process.kill() await self._process.wait() @@ -615,7 +619,7 @@ async def _listening(port: int, process: asyncio.subprocess.Process) -> None: _, writer = await asyncio.open_connection("127.0.0.1", port) writer.close() return - await asyncio.sleep(0.1) + await asyncio.sleep(PLAYWRIGHT_MCP_POLL_S) raise DeviceUnavailable(f"Playwright MCP didn't listen within {PLAYWRIGHT_MCP_START_S:.0f} s") From f99b1c0b1e1603d2b6485561d51cca7924e09c57 Mon Sep 17 00:00:00 2001 From: Edgar Arakelyan Date: Wed, 11 Nov 2026 11:00:15 -0800 Subject: [PATCH 6/7] refactor(drivers): Playwright MCP in a module of its own, pinned by a script - scripts/upgrade_playwright_mcp.py pins a version: the install's downloads in env.toml, every npm package Playwright MCP needs checked against npm's integrity, and the manifest from that version's tools/list. Run at 0.0.84 it writes what was committed. - install-playwright-mcp.sh lays out whatever packages were downloaded, by the name each package.json gives, and fails when a dependency isn't there at the version it names. - The unit test pins the manifest's tool names, so an upgrade that adds a tool fails until someone looks at whether to hide it, and checks the manifest is of the version env.toml downloads. - Playwright MCP's process and sessions move to page/playwright_mcp.py. - ForwardedTool.listed builds a forwarded tool from another server's tools/list entry; the five drivers that forward tools use it. - mcp_port() is the port create_fastmcp_app serves on, which the page driver's CDP origin now reads instead of repeating its default. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 2 +- .../src/agentenv_protocol/__init__.py | 2 + .../agent_env_environment.py | 15 +- .../tests/test_agent_env_environment.py | 10 + .../tests/test_tool_annotations.py | 14 + scripts/upgrade_playwright_mcp.py | 153 +++++++++++ src/agent_env/drivers/README.md | 14 +- .../drivers/envs/agent-desktop/driver.py | 6 +- src/agent_env/drivers/envs/aside/driver.py | 9 +- .../drivers/envs/codex-computer-use/driver.py | 9 +- .../drivers/envs/cua-driver/driver.py | 7 +- src/agent_env/drivers/envs/page/driver.py | 245 +----------------- src/agent_env/drivers/envs/page/env.toml | 5 +- .../envs/page/install-playwright-mcp.sh | 32 ++- .../drivers/envs/page/playwright_mcp.py | 245 ++++++++++++++++++ tst/unit/drivers/page_driver_test.py | 65 +++-- 16 files changed, 546 insertions(+), 287 deletions(-) create mode 100644 scripts/upgrade_playwright_mcp.py create mode 100644 src/agent_env/drivers/envs/page/playwright_mcp.py diff --git a/AGENTS.md b/AGENTS.md index 2188ae130..25d7ba92d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -139,7 +139,7 @@ and deserialized through a registry. | `agents/` | The `agents` bundle: the agents agent-env ships, one folder each under `agents/agents/`, with a smoke task each under `agents/tasks/`. Its Claude Code, `@local/agentenv-framework/agents/claude-code`, is the default agent. An agent's Python packages are its folder's `requirements.txt`, installed in its image, never in agent-env's own environment. A run that pulls one of its agents reads the whole bundle, so `tst/unit/bundle/shipped_bundles_test.py` parses and checks every shipped bundle. | | `computers/` | The `computers` bundle: the base computers agent-env ships as `computer` artifacts, `ubuntu-24.04` (XFCE on Xvfb, supervisord as the session user, settings as `AGENTENV_SETTING_`), and `omarchy-4.5`, a disk alone built on the composing machine from Omarchy's own VM image (its `[forms.disk]` `source`) by `disk/omarchy/`'s recipe, its services systemd user units of the graphical session that start through `with-seed`, booted by the launcher as a `linux-desktop` guest (`env/envs/computer/utils/omarchy_disk.py`, on `thin_disk.py`, which Windows' disks use too), aliased through `[artifacts]`. It has no tasks either. `disk/linux/` is the Linux disk form's recipe: `boot/`, whose Dockerfile makes a disk form (a GPT disk of an EFI system partition holding one UKI, Ubuntu's kernel with a busybox initramfs whose init reads a NoCloud seed and `switch_root`s into the image's entrypoint, and an ext4 root filesystem that is a container image's, plus `/etc/agentenv/vm/image.env` from its OCI config), `tools/`, the disk tools that make it unprivileged on the builder's own platform (`mkfs.ext4 -d`), and `launcher/`, the QEMU launcher that boots one under KVM. A base's `[forms.disk]` with no image is made from its container form; the composer makes each arch's disk from its composed container, keeping the base disk's EFI system partition, keyed by the container's key, the base disk and the recipe; gate 3 probes it where a provider boots disks natively (`disk_image_platforms()`), and a deploy's first boot with KVM probes one left `unverified`; a deploy naming `form = "disk"` boots it through `create_vm(image=..., env=...)`, its settings in the seed. | | `browsers/` | The `browsers` bundle: the website browsers agent-env ships, today `chromium`, a `computer_app` env (Chrome for Testing at the build Playwright pins, for amd64 and arm64, set up for unattended use by managed policies in `/etc/opt/chrome_for_testing/policies/managed`, opened at boot on its `start_urls`, its profile its data). All of it, its pin, install step, policies and command, is in `envs/chromium/chromium.py`; `tst/integration/computer/chromium_app_test.py` runs it. Like `drivers`, it has no tasks. | -| `drivers/` | The `drivers` bundle: the computer drivers agent-env ships as `computer_driver` envs, the reference driver (X11 on Linux, recording through `x11grab`) aliased `reference`, the page driver aliased `page`, a browser computer's, whose screen is the browser's active tab and which serves Playwright MCP's `browser_*` tools beside `computer_*` from a Playwright MCP it runs attached to the browser over a CDP address the relay issues it (`browser_evaluate` and `browser_run_code_unsafe` hidden by default), and trycua's cua-driver aliased `cua-driver` (its daemon from the pinned PyPI wheel, its own instructions, tools and skill resources served to agents through `@server_instructions`, `forwarded_tools` and `@resource`, its tools beside `computer_*`, `computer/v1` on its desktop-scope calls, its skill pack leading the computer skill; `tst/integration/computer/cua_driver_test.py` runs it). It has no tasks: a bundle with neither tasks nor evals is refused unless its `aliases.toml` names something for other bundles to use. The shipped `desktop` example composes it with the `ubuntu-24.04` base into a computer env, which `tst/integration/computer/` deploys, probes, records and has an agent use. | +| `drivers/` | The `drivers` bundle: the computer drivers agent-env ships as `computer_driver` envs, the reference driver (X11 on Linux, recording through `x11grab`) aliased `reference`, the page driver aliased `page`, a browser computer's, whose screen is the browser's active tab and which serves Playwright MCP's `browser_*` tools beside `computer_*` from a Playwright MCP it runs attached to the browser over a CDP address the relay issues it (`browser_evaluate` and `browser_run_code_unsafe` hidden by default; `scripts/upgrade_playwright_mcp.py` pins its version), and trycua's cua-driver aliased `cua-driver` (its daemon from the pinned PyPI wheel, its own instructions, tools and skill resources served to agents through `@server_instructions`, `forwarded_tools` and `@resource`, its tools beside `computer_*`, `computer/v1` on its desktop-scope calls, its skill pack leading the computer skill; `tst/integration/computer/cua_driver_test.py` runs it). It has no tasks: a bundle with neither tasks nor evals is refused unless its `aliases.toml` names something for other bundles to use. The shipped `desktop` example composes it with the `ubuntu-24.04` base into a computer env, which `tst/integration/computer/` deploys, probes, records and has an agent use. | | `examples/` | The example bundles agent-env ships (`hello`, and `echo-mcp`, which needs Docker, and `desktop`, a computer env of `ubuntu-24.04` and `reference` with a probe task and an agent task). Tests run them, but they are examples for users, not fixtures. Every shipped bundle is registered under `agent_env.bundles` in `pyproject.toml`, ` = ""` shipping `src///`. | ### Training helpers diff --git a/packages/agentenv-protocol/src/agentenv_protocol/__init__.py b/packages/agentenv-protocol/src/agentenv_protocol/__init__.py index 61f9cbab9..43dd1659c 100644 --- a/packages/agentenv-protocol/src/agentenv_protocol/__init__.py +++ b/packages/agentenv-protocol/src/agentenv_protocol/__init__.py @@ -21,6 +21,7 @@ extension, get_data, injecting, + mcp_port, reset_data, tool, ) @@ -94,6 +95,7 @@ "AgentEnvStarletteApplication", "ForwardedTool", "create_fastmcp_app", + "mcp_port", "environment_card", "reset_data", "add_data", diff --git a/packages/agentenv-protocol/src/agentenv_protocol/agent_env_environment.py b/packages/agentenv-protocol/src/agentenv_protocol/agent_env_environment.py index 90d50fd51..360005a78 100644 --- a/packages/agentenv-protocol/src/agentenv_protocol/agent_env_environment.py +++ b/packages/agentenv-protocol/src/agentenv_protocol/agent_env_environment.py @@ -254,6 +254,14 @@ class ForwardedTool(NamedTuple): output_schema: dict | None = None hidden_by_default: bool = False + @classmethod + def listed(cls, tool: dict, call: Callable[..., Any], *, hidden_by_default: bool = False) -> "ForwardedTool": + """The tool another server's ``tools/list`` lists as ``tool``, called through ``call``: its name, description, + input and output schemas and annotations as listed.""" + return cls(EnvironmentTool(name=tool["name"], description=tool.get("description"), + inputSchema=tool["inputSchema"]), + call, tool.get("annotations"), tool.get("outputSchema"), hidden_by_default) + class _ResourceMarker(NamedTuple): uri: str @@ -595,6 +603,11 @@ def _add_route(self, app: Any, path: str, methods: list, handler: Callable) -> N app.router.routes.append(Route(path, handler, methods=methods)) +def mcp_port() -> int: + """The port ``create_fastmcp_app`` serves on: ``MCP_PORT``'s, else 18765.""" + return int(os.environ.get("MCP_PORT", "18765")) + + def create_fastmcp_app(handler: Any, *, card: EnvironmentCard, **fastmcp_kwargs: Any) -> Any: """FastMCP app wired for the agent-env deploy contract, with ``handler`` mounted under ``card``; returned un-served.""" try: @@ -603,7 +616,7 @@ def create_fastmcp_app(handler: Any, *, card: EnvironmentCard, **fastmcp_kwargs: raise ImportError("create_fastmcp_app requires the 'mcp' package: pip install 'mcp>=1.25,<2'") from e app = FastMCP(card.name, **{"instructions": _instructions(handler), **fastmcp_kwargs}) app.settings.host = os.environ.get("MCP_HOST", "0.0.0.0") - app.settings.port = int(os.environ.get("MCP_PORT", "18765")) + app.settings.port = mcp_port() app.settings.transport_security.enable_dns_rebinding_protection = False AgentEnvFastMCPApplication(card, handler).add_routes_to_app(app) return app diff --git a/packages/agentenv-protocol/tests/test_agent_env_environment.py b/packages/agentenv-protocol/tests/test_agent_env_environment.py index 7b8897e46..1c6b60018 100644 --- a/packages/agentenv-protocol/tests/test_agent_env_environment.py +++ b/packages/agentenv-protocol/tests/test_agent_env_environment.py @@ -34,6 +34,7 @@ agent_skill, create_fastmcp_app, environment_card, + mcp_port, extension, get_data, reset_data, @@ -1192,6 +1193,15 @@ class _Child(_ServeEnv): assert "items_add_item" in app.tools +def test_an_app_serves_on_the_port_mcp_port_names_else_18765(monkeypatch): + monkeypatch.delenv("MCP_PORT", raising=False) + default = (mcp_port(), create_fastmcp_app(_ToolHandler(), card=EnvironmentCard(name="items")).settings.port) + monkeypatch.setenv("MCP_PORT", "40123") + named = (mcp_port(), create_fastmcp_app(_ToolHandler(), card=EnvironmentCard(name="items")).settings.port) + + assert default == (18765, 18765) and named == (40123, 40123) + + def test_create_fastmcp_app_without_mcp_raises_install_hint(monkeypatch): monkeypatch.setitem(sys.modules, "mcp.server.fastmcp", None) with pytest.raises(ImportError, match="requires the 'mcp' package"): diff --git a/packages/agentenv-protocol/tests/test_tool_annotations.py b/packages/agentenv-protocol/tests/test_tool_annotations.py index 6e2248d35..7c25c0256 100644 --- a/packages/agentenv-protocol/tests/test_tool_annotations.py +++ b/packages/agentenv-protocol/tests/test_tool_annotations.py @@ -268,6 +268,20 @@ async def test_a_forwarded_tool_is_listed_with_its_own_schema_and_called_with_it assert {t.name: t.inputSchema for t in env.mounted_card().capabilities.tools}["click"] == _CLICK +def test_a_tool_another_server_lists_is_forwarded_as_it_lists_it(): + async def call(**arguments) -> CallToolResult: + return CallToolResult(content=[]) + + click = ForwardedTool.listed({"name": "click", "title": "Click", "description": "Click an element.", + "inputSchema": _CLICK, "outputSchema": _FOUND, "annotations": {"readOnlyHint": False}}, + call) + find = ForwardedTool.listed({"name": "find", "inputSchema": {"type": "object"}}, call, hidden_by_default=True) + + assert click == ForwardedTool(EnvironmentTool(name="click", description="Click an element.", inputSchema=_CLICK), + call, {"readOnlyHint": False}, _FOUND) + assert find == ForwardedTool(EnvironmentTool(name="find", inputSchema={"type": "object"}), call, None, None, True) + + _OTHER_FILES = [ {"uri": "skill://other/SKILL.md", "name": "other", "description": "The skill.", "mimeType": "text/markdown"}, {"uri": "skill://other/LINUX.md", "name": "LINUX.md", "mimeType": "text/markdown"}, diff --git a/scripts/upgrade_playwright_mcp.py b/scripts/upgrade_playwright_mcp.py new file mode 100644 index 000000000..2dcd97843 --- /dev/null +++ b/scripts/upgrade_playwright_mcp.py @@ -0,0 +1,153 @@ +"""Pin the page driver's Playwright MCP to a version: its install's downloads in env.toml (Node's Linux tarballs and +every npm package Playwright MCP needs, each with its sha256) and its tools, as that version lists them with ``--caps +vision``, in playwright-mcp-tools.json. Run from anywhere, with network: + + python scripts/upgrade_playwright_mcp.py 0.0.85 [--node 24.22.2] + +Node keeps the version env.toml pins unless ``--node`` names another. The install unpacks the exact versions written +here, so a package that names a range for a dependency is refused. The tools that are new or gone are printed: look at +each new one, hide it in playwright_mcp.py's HIDDEN_TOOLS if it runs the agent's own code, and add it to the unit test's +REVIEWED_TOOLS.""" +from __future__ import annotations + +import argparse +import base64 +import hashlib +import io +import json +import platform +import re +import subprocess +import sys +import tarfile +import tempfile +import urllib.parse +import urllib.request +from pathlib import Path + +PAGE = Path(__file__).resolve().parent.parent / "src" / "agent_env" / "drivers" / "envs" / "page" +ENV_TOML = PAGE / "env.toml" +MANIFEST = PAGE / "playwright-mcp-tools.json" +REGISTRY = "https://registry.npmjs.org" +NODE_DIST = "https://nodejs.org/dist" +PACKAGE = "@playwright/mcp" +CAPS = ["vision"] +#: Node's Linux builds, by the architecture a computer names. +NODE_LINUX = {"amd64": "linux-x64", "arm64": "linux-arm64"} +EXACT = re.compile(r"\d+\.\d+\.\d+(-[0-9A-Za-z.-]+)?") +TIMEOUT_S = 120 + + +def main(argv: list[str]) -> None: + parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) + parser.add_argument("version", help="the @playwright/mcp version to pin, such as 0.0.85") + parser.add_argument("--node", help="the Node version to pin; by default the one env.toml pins") + args = parser.parse_args(argv) + node = args.node or re.search(r"nodejs\.org/dist/v([\d.]+)/", ENV_TOML.read_text()).group(1) + sums = _node_sums(node) + packages = _packages(args.version) + downloads = [{"url": f"{NODE_DIST}/v{node}/node-v{node}-{build}.tar.gz", "sha256": sums[build], + "to": "node.tar.gz", "arch": arch} for arch, build in NODE_LINUX.items()] + downloads += [{"url": p["url"], "sha256": hashlib.sha256(p["bytes"]).hexdigest(), "to": _download_name(p["name"])} + for p in packages] + tools = _listed_tools(node, sums, packages) + before = {t["name"] for t in json.loads(MANIFEST.read_text(encoding="utf-8"))["tools"]} + _write_downloads(downloads) + MANIFEST.write_text(json.dumps({"version": args.version, "caps": CAPS, "tools": tools}, indent=1) + "\n", + encoding="utf-8") + after = {t["name"] for t in tools} + print(f"pinned @playwright/mcp {args.version} on Node {node}: " + + ", ".join(f"{p['name']} {p['version']}" for p in packages)) + print("new tools, to look at:", ", ".join(sorted(after - before)) or "none") + print("tools gone:", ", ".join(sorted(before - after)) or "none") + + +def _packages(version: str) -> list[dict]: + """@playwright/mcp at ``version`` and every package it needs, each with its tarball, checked against npm's + integrity.""" + wanted, found = [(PACKAGE, version)], {} + while wanted: + name, version = wanted.pop(0) + if name in found: + if found[name]["version"] != version: + raise SystemExit(f"{name} is needed at both {found[name]['version']} and {version}") + continue + listed = json.loads(_get(f"{REGISTRY}/{urllib.parse.quote(name, safe='@')}/{version}")) + tarball = _get(listed["dist"]["tarball"]) + algorithm, digest = listed["dist"]["integrity"].split("-", 1) + if base64.b64encode(hashlib.new(algorithm, tarball).digest()).decode() != digest: + raise SystemExit(f"{name} {version}'s tarball doesn't match npm's integrity") + found[name] = {"name": name, "version": version, "url": listed["dist"]["tarball"], "bytes": tarball} + for dependency, spec in (listed.get("dependencies") or {}).items(): + if not EXACT.fullmatch(spec): + raise SystemExit(f"{name} {version} needs {dependency} {spec}, a range; the install takes exact versions") + wanted.append((dependency, spec)) + return list(found.values()) + + +def _node_sums(node: str) -> dict[str, str]: + """Each of Node ``node``'s builds' sha256 by its name, such as linux-x64, from nodejs.org's SHASUMS256.txt.""" + sums = {} + for line in _get(f"{NODE_DIST}/v{node}/SHASUMS256.txt").decode().splitlines(): + digest, file = line.split() + if (m := re.fullmatch(rf"node-v{re.escape(node)}-(.+)\.tar\.gz", file)): + sums[m.group(1)] = digest + return sums + + +def _listed_tools(node: str, sums: dict[str, str], packages: list[dict]) -> list[dict]: + """Playwright MCP's ``tools/list``, run on this machine's Node of the same version, laid out as the install lays it + out, attached to no browser.""" + build = _host() + archive = _get(f"{NODE_DIST}/v{node}/node-v{node}-{build}.tar.gz") + if hashlib.sha256(archive).hexdigest() != sums[build]: + raise SystemExit(f"Node {node}'s {build} build doesn't match SHASUMS256.txt") + with tempfile.TemporaryDirectory() as work: + root = Path(work) + with tarfile.open(fileobj=io.BytesIO(archive)) as tar: + tar.extractall(root / "node", filter="data") + executable = next((root / "node").glob("node-*/bin/node")) + for package in packages: + with tarfile.open(fileobj=io.BytesIO(package["bytes"])) as tar: + for member in tar.getmembers(): + member.name = member.name.split("/", 1)[1] if "/" in member.name else "" + tar.extractall(root / "node_modules" / package["name"], + members=[m for m in tar.getmembers() if m.name], filter="data") + messages = [{"jsonrpc": "2.0", "id": 1, "method": "initialize", + "params": {"protocolVersion": "2025-06-18", "capabilities": {}, + "clientInfo": {"name": "upgrade_playwright_mcp", "version": "0"}}}, + {"jsonrpc": "2.0", "method": "notifications/initialized"}, + {"jsonrpc": "2.0", "id": 2, "method": "tools/list"}] + out = subprocess.run([str(executable), str(root / "node_modules" / PACKAGE / "cli.js"), "--cdp-endpoint", + "http://127.0.0.1:9/", "--caps", ",".join(CAPS)], capture_output=True, text=True, + input="\n".join(map(json.dumps, messages)) + "\n", timeout=TIMEOUT_S, check=True).stdout + return next(r["result"]["tools"] for r in map(json.loads, out.splitlines()) if r.get("id") == 2) + + +def _host() -> str: + """This machine's Node build, such as darwin-arm64.""" + machine = {"x86_64": "x64", "amd64": "x64", "aarch64": "arm64", "arm64": "arm64"}[platform.machine().lower()] + return f"{sys.platform}-{machine}" + + +def _download_name(package: str) -> str: + return package.lstrip("@").replace("/", "-") + ".tgz" + + +def _write_downloads(downloads: list[dict]) -> None: + lines = "".join(" { " + ", ".join(f'{key} = "{value}"' for key, value in d.items()) + " },\n" for d in downloads) + text, count = re.subn(r"^downloads = \[\n.*?^\]\n", lambda _: f"downloads = [\n{lines}]\n", ENV_TOML.read_text(), + count=1, flags=re.MULTILINE | re.DOTALL) + if count != 1: + raise SystemExit(f"{ENV_TOML} has no downloads list to replace") + ENV_TOML.write_text(text) + + +def _get(url: str) -> bytes: + with urllib.request.urlopen(urllib.request.Request(url, headers={"User-Agent": "agentenv-framework"}), + timeout=TIMEOUT_S) as response: + return response.read() + + +if __name__ == "__main__": + main(sys.argv[1:]) diff --git a/src/agent_env/drivers/README.md b/src/agent_env/drivers/README.md index 60759ff84..54e196a3a 100644 --- a/src/agent_env/drivers/README.md +++ b/src/agent_env/drivers/README.md @@ -21,16 +21,18 @@ a tab that paints nothing shows white. It serves the page group too: `computer_n it. The browser ending on its own fails the computer for good: its ready check is `final`, and its tools answer with why. Each call it makes to the browser for an agent's tool carries that call, its tool, id, arguments and the caller's role, in the `AgentEnv-Tool-Call` and `AgentEnv-Role` headers, so the browser's log credits the action to it. Name it as `page`, or by its id, `@local/agentenv-framework/drivers/page`. -Beside those it serves Playwright MCP's `browser_*` tools (0.0.84, `--caps vision`, on Node 24), as -`playwright-mcp-tools.json` lists them. Its install lays Node and Playwright MCP's three npm packages out from pinned -downloads, and it runs Playwright MCP itself, attached to the browser over a CDP address the browser's relay issues to +Beside those it serves Playwright MCP's `browser_*` tools (with `--caps vision`), as `playwright-mcp-tools.json` lists +them, from `playwright_mcp.py`. Its install lays Node and every npm package Playwright MCP needs out from pinned +downloads, refusing a package whose dependencies aren't all there, and it runs Playwright MCP itself, attached to the browser over a CDP address the browser's relay issues to the user `playwright-mcp`, so the guard applies to it as to any CDP client; it never starts a browser of its own. Each caller gets a session of its own there. It starts on the first call, again once it has exited, and again on a fresh address after the browser's data is reset; it ends with the driver, gets none of the driver's environment, and its file access is `~/Uploads`. `browser_evaluate` and `browser_run_code_unsafe`, which run the agent's own code, are hidden by -default: a task's tool access shows them by naming them. To move to another version, update the downloads in -`env.toml` and regenerate the manifest from that version's `tools/list`; the browser computer's integration test checks -the two agree. +default: a task's tool access shows them by naming them. To move to another version, run +`python scripts/upgrade_playwright_mcp.py `, which pins the downloads in `env.toml` and writes the manifest from +that version's `tools/list`, and look at each new tool it prints: hide it in `HIDDEN_TOOLS` if it runs the agent's own +code, and add it to the unit test's `REVIEWED_TOOLS`, which fails until you do. The browser computer's integration test +checks the installed version lists what the manifest does. A run that names it writes it: its files, what its class declares for each platform it supports, read without `mss`, `python-xlib` or PyAV installed, and how a computer installs it. Its `env.toml`'s `[install.linux]` has a diff --git a/src/agent_env/drivers/envs/agent-desktop/driver.py b/src/agent_env/drivers/envs/agent-desktop/driver.py index 67c852676..7cab2931d 100644 --- a/src/agent_env/drivers/envs/agent-desktop/driver.py +++ b/src/agent_env/drivers/envs/agent-desktop/driver.py @@ -28,7 +28,7 @@ from mcp.client.stdio import stdio_client from mcp.types import CallToolResult, ImageContent, TextContent -from agentenv_protocol import EnvironmentTool, ForwardedTool, computer, server_instructions +from agentenv_protocol import ForwardedTool, computer, server_instructions from agentenv_protocol.computer import ComputerSettings, ReadyCheck, computer_driver @@ -108,9 +108,7 @@ def __init__(self) -> None: self._skill: tuple[str, str, dict[str, str]] | None = None def forwarded_tools(self) -> list[ForwardedTool]: - return [ForwardedTool(EnvironmentTool(name=t["name"], description=t["description"], inputSchema=t["inputSchema"]), - functools.partial(self._forward, t["name"]), t.get("annotations"), t.get("outputSchema")) - for t in _manifest()["tools"]] + return [ForwardedTool.listed(t, functools.partial(self._forward, t["name"])) for t in _manifest()["tools"]] @server_instructions def agent_desktop_instructions(self) -> str: diff --git a/src/agent_env/drivers/envs/aside/driver.py b/src/agent_env/drivers/envs/aside/driver.py index 81ca5329f..1418bd2b5 100644 --- a/src/agent_env/drivers/envs/aside/driver.py +++ b/src/agent_env/drivers/envs/aside/driver.py @@ -28,7 +28,7 @@ from mcp.types import CONNECTION_CLOSED, CallToolResult, InitializeResult, TextContent from pydantic import Field -from agentenv_protocol import EnvironmentTool, ForwardedTool, ToolContext, server_instructions +from agentenv_protocol import ForwardedTool, ToolContext, server_instructions from agentenv_protocol.computer import ComputerSettings, MacOSDriver, ReadyCheck, computer_driver HERE = Path(__file__).resolve().parent @@ -89,10 +89,9 @@ def __init__(self) -> None: self._closing: set[asyncio.Task] = set() def forwarded_tools(self) -> list[ForwardedTool]: - return [ForwardedTool(EnvironmentTool(name=t["name"], description=t["description"], - inputSchema=t["inputSchema"]), - functools.partial(self._forward, t["name"]), - {**(t.get("annotations") or {}), **ANNOTATIONS.get(t["name"], {})}) + return [ForwardedTool.listed({**t, "annotations": {**(t.get("annotations") or {}), + **ANNOTATIONS.get(t["name"], {})}}, + functools.partial(self._forward, t["name"])) for t in _manifest()["tools"]] @server_instructions diff --git a/src/agent_env/drivers/envs/codex-computer-use/driver.py b/src/agent_env/drivers/envs/codex-computer-use/driver.py index 969d6c11c..00dc813ea 100644 --- a/src/agent_env/drivers/envs/codex-computer-use/driver.py +++ b/src/agent_env/drivers/envs/codex-computer-use/driver.py @@ -39,7 +39,7 @@ from mcp.types import (CONNECTION_CLOSED, CallToolResult, ElicitRequestParams, ElicitResult, InitializeResult, TextContent) -from agentenv_protocol import EnvironmentTool, ForwardedTool, ToolContext, computer, server_instructions +from agentenv_protocol import ForwardedTool, ToolContext, computer, server_instructions from agentenv_protocol.computer import ComputerSettings, ReadyCheck, computer_driver logger = logging.getLogger(__name__) @@ -155,10 +155,9 @@ def __init__(self) -> None: self._closing: set[asyncio.Task] = set() def forwarded_tools(self) -> list[ForwardedTool]: - return [ForwardedTool(EnvironmentTool(name=t["name"], description=t["description"], - inputSchema=_one_object(t["inputSchema"])), - functools.partial(self._forward, t["name"]), - {**(t.get("annotations") or {}), **ANNOTATIONS.get(t["name"], {})}) + return [ForwardedTool.listed({**t, "inputSchema": _one_object(t["inputSchema"]), + "annotations": {**(t.get("annotations") or {}), **ANNOTATIONS.get(t["name"], {})}}, + functools.partial(self._forward, t["name"])) for t in _manifest()["tools"] if t["name"] in FORWARDED] @server_instructions diff --git a/src/agent_env/drivers/envs/cua-driver/driver.py b/src/agent_env/drivers/envs/cua-driver/driver.py index 137dad4e4..e9db3f4e4 100644 --- a/src/agent_env/drivers/envs/cua-driver/driver.py +++ b/src/agent_env/drivers/envs/cua-driver/driver.py @@ -40,7 +40,7 @@ from Xlib import display as x_display from Xlib.error import ConnectionClosedError, DisplayConnectionError -from agentenv_protocol import EnvironmentTool, ForwardedTool, ToolContext, resource, server_instructions +from agentenv_protocol import ForwardedTool, ToolContext, resource, server_instructions from agentenv_protocol.computer import ( PRINTABLE_KEYS, ComputerDriverEnvironment, @@ -158,9 +158,8 @@ def __init__(self) -> None: self._reset_bind: dict | None = None def forwarded_tools(self) -> list[ForwardedTool]: - return [ForwardedTool(EnvironmentTool(name=t["name"], description=t["description"], inputSchema=t["inputSchema"]), - functools.partial(self._forward, t["name"], not (t["annotations"] or {}).get("readOnlyHint")), - t["annotations"] or None, t.get("outputSchema")) + return [ForwardedTool.listed(t, functools.partial(self._forward, t["name"], + not (t["annotations"] or {}).get("readOnlyHint"))) for t in _manifest()["tools"]] @resource("skill://cua-driver/{file}", listed=lambda env: _manifest()["resources"]) diff --git a/src/agent_env/drivers/envs/page/driver.py b/src/agent_env/drivers/envs/page/driver.py index 574719e92..3f8b87f87 100644 --- a/src/agent_env/drivers/envs/page/driver.py +++ b/src/agent_env/drivers/envs/page/driver.py @@ -8,25 +8,15 @@ import asyncio import base64 -import contextlib -import functools -import json -import logging import math -import os -import socket import string -import tempfile -from datetime import timedelta from pathlib import Path -from typing import Any, Awaitable, Callable, Optional +from typing import Any, Optional import httpx -from mcp import ClientSession -from mcp.client.streamable_http import streamable_http_client -from mcp.types import CallToolResult, TextContent +from mcp.types import CallToolResult -from agentenv_protocol import EnvironmentTool, ForwardedTool, client +from agentenv_protocol import ForwardedTool, client, mcp_port from agentenv_protocol.website_browser.log import TOOL_CALL_HEADER, tool_call_header from agentenv_protocol.computer import ( ComputerDriverEnvironment, @@ -67,6 +57,8 @@ from agentenv_protocol.tool_context import ROLE_HEADER, ToolContext from agentenv_protocol.website_browser import BROWSER +import playwright_mcp + BUTTONS = {"left": 1, "right": 2, "middle": 4, "back": 8, "forward": 16} MODIFIERS = {"Alt": 1, "Control": 2, "Meta": 4, "Shift": 8} #: Keys that aren't characters: each one's DOM code, its Windows virtual key code, and the text a press of it types. @@ -94,33 +86,6 @@ BLANK = b"\xff\xff\xff" #: How long one call to the browser's server may take. CALL_TIMEOUT_S = 30.0 -#: Playwright MCP, as its install lays it out beside the driver, and its tools as 0.0.84 lists them with ``--caps -#: vision``, which an integration test checks against the one installed. -HERE = Path(__file__).resolve().parent -NODE = HERE / "node" / "bin" / "node" -PLAYWRIGHT_MCP = HERE / "node_modules" / "@playwright" / "mcp" / "cli.js" -PLAYWRIGHT_MCP_TOOLS = HERE / "playwright-mcp-tools.json" -#: Playwright MCP's tools that run the agent's own code, in the page and in Playwright MCP's own process: hidden until -#: a task's tool access names them. -HIDDEN_TOOLS = frozenset({"browser_evaluate", "browser_run_code_unsafe"}) -#: Whom the browser issues Playwright MCP's CDP address to, as its record names the connections. -PLAYWRIGHT_MCP_USER = "playwright-mcp" -#: What Playwright MCP's process keeps of the driver's environment, which holds the deploy's tokens and settings. -KEPT_ENVIRONMENT = ("PATH", "HOME", "USER", "LOGNAME", "LANG", "LC_ALL", "TZ", "TMPDIR") -#: Where Playwright MCP writes what it names itself, and its log. -PLAYWRIGHT_MCP_OUTPUT = Path(tempfile.gettempdir()) / "agentenv-playwright-mcp" -#: How long Playwright MCP may take to listen or to open a session, and one of its tools to answer. -PLAYWRIGHT_MCP_START_S = 30.0 -PLAYWRIGHT_MCP_CALL_TIMEOUT_S = 300.0 -#: How often a start checks whether Playwright MCP listens yet, and how long a stop waits for it to end on SIGTERM -#: before it's killed. -PLAYWRIGHT_MCP_POLL_S = 0.1 -PLAYWRIGHT_MCP_STOP_S = 10.0 -#: How long a caller's session on Playwright MCP is kept with no call: a caller's own MCP session ends with no word to -#: the driver, and each holds a connection to the browser. -PLAYWRIGHT_MCP_SESSION_IDLE_S = 3600.0 - -logger = logging.getLogger(__name__) @computer_driver(name="page", supports=[{"os": {"type": "linux", "display": ["none"]}, "arch": ["amd64", "arm64"]}]) @@ -144,15 +109,12 @@ def __init__(self) -> None: self._pointer = (0.0, 0.0) self._buttons = 0 self._held: list[str] = [] - self._playwright = _PlaywrightMcp(self._playwright_endpoint) + self._playwright = playwright_mcp.PlaywrightMcp(self._playwright_endpoint) def forwarded_tools(self) -> list[ForwardedTool]: - """Playwright MCP's tools, as its manifest lists them, each called in the caller's own session there; those - that run the agent's own code hidden until a task's tool access names them.""" - return [ForwardedTool(EnvironmentTool(name=t["name"], description=t["description"], inputSchema=t["inputSchema"]), - functools.partial(self._playwright_call, t["name"]), t.get("annotations"), - t.get("outputSchema"), hidden_by_default=t["name"] in HIDDEN_TOOLS) - for t in json.loads(PLAYWRIGHT_MCP_TOOLS.read_text(encoding="utf-8"))["tools"]] + """Playwright MCP's, called through a method rather than ``self._playwright``, since reading the class at put + makes it without ``__init__``.""" + return playwright_mcp.tools(self._playwright_call) async def _playwright_call(self, tool: str, /, **arguments: Any) -> CallToolResult: return await self._playwright.call(tool, **arguments) @@ -161,8 +123,8 @@ async def _playwright_endpoint(self) -> str: """The CDP address the browser issues Playwright MCP, at this driver's own loopback address, where its relay is reached as any agent reaches it: the same one until the browser's data is reset, which ends it.""" folder, _ = await self._found() - origin = f"http://127.0.0.1:{os.environ.get('MCP_PORT', '18765')}/svc/{folder.name}" - return (await self._call("cdp", {"user": PLAYWRIGHT_MCP_USER, "origin": origin}, credited=False))["ws_url"] + origin = f"http://127.0.0.1:{mcp_port()}/svc/{folder.name}" + return (await self._call("cdp", {"user": playwright_mcp.CDP_USER, "origin": origin}, credited=False))["ws_url"] @display async def viewport(self) -> Display: @@ -455,190 +417,5 @@ def _steps(path: list) -> list[tuple[float, float]]: return steps -class _PlaywrightMcp: - """Playwright MCP, attached to the browser over the CDP address ``endpoint`` gives, and its sessions there, one per - caller of the driver's tools, so each agent keeps its own snapshots and tab, each closed once idle for - PLAYWRIGHT_MCP_SESSION_IDLE_S. It starts on the first call, again once it has exited or failed to start, and again on - a fresh address once the browser's data reset has ended the last; it ends with the driver, so a driver started again - finds none of its predecessor's.""" - - def __init__(self, endpoint: Callable[[], Awaitable[str]]) -> None: - self._endpoint = endpoint - self._lock = asyncio.Lock() - self._process: Optional[asyncio.subprocess.Process] = None - self._running_on: Optional[str] = None - self._url = "" - self._sessions: dict[Optional[str], _Session] = {} - - async def call(self, tool: str, /, **arguments: Any) -> CallToolResult: - """``tool`` called on Playwright MCP as it answers, or refused saying why it couldn't be. A call Playwright MCP - or its session ends under fails then, and isn't sent again, since it may have acted.""" - try: - held, process = await self._session(ToolContext.current().caller.session) - except DeviceUnavailable as e: - return _refused(e.message) - except Exception as e: - return _refused(f"Playwright MCP didn't answer {tool}: {type(e).__name__}: {e}") - called = asyncio.ensure_future(held.session.call_tool( - tool, arguments, read_timeout_seconds=timedelta(seconds=PLAYWRIGHT_MCP_CALL_TIMEOUT_S))) - exited = asyncio.ensure_future(process.wait()) - try: - await asyncio.wait({called, exited, held.owner}, return_when=asyncio.FIRST_COMPLETED) - except asyncio.CancelledError: - called.cancel() - with contextlib.suppress(asyncio.CancelledError, Exception): - await called - raise - finally: - exited.cancel() - if not called.done(): - called.cancel() - held.close.set() - return _refused(f"Playwright MCP ended during {tool}, which may have acted before it did") - try: - return called.result() - except Exception as e: - held.close.set() - return _refused(f"Playwright MCP didn't answer {tool}: {type(e).__name__}: {e}") - finally: - held.used_at = asyncio.get_running_loop().time() - - async def _session(self, caller: Optional[str]) -> tuple[_Session, asyncio.subprocess.Process]: - async with self._lock: - url = await self._running() - now = asyncio.get_running_loop().time() - for idle in [key for key, held in self._sessions.items() - if key != caller and now - held.used_at > PLAYWRIGHT_MCP_SESSION_IDLE_S]: - self._sessions.pop(idle).close.set() - held = self._sessions.get(caller) - if held is None or held.url != url or held.owner.done() or held.close.is_set(): - if held is not None: - held.close.set() - try: - held = self._sessions[caller] = await _Session.open(url) - except DeviceUnavailable: - await self._stop() - raise - held.used_at = now - return held, self._process - - async def _running(self) -> str: - """Playwright MCP's MCP URL, started on the browser's current address for it unless it runs there already.""" - endpoint = await self._endpoint() - if self._process is not None and self._process.returncode is None and endpoint == self._running_on: - return self._url - await self._stop() - port = _free_port() - PLAYWRIGHT_MCP_OUTPUT.mkdir(parents=True, exist_ok=True) - uploads = Path.home() / "Uploads" - uploads.mkdir(parents=True, exist_ok=True) - with (PLAYWRIGHT_MCP_OUTPUT / "playwright-mcp.log").open("ab") as log: - self._process = await asyncio.create_subprocess_exec( - *_command(endpoint, port), cwd=uploads, env=_environment(), stdin=asyncio.subprocess.DEVNULL, - stdout=asyncio.subprocess.DEVNULL, stderr=log) - try: - await _listening(port, self._process) - except BaseException: - await self._stop() - raise - self._running_on, self._url = endpoint, f"http://127.0.0.1:{port}/mcp" - return self._url - - async def _stop(self) -> None: - for held in self._sessions.values(): - held.close.set() - self._sessions.clear() - if self._process is not None and self._process.returncode is None: - self._process.terminate() - with contextlib.suppress(asyncio.TimeoutError): - await asyncio.wait_for(self._process.wait(), PLAYWRIGHT_MCP_STOP_S) - if self._process.returncode is None: - self._process.kill() - await self._process.wait() - self._process, self._running_on = None, None - - -class _Session: - """One MCP session on Playwright MCP, held open by a task of its own until ``close`` is set, since mcp's client - contexts are anyio task groups, which only the task that entered one can leave.""" - - def __init__(self, url: str) -> None: - self.url = url - self.close = asyncio.Event() - self.session: Optional[ClientSession] = None - self.owner: Optional[asyncio.Task] = None - self.used_at = 0.0 - - @classmethod - async def open(cls, url: str) -> _Session: - """A session on Playwright MCP at ``url``; DeviceUnavailable when it doesn't open one in time, wedged.""" - held = cls(url) - ready: asyncio.Future = asyncio.get_running_loop().create_future() - held.owner = asyncio.create_task(held._own(ready)) - try: - held.session = await asyncio.wait_for(ready, PLAYWRIGHT_MCP_START_S) - except asyncio.TimeoutError: - held.close.set() - held.owner.cancel() - raise DeviceUnavailable(f"Playwright MCP didn't open a session within {PLAYWRIGHT_MCP_START_S:.0f} s") - return held - - async def _own(self, ready: asyncio.Future) -> None: - try: - timeout = httpx.Timeout(PLAYWRIGHT_MCP_CALL_TIMEOUT_S, connect=PLAYWRIGHT_MCP_START_S) - async with httpx.AsyncClient(timeout=timeout, trust_env=False) as http, \ - streamable_http_client(self.url, http_client=http) as (read, write, _), \ - ClientSession(read, write) as session: - await session.initialize() - if ready.done(): - return - ready.set_result(session) - await self.close.wait() - except BaseException as e: - if not ready.done(): - ready.set_exception(e if isinstance(e, Exception) else RuntimeError(repr(e))) - elif not isinstance(e, asyncio.CancelledError): - logger.warning(f"a session on Playwright MCP ended: {type(e).__name__}: {e}") - if not isinstance(e, Exception): - raise - - -def _command(endpoint: str, port: int) -> list[str]: - """Playwright MCP on loopback ``port``, attached to the browser at ``endpoint``, never a browser of its own, with - its file access confined to its working folder, the session user's ~/Uploads; ``setpriv`` ends it with the driver.""" - return ["setpriv", "--pdeathsig", "TERM", "--", str(NODE), str(PLAYWRIGHT_MCP), "--cdp-endpoint", endpoint, - "--host", "127.0.0.1", "--port", str(port), "--caps", "vision", "--ignore-roots", - "--output-dir", str(PLAYWRIGHT_MCP_OUTPUT)] - - -def _environment() -> dict[str, str]: - return {name: os.environ[name] for name in KEPT_ENVIRONMENT if name in os.environ} - - -def _free_port() -> int: - with socket.socket() as probe: - probe.bind(("127.0.0.1", 0)) - return probe.getsockname()[1] - - -async def _listening(port: int, process: asyncio.subprocess.Process) -> None: - """Once Playwright MCP listens on ``port``; DeviceUnavailable when it exits first or takes too long.""" - deadline = asyncio.get_running_loop().time() + PLAYWRIGHT_MCP_START_S - while asyncio.get_running_loop().time() < deadline: - if process.returncode is not None: - raise DeviceUnavailable(f"Playwright MCP exited at start with {process.returncode}; its log is in " - f"{PLAYWRIGHT_MCP_OUTPUT}") - with contextlib.suppress(OSError): - _, writer = await asyncio.open_connection("127.0.0.1", port) - writer.close() - return - await asyncio.sleep(PLAYWRIGHT_MCP_POLL_S) - raise DeviceUnavailable(f"Playwright MCP didn't listen within {PLAYWRIGHT_MCP_START_S:.0f} s") - - -def _refused(message: str) -> CallToolResult: - return CallToolResult(isError=True, content=[TextContent(type="text", text=message)]) - - if __name__ == "__main__": PageDriver().serve() diff --git a/src/agent_env/drivers/envs/page/env.toml b/src/agent_env/drivers/envs/page/env.toml index 9fc8c6514..b2f96f09b 100644 --- a/src/agent_env/drivers/envs/page/env.toml +++ b/src/agent_env/drivers/envs/page/env.toml @@ -4,8 +4,9 @@ port = 18765 [install.linux] python = "requirements.txt" -# Node and Playwright MCP 0.0.84, whose browser_* tools the driver serves beside its own: Node 24.22.2's Linux tarball -# (sha256 from nodejs.org's SHASUMS256.txt) and Playwright MCP's three npm packages (checked against npm's integrity). +# Node and Playwright MCP, whose browser_* tools the driver serves beside its own, as scripts/upgrade_playwright_mcp.py +# pins them: Node's Linux tarballs (sha256 from nodejs.org's SHASUMS256.txt) and every npm package Playwright MCP needs +# (checked against npm's integrity), which install-playwright-mcp.sh lays out. downloads = [ { url = "https://nodejs.org/dist/v24.22.2/node-v24.22.2-linux-x64.tar.gz", sha256 = "361718c444e49af2ca5a110c83ba6a8901143f23ab285ebbef0341f391f482a9", to = "node.tar.gz", arch = "amd64" }, { url = "https://nodejs.org/dist/v24.22.2/node-v24.22.2-linux-arm64.tar.gz", sha256 = "95c264150aceb7f3437a29fd6984924f5f8d1511ea3cbd6c508b83f6c555dae2", to = "node.tar.gz", arch = "arm64" }, diff --git a/src/agent_env/drivers/envs/page/install-playwright-mcp.sh b/src/agent_env/drivers/envs/page/install-playwright-mcp.sh index aba7eb83f..213b25ca6 100755 --- a/src/agent_env/drivers/envs/page/install-playwright-mcp.sh +++ b/src/agent_env/drivers/envs/page/install-playwright-mcp.sh @@ -1,7 +1,7 @@ #!/bin/sh # Node and Playwright MCP, from the tarballs the install downloaded and checked, with no npm install, since an install -# has no network: Node's binary and licence in node/, and Playwright MCP's three npm packages in node_modules/, each -# where npm would put it. +# has no network: Node's binary and licence in node/, and each npm package in node_modules/ where npm would put it, by +# the name its package.json gives. A package whose dependencies aren't all there, at the versions it names, fails it. set -eu work=$(mktemp -d) tar -xzf node.tar.gz -C "$work" @@ -9,11 +9,23 @@ mkdir -p node/bin mv "$work"/node-*/bin/node node/bin/node mv "$work"/node-*/LICENSE node/LICENSE rm -rf "$work" node.tar.gz -unpack() { - mkdir -p "node_modules/$2" - tar -xzf "$1" -C "node_modules/$2" --strip-components=1 - rm -f "$1" -} -unpack playwright-mcp.tgz @playwright/mcp -unpack playwright.tgz playwright -unpack playwright-core.tgz playwright-core +for package in *.tgz; do + work=$(mktemp -d) + tar -xzf "$package" -C "$work" --strip-components=1 + name=$(node/bin/node -p 'require(process.argv[1]).name' "$work/package.json") + mkdir -p "node_modules/$(dirname "$name")" + mv "$work" "node_modules/$name" + rm -f "$package" +done +node/bin/node -e ' +const fs = require("fs"), path = require("path"); +const names = fs.readdirSync("node_modules").flatMap(n => n.startsWith("@") + ? fs.readdirSync(path.join("node_modules", n)).map(m => `${n}/${m}`) : [n]); +const unmet = names.flatMap(name => Object.entries( + require(path.resolve("node_modules", name, "package.json")).dependencies || {}).flatMap(([dependency, version]) => { + const file = path.resolve("node_modules", dependency, "package.json"); + const found = fs.existsSync(file) ? require(file).version : "none"; + return found === version ? [] : [`${name} needs ${dependency} ${version}, found ${found}`]; +})); +if (unmet.length) { console.error(unmet.join("\n")); process.exit(1); } +' diff --git a/src/agent_env/drivers/envs/page/playwright_mcp.py b/src/agent_env/drivers/envs/page/playwright_mcp.py new file mode 100644 index 000000000..a3d61c3cf --- /dev/null +++ b/src/agent_env/drivers/envs/page/playwright_mcp.py @@ -0,0 +1,245 @@ +"""Playwright MCP for the page driver: its install, laid out beside the driver by ``install-playwright-mcp.sh`` from +the downloads ``scripts/upgrade_playwright_mcp.py`` pins; its tools, as its manifest lists them; and its process and +sessions, one per caller of the driver's tools, which the driver forwards those tools to.""" +from __future__ import annotations + +import asyncio +import contextlib +import functools +import json +import logging +import os +import socket +import tempfile +from datetime import timedelta +from pathlib import Path +from typing import Any, Awaitable, Callable, Optional + +import httpx +from mcp import ClientSession +from mcp.client.streamable_http import streamable_http_client +from mcp.types import CallToolResult, TextContent + +from agentenv_protocol import ForwardedTool +from agentenv_protocol.computer import DeviceUnavailable +from agentenv_protocol.tool_context import ToolContext + +#: Playwright MCP, as its install lays it out beside the driver, and its tools as its version lists them with ``--caps +#: vision``, which ``scripts/upgrade_playwright_mcp.py`` writes and an integration test checks against the one installed. +HERE = Path(__file__).resolve().parent +NODE = HERE / "node" / "bin" / "node" +CLI = HERE / "node_modules" / "@playwright" / "mcp" / "cli.js" +TOOLS = HERE / "playwright-mcp-tools.json" +#: Playwright MCP's tools that run the agent's own code, in the page and in Playwright MCP's own process: hidden until +#: a task's tool access names them. +HIDDEN_TOOLS = frozenset({"browser_evaluate", "browser_run_code_unsafe"}) +#: Whom the browser issues Playwright MCP's CDP address to, as its record names the connections. +CDP_USER = "playwright-mcp" +#: What Playwright MCP's process keeps of the driver's environment, which holds the deploy's tokens and settings. +KEPT_ENVIRONMENT = ("PATH", "HOME", "USER", "LOGNAME", "LANG", "LC_ALL", "TZ", "TMPDIR") +#: Where Playwright MCP writes what it names itself, and its log. +OUTPUT = Path(tempfile.gettempdir()) / "agentenv-playwright-mcp" +#: How long Playwright MCP may take to listen or to open a session, and one of its tools to answer. +START_S = 30.0 +CALL_TIMEOUT_S = 300.0 +#: How often a start checks whether Playwright MCP listens yet, and how long a stop waits for it to end on SIGTERM +#: before it's killed. +POLL_S = 0.1 +STOP_S = 10.0 +#: How long a caller's session on Playwright MCP is kept with no call: a caller's own MCP session ends with no word to +#: the driver, and each holds a connection to the browser. +SESSION_IDLE_S = 3600.0 + +logger = logging.getLogger(__name__) + + +def tools(call: Callable[..., Awaitable[CallToolResult]]) -> list[ForwardedTool]: + """Playwright MCP's tools, as its manifest lists them, each called through ``call`` with its name first; those that + run the agent's own code hidden until a task's tool access names them.""" + return [ForwardedTool.listed(t, functools.partial(call, t["name"]), hidden_by_default=t["name"] in HIDDEN_TOOLS) + for t in json.loads(TOOLS.read_text(encoding="utf-8"))["tools"]] + + +class PlaywrightMcp: + """Playwright MCP, attached to the browser over the CDP address ``endpoint`` gives, and its sessions there, one per + caller of the driver's tools, so each agent keeps its own snapshots and tab, each closed once idle for + SESSION_IDLE_S. It starts on the first call, again once it has exited or failed to start, and again on a fresh + address once the browser's data reset has ended the last; it ends with the driver, so a driver started again finds + none of its predecessor's.""" + + def __init__(self, endpoint: Callable[[], Awaitable[str]]) -> None: + self._endpoint = endpoint + self._lock = asyncio.Lock() + self._process: Optional[asyncio.subprocess.Process] = None + self._running_on: Optional[str] = None + self._url = "" + self._sessions: dict[Optional[str], _Session] = {} + + async def call(self, tool: str, /, **arguments: Any) -> CallToolResult: + """``tool`` called on Playwright MCP as it answers, or refused saying why it couldn't be. A call Playwright MCP + or its session ends under fails then, and isn't sent again, since it may have acted.""" + try: + held, process = await self._session(ToolContext.current().caller.session) + except DeviceUnavailable as e: + return _refused(e.message) + except Exception as e: + return _refused(f"Playwright MCP didn't answer {tool}: {type(e).__name__}: {e}") + called = asyncio.ensure_future(held.session.call_tool( + tool, arguments, read_timeout_seconds=timedelta(seconds=CALL_TIMEOUT_S))) + exited = asyncio.ensure_future(process.wait()) + try: + await asyncio.wait({called, exited, held.owner}, return_when=asyncio.FIRST_COMPLETED) + except asyncio.CancelledError: + called.cancel() + with contextlib.suppress(asyncio.CancelledError, Exception): + await called + raise + finally: + exited.cancel() + if not called.done(): + called.cancel() + held.close.set() + return _refused(f"Playwright MCP ended during {tool}, which may have acted before it did") + try: + return called.result() + except Exception as e: + held.close.set() + return _refused(f"Playwright MCP didn't answer {tool}: {type(e).__name__}: {e}") + finally: + held.used_at = asyncio.get_running_loop().time() + + async def _session(self, caller: Optional[str]) -> tuple[_Session, asyncio.subprocess.Process]: + async with self._lock: + url = await self._running() + now = asyncio.get_running_loop().time() + for idle in [key for key, held in self._sessions.items() + if key != caller and now - held.used_at > SESSION_IDLE_S]: + self._sessions.pop(idle).close.set() + held = self._sessions.get(caller) + if held is None or held.url != url or held.owner.done() or held.close.is_set(): + if held is not None: + held.close.set() + try: + held = self._sessions[caller] = await _Session.open(url) + except DeviceUnavailable: + await self._stop() + raise + held.used_at = now + return held, self._process + + async def _running(self) -> str: + """Playwright MCP's MCP URL, started on the browser's current address for it unless it runs there already.""" + endpoint = await self._endpoint() + if self._process is not None and self._process.returncode is None and endpoint == self._running_on: + return self._url + await self._stop() + port = _free_port() + OUTPUT.mkdir(parents=True, exist_ok=True) + uploads = Path.home() / "Uploads" + uploads.mkdir(parents=True, exist_ok=True) + with (OUTPUT / "playwright-mcp.log").open("ab") as log: + self._process = await asyncio.create_subprocess_exec( + *_command(endpoint, port), cwd=uploads, env=_environment(), stdin=asyncio.subprocess.DEVNULL, + stdout=asyncio.subprocess.DEVNULL, stderr=log) + try: + await _listening(port, self._process) + except BaseException: + await self._stop() + raise + self._running_on, self._url = endpoint, f"http://127.0.0.1:{port}/mcp" + return self._url + + async def _stop(self) -> None: + for held in self._sessions.values(): + held.close.set() + self._sessions.clear() + if self._process is not None and self._process.returncode is None: + self._process.terminate() + with contextlib.suppress(asyncio.TimeoutError): + await asyncio.wait_for(self._process.wait(), STOP_S) + if self._process.returncode is None: + self._process.kill() + await self._process.wait() + self._process, self._running_on = None, None + + +class _Session: + """One MCP session on Playwright MCP, held open by a task of its own until ``close`` is set, since mcp's client + contexts are anyio task groups, which only the task that entered one can leave.""" + + def __init__(self, url: str) -> None: + self.url = url + self.close = asyncio.Event() + self.session: Optional[ClientSession] = None + self.owner: Optional[asyncio.Task] = None + self.used_at = 0.0 + + @classmethod + async def open(cls, url: str) -> _Session: + """A session on Playwright MCP at ``url``; DeviceUnavailable when it doesn't open one in time, wedged.""" + held = cls(url) + ready: asyncio.Future = asyncio.get_running_loop().create_future() + held.owner = asyncio.create_task(held._own(ready)) + try: + held.session = await asyncio.wait_for(ready, START_S) + except asyncio.TimeoutError: + held.close.set() + held.owner.cancel() + raise DeviceUnavailable(f"Playwright MCP didn't open a session within {START_S:.0f} s") + return held + + async def _own(self, ready: asyncio.Future) -> None: + try: + timeout = httpx.Timeout(CALL_TIMEOUT_S, connect=START_S) + async with httpx.AsyncClient(timeout=timeout, trust_env=False) as http, \ + streamable_http_client(self.url, http_client=http) as (read, write, _), \ + ClientSession(read, write) as session: + await session.initialize() + if ready.done(): + return + ready.set_result(session) + await self.close.wait() + except BaseException as e: + if not ready.done(): + ready.set_exception(e if isinstance(e, Exception) else RuntimeError(repr(e))) + elif not isinstance(e, asyncio.CancelledError): + logger.warning(f"a session on Playwright MCP ended: {type(e).__name__}: {e}") + if not isinstance(e, Exception): + raise + + +def _command(endpoint: str, port: int) -> list[str]: + """Playwright MCP on loopback ``port``, attached to the browser at ``endpoint``, never a browser of its own, with + its file access confined to its working folder, the session user's ~/Uploads; ``setpriv`` ends it with the driver.""" + return ["setpriv", "--pdeathsig", "TERM", "--", str(NODE), str(CLI), "--cdp-endpoint", endpoint, + "--host", "127.0.0.1", "--port", str(port), "--caps", "vision", "--ignore-roots", + "--output-dir", str(OUTPUT)] + + +def _environment() -> dict[str, str]: + return {name: os.environ[name] for name in KEPT_ENVIRONMENT if name in os.environ} + + +def _free_port() -> int: + with socket.socket() as probe: + probe.bind(("127.0.0.1", 0)) + return probe.getsockname()[1] + + +async def _listening(port: int, process: asyncio.subprocess.Process) -> None: + """Once Playwright MCP listens on ``port``; DeviceUnavailable when it exits first or takes too long.""" + deadline = asyncio.get_running_loop().time() + START_S + while asyncio.get_running_loop().time() < deadline: + if process.returncode is not None: + raise DeviceUnavailable(f"Playwright MCP exited at start with {process.returncode}; its log is in " + f"{OUTPUT}") + with contextlib.suppress(OSError): + _, writer = await asyncio.open_connection("127.0.0.1", port) + writer.close() + return + await asyncio.sleep(POLL_S) + raise DeviceUnavailable(f"Playwright MCP didn't listen within {START_S:.0f} s") + + +def _refused(message: str) -> CallToolResult: + return CallToolResult(isError=True, content=[TextContent(type="text", text=message)]) diff --git a/tst/unit/drivers/page_driver_test.py b/tst/unit/drivers/page_driver_test.py index 7687b4140..cf1569aec 100644 --- a/tst/unit/drivers/page_driver_test.py +++ b/tst/unit/drivers/page_driver_test.py @@ -1,8 +1,9 @@ """The page driver against a fake browser server on the app's socket: it passes the definition checks with a page screen, turns tool calls into the CDP input they mean, shows a tab that paints nothing as white, and once the browser has ended on its own, its ready check fails for good and its calls say why. Beside its own tools it serves Playwright -MCP's from their manifest, the two that run the agent's code hidden, on a Playwright MCP it runs on the CDP address the -browser issues it under the driver's loopback origin: here a stand-in, which starts on the first call, keeps a session +MCP's from their manifest, which is of the version its install downloads and lists only tools looked at, the two that run +the agent's code hidden, on a Playwright MCP it runs on the CDP address the browser issues it under the driver's loopback +origin, a check its tool calls don't credit to the agent: here a stand-in, which starts on the first call, keeps a session per caller, starts again on a new address or once killed, and never gets the driver's environment; a call it ends under fails then, a caller's session left idle is closed, and a start that never listens is stopped and tried afresh.""" from __future__ import annotations @@ -12,9 +13,12 @@ import importlib.util import json import os +import re import signal import sys +import tomllib from pathlib import Path +from unittest import mock import httpx import pytest @@ -30,6 +34,18 @@ from agentenv_protocol.website_browser.log import TOOL_CALL_HEADER DRIVER = Path(agent_env.__file__).parent / "drivers" / "envs" / "page" / "driver.py" +PLAYWRIGHT_MCP = DRIVER.parent / "playwright_mcp.py" +#: Playwright MCP's tools, each looked at for whether it runs the agent's own code and so is hidden by default: an +#: upgrade that adds a tool fails here until it's been looked at too. +REVIEWED_TOOLS = { + "browser_click", "browser_close", "browser_console_messages", "browser_drag", "browser_drop", + "browser_emulate_media", "browser_evaluate", "browser_file_upload", "browser_fill_form", "browser_find", + "browser_handle_dialog", "browser_hover", "browser_mouse_click_xy", "browser_mouse_down", "browser_mouse_drag_xy", + "browser_mouse_move_xy", "browser_mouse_up", "browser_mouse_wheel", "browser_navigate", "browser_navigate_back", + "browser_network_request", "browser_network_requests", "browser_press_key", "browser_resize", + "browser_run_code_unsafe", "browser_select_option", "browser_snapshot", "browser_tabs", "browser_take_screenshot", + "browser_type", "browser_wait_for", +} NO_DISPLAY = Platform(os="linux", distribution="ubuntu", version="24.04", arch="amd64", display="none") PAGE_OPS = ("page/viewport", "page/capture", "page/input", "page/state", "navigate", "history", "tabs/list", "tabs/open", "tabs/activate", "tabs/close", "dialog/answer", "files/choose", "cdp") @@ -73,13 +89,23 @@ async def browser_wait_for(time: float) -> str: PNG = b"\x89PNG\r\n\x1a\n" + b"\x00\x00\x00\rIHDR" + (2).to_bytes(4, "big") + (1).to_bytes(4, "big") -def _load(): - spec = importlib.util.spec_from_file_location("page_driver", DRIVER) +def _module(name: str, path: Path): + spec = importlib.util.spec_from_file_location(name, path) module = importlib.util.module_from_spec(spec) spec.loader.exec_module(module) return module +def _playwright_mcp(): + return _module("playwright_mcp", PLAYWRIGHT_MCP) + + +def _load(): + """driver.py, importing its own playwright_mcp.py as it does where it runs.""" + with mock.patch.dict(sys.modules, {"playwright_mcp": _playwright_mcp()}): + return _module("page_driver", DRIVER) + + @pytest.fixture def page(monkeypatch, tmp_path): """The driver, on a computer with no display, its one browser in ``tmp_path``, whose server answers ``served``, @@ -269,12 +295,21 @@ def test_it_serves_playwright_mcps_tools_from_their_manifest_and_hides_the_two_t manifest = json.loads((DRIVER.parent / "playwright-mcp-tools.json").read_text()) forwarded = page.forwarded_tools() - assert manifest["version"] == "0.0.84" and manifest["caps"] == ["vision"] + assert manifest["caps"] == ["vision"] and {t["name"] for t in manifest["tools"]} == REVIEWED_TOOLS assert [f.tool.name for f in forwarded] == [t["name"] for t in manifest["tools"]] assert [f.tool.inputSchema for f in forwarded] == [t["inputSchema"] for t in manifest["tools"]] assert {f.tool.name for f in forwarded if f.hidden_by_default} == {"browser_evaluate", "browser_run_code_unsafe"} +def test_its_manifest_is_of_the_playwright_mcp_its_install_downloads(): + manifest = json.loads((DRIVER.parent / "playwright-mcp-tools.json").read_text()) + downloads = tomllib.loads((DRIVER.parent / "env.toml").read_text())["install"]["linux"]["downloads"] + + versions = [m.group(1) for d in downloads if (m := re.search(r"/@playwright/mcp/-/mcp-(.+)\.tgz$", d["url"]))] + + assert versions == [manifest["version"]] + + @pytest.mark.asyncio async def test_it_asks_the_browser_for_playwright_mcps_address_under_its_own_loopback_origin_as_its_own_call( page, monkeypatch): @@ -288,12 +323,12 @@ async def test_it_asks_the_browser_for_playwright_mcps_address_under_its_own_loo assert endpoint == "ws://127.0.0.1:18765/svc/chromium/cdp/t/devtools/browser" assert page.sent[0] == ("cdp", {"user": "playwright-mcp", "origin": f"http://127.0.0.1:18765/svc/{page._browser[0].name}"}) - assert TOOL_CALL_HEADER not in page.headers[0], "the browser's log credits the agent's call with no address check" + assert TOOL_CALL_HEADER not in page.headers[0] assert TOOL_CALL_HEADER in page.headers[1] def test_playwright_mcp_attaches_to_the_browser_never_launches_one_and_ends_with_the_driver(): - command = _load()._command("ws://127.0.0.1:18765/svc/chromium/cdp/t/devtools/browser", 40123) + command = _playwright_mcp()._command("ws://127.0.0.1:18765/svc/chromium/cdp/t/devtools/browser", 40123) assert command[:4] == ["setpriv", "--pdeathsig", "TERM", "--"] assert command[command.index("--cdp-endpoint") + 1] == "ws://127.0.0.1:18765/svc/chromium/cdp/t/devtools/browser" @@ -334,21 +369,21 @@ async def snapshot(caller: str) -> dict: @pytest.fixture def stand_in(monkeypatch, tmp_path): - """A ``_PlaywrightMcp`` on the stand-in, its address the last of ``endpoints``, and a way to call it as a caller.""" + """A ``PlaywrightMcp`` on the stand-in, its address the last of ``endpoints``, and a way to call it as a caller.""" enable_socket() - module = _load() + module = _playwright_mcp() fake = tmp_path / "fake_playwright_mcp.py" fake.write_text(FAKE_PLAYWRIGHT_MCP) monkeypatch.setattr(module, "_command", lambda endpoint, port: [ sys.executable, str(fake), "--cdp-endpoint", endpoint, "--port", str(port), "--marker", str(tmp_path / "m")]) - monkeypatch.setattr(module, "PLAYWRIGHT_MCP_OUTPUT", tmp_path / "out") + monkeypatch.setattr(module, "OUTPUT", tmp_path / "out") monkeypatch.setenv("HOME", str(tmp_path / "home")) endpoints = ["ws://127.0.0.1:18765/svc/chromium/cdp/one/devtools/browser"] async def endpoint() -> str: return endpoints[-1] - playwright = module._PlaywrightMcp(endpoint) + playwright = module.PlaywrightMcp(endpoint) async def called(caller: str, tool: str = "browser_snapshot", arguments: dict | None = None): with bound(ToolContext.as_caller(session=caller)): @@ -396,7 +431,7 @@ async def test_a_call_its_caller_cancels_ends_its_call_on_playwright_mcp_too(sta @pytest.mark.asyncio async def test_a_callers_session_left_idle_is_closed_when_another_caller_comes(stand_in, monkeypatch): module, playwright, _, called = stand_in - monkeypatch.setattr(module, "PLAYWRIGHT_MCP_SESSION_IDLE_S", 0.5) + monkeypatch.setattr(module, "SESSION_IDLE_S", 0.5) try: await called("ada") ada = playwright._sessions["ada"] @@ -413,7 +448,7 @@ async def test_a_callers_session_left_idle_is_closed_when_another_caller_comes(s @pytest.mark.asyncio async def test_a_start_that_never_listens_is_stopped_and_the_next_call_starts_afresh(stand_in, monkeypatch, tmp_path): module, playwright, endpoints, called = stand_in - monkeypatch.setattr(module, "PLAYWRIGHT_MCP_START_S", 2.0) + monkeypatch.setattr(module, "START_S", 2.0) endpoints.append("ws://127.0.0.1:18765/svc/chromium/cdp/silent-once") try: refused = await called("ada") @@ -438,7 +473,7 @@ def _gone(pid: int) -> bool: @pytest.mark.asyncio async def test_a_call_slower_than_its_bound_is_refused_and_its_session_opened_afresh(stand_in, monkeypatch): module, playwright, _, called = stand_in - monkeypatch.setattr(module, "PLAYWRIGHT_MCP_CALL_TIMEOUT_S", 1.0) + monkeypatch.setattr(module, "CALL_TIMEOUT_S", 1.0) try: await called("ada") first = playwright._sessions["ada"] @@ -455,7 +490,7 @@ async def test_a_call_slower_than_its_bound_is_refused_and_its_session_opened_af async def test_a_playwright_mcp_that_never_opens_a_session_is_stopped_and_the_next_call_starts_afresh( stand_in, monkeypatch, tmp_path): module, playwright, endpoints, called = stand_in - monkeypatch.setattr(module, "PLAYWRIGHT_MCP_START_S", 2.0) + monkeypatch.setattr(module, "START_S", 2.0) endpoints.append("ws://127.0.0.1:18765/svc/chromium/cdp/wedged-once") try: refused = await asyncio.wait_for(called("ada"), 20) From 1751e48d4af86c605c8261f6c7464a1993ac7527 Mon Sep 17 00:00:00 2001 From: Edgar Arakelyan Date: Wed, 11 Nov 2026 14:31:04 -0800 Subject: [PATCH 7/7] fix(drivers): let the session user read Playwright MCP's packages The install moved each package's mktemp folder into node_modules, and mktemp makes its folder 0700, so the session user's Playwright MCP couldn't load its own cli.js and exited at start. Co-Authored-By: Claude Opus 5.5 --- src/agent_env/drivers/envs/page/install-playwright-mcp.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/agent_env/drivers/envs/page/install-playwright-mcp.sh b/src/agent_env/drivers/envs/page/install-playwright-mcp.sh index 213b25ca6..97e299362 100755 --- a/src/agent_env/drivers/envs/page/install-playwright-mcp.sh +++ b/src/agent_env/drivers/envs/page/install-playwright-mcp.sh @@ -11,6 +11,8 @@ mv "$work"/node-*/LICENSE node/LICENSE rm -rf "$work" node.tar.gz for package in *.tgz; do work=$(mktemp -d) + # mktemp's folder is its owner's alone, and the session user runs what's in it. + chmod 755 "$work" tar -xzf "$package" -C "$work" --strip-components=1 name=$(node/bin/node -p 'require(process.argv[1]).name' "$work/package.json") mkdir -p "node_modules/$(dirname "$name")"