Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,9 @@
- Added an optional `intent` to `browser_act` across Core RPC, the extension, MCP, OMP, Pi, and the TypeScript SDK. When an agent states the purpose of a batch in one line of up to 120 characters, such as "Open the bookmarks folder", the agent pointer shows it as its caption instead of the generated action label, and captions now wrap to two lines. Requests with an empty, overlong, or multi-line intent fail validation.
- Fixed `hover`, `double_click`, `context_click`, and `drag` failing on targets below the fold or on scrolled pages, most visibly in `placement: "new_window"` task windows, which are shorter than the user's window. These gestures now scroll the target into view first, the obscured-target check hit-tests in the document coordinates Chrome expects instead of viewport coordinates, and the agent pointer is hidden during that check so it is not mistaken for an element covering the target. A target that still lies outside the viewport fails with `action_failed` naming the viewport instead of a raw DevTools error.
- Restored the Agent pointer setting, which had no effect since the 2.0 runtime rewrite. In a visible task tab, Standard actions now show a floating cursor with a soft glow and a caption naming the action and its target, gliding between targets over a dot-matrix veil that lights up near the cursor. Clicks, double-clicks, and right-clicks play a ripple through the dot field; typing, filling, selecting, and uploads show a focus ring; hovers move the cursor; drags travel from source to target. Same-origin iframe targets are mapped into the top viewport, and targets in cross-origin frames or on the scripting route show no pointer. The overlay lives in a closed, `aria-hidden` shadow root, receives no input, fades after 20 seconds of inactivity, is excluded from accessibility, text, and HTML snapshots, and is hidden while screenshots are captured. Captions read visible labels only, never field values. Turn it off in the popup to skip the extra glide before each action.
- Preserve safe email-verification failure codes across the host boundary while discarding native messages, recovery text, and details that could contain a code. Report missing recipient evidence separately from missing email-verification context.
- Added policy-gated email verification through `browser_credentials` with `action: "fill_email"` for signup and email sign-in. Use one fresh, authenticated Gmail message for an exact origin, pathname, recipient, purpose, and owned tab. Fill a complete-code input or ordered split inputs without exposing the code or issuing an explicit submit action. Keep existing password, MFA, human-verification, origin, and Commit safeguards.
- Accept a compatible masked recipient automatically when it fits the exact granted recipient and no conflicting address is shown. Fill the email code directly without staging or popup approval, while retaining password, MFA, and human-challenge safeguards.
- Recognize image-only ChatGPT turns, including authenticated generated-image URLs and images outside the empty assistant text body. Ignore progressive previews until rendering completes, and ignore the stale Stop button ChatGPT leaves in its hidden home composer after the first prompt.
- Save generated images through the owned tab's Download control, using its attributed download cursor and filename. Locate each image's Share button within its own gallery, so thread length and missing `alt` text do not matter, and reload the owned tab once when Chrome blocks a second download on the same page. Distinguish modal dialogs from the conversation container, preserve browser downloads, and reject ambiguous local file matches.
- Bound GPT-Control diagnostic PNG captures to a 1024 by 768 CSS-pixel viewport while retaining the 750 KB payload limit, avoiding oversized full-resolution captures on large displays.
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,8 +54,8 @@ Standard mode exposes exactly ten tools:
| `browser_wait` | Wait for load, URL, text, selector, network-idle, task-attributed download, or challenge-cleared conditions supported by the tab's route. |
| `browser_tabs` | List only tabs owned by the current task, including each tab's automation route. |
| `browser_handoff` | Post a non-blocking attention notice asking the user to complete human-only input, then verify the page and resolve or dismiss it by notice ID. A `challenge_cleared` completion ties the notice to a detected bot challenge. |
| `browser_commit` | Execute one staged consequential action, including an approved bookmark removal. |
| `browser_credentials` | Prepare and fill an origin-matching 1Password login through opaque, short-lived host tokens. Available by default when the local `op` CLI is usable; owner-only policy can disable or constrain it. |
| `browser_commit` | Execute one human-approved staged action, including bookmark removal. |
| `browser_credentials` | Prepare and fill an origin-matching 1Password login through opaque, short-lived host tokens, or enter a policy-approved signup or email sign-in code privately. A compatible masked recipient is accepted when it fits the exact granted recipient and no conflicting address is shown. |
| `browser_profile` | Search and organize bookmarks, search history, list recently closed or synced sessions and reopen one as background task tabs, and find, cancel, pause, or resume downloads. Works on the Chrome profile rather than a task tab and never focuses Chrome. Bookmark removal always stages for popup review; history is read-only. |
| `browser_finish` | Finish the task, apply its cleanup policy, return closed and retained tab receipts, and release ownership. |

Expand Down
8 changes: 6 additions & 2 deletions docs/adr/0001-agenttab-runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ A task workspace is visible in Chrome. Task-owned tabs are grouped for display,

**Needs your attention** is the human-only input boundary. AgentTab MUST route passkeys, security keys, CAPTCHA, payment secrets, account recovery, unsupported verification, and any credential workflow that returns `needs_user` to an advisory attention notice. The managed 1Password broker is available by default and MAY fill an origin-matching Login item through the private host-to-extension path, but MUST NOT expose the value to an agent or submit the form. Owner-only policy MAY disable or constrain the broker.

*Amended 2026-10-30:* An explicitly enabled email-verification broker MAY fill signup or email-sign-in codes through the same private host-to-extension boundary. It MUST bind an exact managed origin, pathname, recipient, purpose, and authenticated Gmail message, return metadata only, and leave password, MFA, passkey, recovery, human-verification, and Commit safeguards unchanged.

*Amended 2026-09-07:* Notices are non-blocking display metadata. A request MUST NOT pause the scheduler, gate command admission, focus a tab, activate a window, or open the popup; no global observation blackout or host handoff admission state exists. The agent asks the user in chat, the user reports back, and the agent MUST verify the page itself before resolving or dismissing the notice by ID. AgentTab MUST NOT capture human keystrokes, and no tool result may report the user's work as completed on creation. Only the user's explicit popup Open tab action may focus the noticed tab.

### Commit
Expand All @@ -47,7 +49,7 @@ Task ownership is an execution and coordination boundary. It is not cookie, iden
Standard mode MUST NOT expose:

- raw cookies or browser storage
- credential values, saved-item lists, passkeys, payment secrets, or human-only input; an explicitly enabled managed broker MAY fill one origin-matching Login item without returning its value
- credential values, saved-item lists, passkeys, payment secrets, or human-only input; a managed broker MAY privately fill an origin-matching Login item or an explicitly granted email-verification code without returning its value
- arbitrary JavaScript execution
- raw Chrome DevTools Protocol access
- coordinate-based actions
Expand Down Expand Up @@ -93,7 +95,9 @@ MCP, OMP, CLI, TypeScript, and Python are adapters over Core RPC. They are not a
9. `browser_profile`
10. `browser_finish`

`browser_credentials` is enabled by default. Managed policy MAY disable or constrain it. It MUST derive the page origin and task ownership in the host, enforce a candidate and attempt limit no greater than three, use one-use short-lived tokens, and keep credential values out of Core RPC, adapters, responses, and audit output.
The 1Password actions of `browser_credentials` are enabled by default. Managed policy MAY disable or constrain them. They MUST derive the page origin and task ownership in the host, enforce a candidate and attempt limit no greater than three, use one-use short-lived tokens, and keep credential values out of Core RPC, adapters, responses, and audit output. The separate `fill_email` action MUST be disabled by default and require an exact email-code grant, fresh authenticated mail, and the same ownership, revision, and private-value boundaries.

On a compatible masked-recipient form, `fill_email` MUST enter the code directly when the bounded ASCII `*` mask fits the exact granted recipient and no conflicting address is shown. Every mask on the page MUST be the same bounded mask. An absent or incompatible recipient MUST fail before any write. No email-specific stage, popup approval, or `browser_commit` step is required. Retain every password, MFA, passkey, and human-challenge safeguard.

`browser_profile` operates on the Chrome profile rather than a task tab. Its read operations MUST NOT carry an idempotency key and its write operations MUST. Bookmark removal MUST always stage for popup review regardless of YOLO mode, session restore MUST open owned background tabs instead of calling `chrome.sessions.restore`, and the host MUST refuse session restore while origin policy is constrained. Audit MUST NOT record its titles, URLs, or query text.

Expand Down
20 changes: 18 additions & 2 deletions docs/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,8 +68,8 @@ For MCP, the capability store namespace is `mcp`; OMP uses `omp`; Pi uses `pi`.
| `browser_wait` | Requires `tab_id` and one load, URL, text, selector, value, network-idle, download, or `challenge_cleared` condition. Selector waits may set `attached`, `detached`, `visible`, `hidden`, or `enabled` state and scope to a `frame_id`; value waits require selector and expected value and may scope to a frame. A download wait may resume after only its opaque tab-owned `after` cursor. A `challenge_cleared` wait takes the `challenge_id` of an active challenge episode on the tab and completes with `matched: true` once the episode clears or is superseded; if the episode reports it no longer expects automatic clearing, the wait returns early with outcome `needs_user` and result `status: "challenge_needs_user"`; an unknown, ended, or other-task episode fails with `challenge_not_found` and outcome `not_started`, and a timeout carries the latest event in `error.details.challenge`. `timeout_ms` is at most 120 seconds. Load and URL waits also follow Chrome's top-frame navigation events, with polling as a fallback: a load condition matches a top-frame load that completes after the wait starts, and a URL condition also matches committed navigations, same-document history changes (`pushState` and `replaceState`), and fragment changes seen during the wait. A top-frame navigation error during a load or URL wait fails with `navigation_failed`, `outcome: "unknown"`, and Chrome's network error name, never the URL; a superseded or cancelled navigation does not fail the wait. A `tab_only` route accepts load and URL conditions only; frame-scoped reads, value/selector state, network-idle, and download attribution, and challenge waits require the tab-scoped debugger connection available on the `full` route. |
| `browser_tabs` | Takes an empty object and lists only the current task's tabs, including each tab's `automation_route`. |
| `browser_handoff` | Takes one of four operations. `request` requires a task tab, expected page revision, and prompt; optional `completion: {kind: "url"\|"selector"\|"challenge_cleared", value}` and `timeout_ms` (default five minutes, reminder expiry only). It creates a `notice_id` scoped to the task and tab and returns immediately without pausing, focusing, or blocking any agent work. `status`, `resolve`, and `dismiss` take that `notice_id`. `status` also reports `user_idle_state` as `active`, `idle`, or `locked` from Chrome's idle detection, where `idle` means no keyboard or mouse input for 60 seconds; the field is omitted when Chrome cannot report it. The agent tells the user in chat, verifies the page itself, then resolves or dismisses; a `resolve` against an unmet condition fails with `completion_not_met` and leaves the notice open. Works on a `tab_only` route, but selector and `challenge_cleared` completion require the `full` route. A `challenge_cleared` completion takes the `challenge_id` of an active challenge episode on the request's tab; an unknown, ended, or other-task episode, or one on another tab, fails with `challenge_not_found` and outcome `not_started` and creates no notice. The notice then binds to the episode, so each later challenge event carries its `notice_id`. `resolve` rescans the tab first and succeeds only when the episode has cleared or been superseded, returning the terminal event as `challenge`; otherwise it fails with `completion_not_met`, `error.details.challenge` holds the current active event, and the notice stays open. |
| `browser_commit` | Requires the staged token returned by a prior `commit_required` action and executes that one staged operation. On a `tab_only` route, only a staged close can execute; page-dependent staged actions require the `full` route. A `browser_profile` bookmark removal stage is bound to the task rather than a tab. |
| `browser_credentials` | `prepare` requires a task tab and expected page revision, then returns an opaque short-lived token when the default-enabled 1Password broker is available and one through three Login items match the host-derived current origin. Owner-only policy can disable it. `fill` consumes that token and selected username, password, or one-time-code field refs without returning any value. `next` advances to another bounded candidate. It never submits the form. |
| `browser_commit` | Requires the staged token from a prior `commit_required` result and human popup approval. Execute one bound action. On a `tab_only` route, execute only a staged close; page-dependent actions require the `full` route. A `browser_profile` bookmark removal stage is bound to the task rather than a tab. |
| `browser_credentials` | `prepare` requires a task tab and expected page revision, then returns an opaque short-lived token when the default-enabled 1Password broker is available and one through three Login items match the host-derived current origin. `fill` consumes that token and selected username, password, or one-time-code field refs without returning any value; `next` advances to another bounded candidate. Separately, opt-in `fill_email` takes a Gmail message ID, recipient, signup or email-sign-in purpose, and ordered `otp_refs`, bound to an exact managed origin and pathname. A compatible masked recipient is accepted when it fits the exact granted recipient and no conflicting address is shown. Use `browser_act` for explicit submission; you may complete verification on entering the final code character. |
| `browser_profile` | Requires one `operation` over the Chrome profile rather than a task tab: bookmark search, tree, update, create, move, and remove; history search; recently closed and synced sessions, plus restore; download search, cancel, pause, and resume. Read operations take no idempotency key and write operations require one. `bookmarks_remove` always returns `commit_required`. Never focuses Chrome. See [Profile operations](#profile-operations). |
| `browser_finish` | Accepts `disposition: "auto" | "close" | "keep"` and optional task-owned `keep_tab_ids`. Automatic mode follows the popup cleanup policy: close task-created tabs while retaining adopted tabs, ask for confirmation, or retain all tabs. Successful finalization ungroups retained tabs, releases ownership, closes the Core connection, and returns closed and retained tab IDs. A tab Chrome has not closed two seconds after removal, as when a "Leave site?" prompt holds it, appears in neither list and closes if the user confirms leaving. An open handoff notice never defers finalization; staged Commit review and other in-flight work defer it without destroying resumability. When the extension refuses to finish, such as for a `keep_tab_ids` entry the task does not own, the error carries the extension's own code and message, and the task stays open. |

Expand Down Expand Up @@ -142,6 +142,22 @@ Raw TypeScript and Python SDK clients raise `AgentTabTransportError` for an ambi

Call `browser_credentials` on an ordinary sign-in page before requesting manual password entry. The broker is available by default unless owner-only policy disables it. `prepare` derives the current origin from host-owned tab state. `fill` accepts only accessibility refs and returns filled-field booleans; credential material never crosses Core RPC. Submit separately through `browser_act`, inspect the result, and use `next` only after the site rejects the current candidate.

For an explicitly granted signup or email-sign-in page, use `action: "fill_email"`
with `tab_id`, `expected_page_revision`, `message_id`, `recipient`, `purpose`
(`signup` or `email_sign_in`), and `otp_refs`. One ref receives the complete
code; multiple refs receive one character each in document order. No code,
origin, credential token, or mailbox query belongs in this request. The host
reads only that message from the configured Gmail mailbox and returns
`status: "filled"`, `filled_email_code: true`, `filled_fields`, and
`page_revision`. The email broker is disabled by default; see
[setup](setup.md#email-verification-broker). Submit separately and retain
normal attention notices for MFA, passkeys, recovery, and human verification.

`fill_email` accepts a full recipient or one compatible bounded ASCII `*`
mask repeated consistently on the page, provided no conflicting full address
that could match the mask is shown. It fills directly without popup review.
Missing or incompatible recipient evidence fails before any write.

Call `browser_handoff` with `operation: "request"` when credential preparation returns `needs_user`, the bounded candidates fail, or the site requires a passkey, security key, CAPTCHA, payment secret, account recovery, or unsupported verification. The request records an advisory notice for that exact task tab and returns its `notice_id`; it does not pause the scheduler, focus or activate the tab, open the popup, or block any browser operation for any task. The popup lists the notice with an optional Open tab and Dismiss control for the user.

The agent then tells the user in its own chat what to finish, waits for the user to report back, and verifies the page itself with a fresh snapshot or wait. When the step looks done, it calls `resolve` with the `notice_id`; when a provided completion condition has not been met, `resolve` fails with `completion_not_met` and the notice stays open. `dismiss` drops the reminder without claiming success, and `status` re-reads one notice. Terminal operations are idempotent and scoped server-side to the authenticated task. Expiry only retires the reminder; it never cancels work or fabricates success.
Expand Down
Loading
Loading