feat(browser): discover and drive nonstandard controls

README.i18n.yaml

@@ -1,5 +1,5 @@
 # Bilingual consistency record for the repository README files.
 # README.md is the default English page; README.zh.md is its Chinese counterpart.
 # Values are git blob hashes from the last confirmed-consistent review.
-README.md: 1163cc9c9a7332a95f8b2448461243f0813ffdf2
-README.zh.md: 6ce7a8314f10dcd45316906f79991e0604b0814c
+README.md: ebbf7f40daeaad7d88e8fbbb36432408d2e40698
+README.zh.md: c7a10e409a768cc84ea6988f884927f345b9425f

README.md

@@ -49,10 +49,12 @@ The paired Playwright / extension duration ratio was **1.24** (95% CI **1.16–1
 
 | Capability | Tool | Notes |
 |---|---|---|
-| Read page | `browser_snapshot` | Structured text snapshot: title, URL, main text, numbered controls, and masked form fields; `delta: true` returns only changes |
-| Click element | `browser_click` | Click links, buttons, checkboxes, and other controls by inventory number |
-| Fill forms | `browser_type` | React/Vue-compatible input; `replace` clears the field first |
-| Press keys | `browser_press` | Keyboard events such as Enter, Tab, Escape, and arrow keys |
+| Read page | `browser_snapshot` | Structured text snapshot: title, URL, main text, numbered controls (including heuristic clickables), and masked form fields; `region` scopes text **and** inventory; `delta: true` returns only changes |
+| Click element | `browser_click` | Click by snapshot index, CSS selector, or visible text (exactly one). Prefer higher `depth` for nested heuristics. Open picker panels expose `[overlay]` options; prefer `text:"2024"` / `text:"01"` for year/month cells. Dispatches pointerdown→mousedown→focus→mouseup→click |
+| Fill forms | `browser_type` | Address by index or selector; React/Vue-compatible input; `replace` clears first; hidden date/select inputs are listed and writable. Writing a hidden input does **not** update controlled design-system pickers — click the panel cells instead |
+| Focus element | `browser_focus` | Focus by index or selector before press/type |
+| Upload file | `browser_upload` | Host reads an absolute local path (size/extension limits) into `input[type=file]`; requires approval |
+| Press keys | `browser_press` | Sends a key to the focused element (Enter, Escape, arrows, Backspace, Delete). Does not move focus via Tab, does not produce IME text, and does not synthesize form submit while a picker/dropdown overlay is open |
 | Scroll | `browser_scroll` | Viewport scrolling: up, down, top, and bottom |
 | Navigate | `browser_navigate` / `browser_open_tab` / `browser_back` / `browser_forward` / `browser_reload` | Navigation inside the controlled tab, or open a URL in a new tab and follow it (`active:false` keeps the current tab in front) |
 | List tabs | `browser_list_tabs` | List accessible tabs with stable IDs, titles, URLs, window/index metadata, and active/controlled state |
@@ -63,6 +65,13 @@ The paired Playwright / extension duration ratio was **1.24** (95% CI **1.16–1
 | Send images | `session.prompt` / `session.attachment` | Host-capability-gated image drafts, image-only prompts, and durable history previews |
 | Quote a selection | side panel composer | Text you highlight in the page appears in the composer and is sent with your next message as fenced, attributed page content |
 
+### Known limits
+
+- Closed Shadow DOM is not readable from the content script.
+- `browser_press` synthesizes keyboard events on the focused element; it does not move focus via Tab, cannot drive IME composition, and does not synthesize form submit while a floating picker/dropdown is open (persistent in-flow panels do not count). Trusted-event-only flows need CDP / `chrome.debugger` (out of scope).
+- Nested heuristic controls expose `depth`; when a shallow click does nothing, retry a higher-depth sibling on the same chain.
+- Visually hidden inputs are listed (values always masked) so agents can address them; assigning them with `browser_type` will not open panels or update React/Vue picker state — click overlay cells instead.
+
 ## Repository layout
 
 ```
@@ -78,7 +87,7 @@ scripts/install.ps1
 - **Your real browser, not a headless copy**: the model works in the page you already have open, retaining logins, sessions, and cookies.
 - **A text-first page interface**: numbered controls, stable IDs across snapshots, delta updates, and masked sensitive values make pages operable without screenshots; user-attached chat images use dsh's separate multimodal message path.
 - **Pointing instead of describing**: highlight the passage you mean and the side panel quotes it, so "explain this" needs no page tour. The quote is captured only while a panel is open, and nothing is sent until you send the message.
-- **A narrow privacy boundary**: passwords and payment-card values are always rendered as `••••` and never leave the page.
+- **A narrow privacy boundary**: passwords, payment-card values, token/OTP-like fields, and CSS-hidden form inventory values are always rendered as `••••` and never leave the page.
 - **A guarded bridge**: authenticated handshakes protect remote connections, privileged gateway methods reject non-loopback callers, and the extension binds tools to one user-controlled tab.
 
 ## Detailed installation and usage

README.zh.md

@@ -49,10 +49,12 @@ Playwright / 扩展的配对耗时比为 **1.24**（95% CI **1.16–1.34**）：
 
 | 能力 | 工具 | 说明 |
 |---|---|---|
-| 读取页面 | `browser_snapshot` | 结构化文本快照：标题/URL/正文/编号交互清单/表单字段（敏感值掩码）；`delta: true` 只返回变化 |
-| 点击元素 | `browser_click` | 按编号点击链接/按钮/复选框等 |
-| 填写表单 | `browser_type` | 输入文本（React/Vue 受控组件兼容），`replace` 清空重填 |
-| 按键 | `browser_press` | 键盘事件（Enter/Tab/Escape/方向键…） |
+| 读取页面 | `browser_snapshot` | 结构化文本快照：标题/URL/正文/编号交互清单（含启发式可点击项）/表单字段（敏感值掩码）；`region` 同时限定正文与清单；`delta: true` 只返回变化 |
+| 点击元素 | `browser_click` | 按编号、CSS 选择器或可见文本三选一点击。嵌套启发式优先更高 `depth`；打开的日期面板会露出 `[overlay]` 选项，年份/月份请用 `text:"2024"` / `text:"01"`。派发 pointerdown→mousedown→focus→mouseup→click |
+| 填写表单 | `browser_type` | 按编号或选择器定位；兼容 React/Vue 受控输入；`replace` 清空重填；隐藏的日期/下拉真实 input 可列出并写入。**写入隐藏 input 不会驱动受控日期组件**——应点击面板单元格 |
+| 聚焦元素 | `browser_focus` | 按编号或选择器聚焦，便于后续按键/输入 |
+| 上传文件 | `browser_upload` | Host 读取本机绝对路径（有大小/扩展名限制）写入 `input[type=file]`；需审批 |
+| 按键 | `browser_press` | 向当前焦点元素发送按键（Enter/Escape/方向键/Backspace/Delete）。不是真实 Tab 焦点遍历，不支持 IME；浮层打开时不会合成 form submit |
 | 滚动 | `browser_scroll` | 视口滚动（up/down/top/bottom） |
 | 页面导航 | `browser_navigate` / `browser_open_tab` / `browser_back` / `browser_forward` / `browser_reload` | 受控标签页内导航，或新开标签页并跟随（`active:false` 时保持当前页在前台） |
 | 列出标签页 | `browser_list_tabs` | 列出可访问标签页的稳定 ID、标题、URL、窗口/顺序以及活动/受控状态 |
@@ -63,6 +65,13 @@ Playwright / 扩展的配对耗时比为 **1.24**（95% CI **1.16–1.34**）：
 | 发送图片 | `session.prompt` / `session.attachment` | 按宿主能力启用图片草稿、纯图片消息和持久历史预览 |
 | 引用选中内容 | 侧栏输入框 | 在页面里划选的文字会出现在输入框，随下一条消息一起发送，并带上来源与不可信内容边界 |
 
+### 已知限制
+
+- Closed Shadow DOM 对内容脚本不可读。
+- `browser_press` 只在焦点元素上合成键盘事件，不能用 Tab 真实移动焦点，也不能驱动 IME；需要可信事件的场景需 CDP / `chrome.debugger`（不在本仓库范围）。
+- 嵌套启发式控件带 `depth`；浅层点击无效时，应改点同链路上更高 depth 的项。
+- 视觉隐藏的 input 只反映受控 picker 的当前值；用 `browser_type` 写入它们不会打开面板，也不会更新 React/Vue 组件状态。
+
 ## 组成
 
 ```

extensions/dsh-browser/manifest.firefox.json

@@ -3,7 +3,7 @@
   "name": "__MSG_extensionName__",
   "description": "__MSG_extensionDescription__",
   "default_locale": "en",
-  "version": "0.1.4",
+  "version": "0.1.5",
   "browser_specific_settings": {
     "gecko": {
       "id": "dsh-browser@lum1104.github.io",

extensions/dsh-browser/manifest.json

@@ -3,7 +3,7 @@
   "name": "__MSG_extensionName__",
   "description": "__MSG_extensionDescription__",
   "default_locale": "en",
-  "version": "0.1.4",
+  "version": "0.1.5",
   "minimum_chrome_version": "116",
   "permissions": [
     "sidePanel",

extensions/dsh-browser/package.json

@@ -1,7 +1,7 @@
 {
   "name": "dsh-browser-extension",
   "description": "Chrome and Firefox MV3 extension: sidebar chat with a local dsh instance and text-only read/operate of a user-controlled tab through the dsh browser bridge",
-  "version": "0.1.4",
+  "version": "0.1.5",
   "author": "Yuxiang Lin",
   "license": "MIT",
   "private": true,

extensions/dsh-browser/src/background/authorization.ts

@@ -9,6 +9,8 @@ const PAGE_READS = new Set(['browser_snapshot', 'browser_get_text'])
 const STATE_CHANGING_ACTIONS = new Set([
   'browser_click',
   'browser_type',
+  'browser_focus',
+  'browser_upload',
   'browser_press',
   'browser_navigate',
   'browser_open_tab',
@@ -110,15 +112,33 @@ function summarizeAction(call: ToolCall, locale: UiLocale): string {
   const frame = typeof call.args.frame === 'number' && call.args.frame !== 0
     ? localized(locale, `, iframe ${call.args.frame}`, `，iframe ${call.args.frame}`)
     : ''
-  const index = typeof call.args.index === 'number' ? call.args.index : '?'
   switch (call.name) {
-    case 'browser_click': return localized(locale, `Click element [${index}]${frame}`, `点击元素 [${index}]${frame}`)
+    case 'browser_click': return localized(
+      locale,
+      `Click ${describeTarget(call.args)}${frame}`,
+      `点击${describeTargetZh(call.args)}${frame}`,
+    )
     case 'browser_type': {
       const length = typeof call.args.text === 'string' ? call.args.text.length : 0
       return localized(
         locale,
-        `Enter ${length} characters in element [${index}]${frame} (the text is not shown in this dialog)`,
-        `向元素 [${index}] 输入 ${length} 个字符${frame}（文本内容不会显示在确认框）`,
+        `Enter ${length} characters in ${describeTarget(call.args)}${frame} (the text is not shown in this dialog)`,
+        `向${describeTargetZh(call.args)}输入 ${length} 个字符${frame}（文本内容不会显示在确认框）`,
+      )
+    }
+    case 'browser_focus': return localized(
+      locale,
+      `Focus ${describeTarget(call.args)}${frame}`,
+      `聚焦${describeTargetZh(call.args)}${frame}`,
+    )
+    case 'browser_upload': {
+      const name = typeof call.args.name === 'string' && call.args.name !== ''
+        ? call.args.name
+        : typeof call.args.path === 'string' ? basename(call.args.path) : 'file'
+      return localized(
+        locale,
+        `Upload “${safeInline(name)}” to ${describeTarget(call.args)}${frame}`,
+        `上传「${safeInline(name)}」到${describeTargetZh(call.args)}${frame}`,
       )
     }
     case 'browser_press': return localized(
@@ -152,6 +172,34 @@ function summarizeAction(call: ToolCall, locale: UiLocale): string {
   }
 }
 
+function describeTarget(args: Record<string, unknown>): string {
+  if (typeof args.selector === 'string' && args.selector !== '') {
+    return `element matching selector “${safeInline(args.selector, 80)}”`
+  }
+  if (typeof args.text === 'string' && args.text !== '' && args.index === undefined && args.selector === undefined) {
+    // click-by-text only; type uses text as payload
+    return `element with text “${safeInline(args.text)}”`
+  }
+  const index = typeof args.index === 'number' ? args.index : '?'
+  return `element [${index}]`
+}
+
+function describeTargetZh(args: Record<string, unknown>): string {
+  if (typeof args.selector === 'string' && args.selector !== '') {
+    return `匹配选择器「${safeInline(args.selector, 80)}」的元素`
+  }
+  if (typeof args.text === 'string' && args.text !== '' && args.index === undefined && args.selector === undefined) {
+    return `文本为「${safeInline(args.text)}」的元素`
+  }
+  const index = typeof args.index === 'number' ? args.index : '?'
+  return `元素 [${index}]`
+}
+
+function basename(path: string): string {
+  const parts = path.split(/[/\\]/)
+  return parts[parts.length - 1] || path
+}
+
 function displayUrl(value: string, locale: UiLocale): string {
   try {
     const url = new URL(value)

extensions/dsh-browser/src/background/tools.ts

@@ -51,6 +51,8 @@ const CONTENT_SCRIPT_FILE = 'content.js'
 const ACTION_DELTA_TOOLS = new Set([
   'browser_click',
   'browser_type',
+  'browser_focus',
+  'browser_upload',
   'browser_press',
   'browser_scroll',
   'browser_wait',
@@ -74,13 +76,21 @@ const TAB_NATIVE_TOOLS = new Set([
 const STATE_CHANGING_PAGE_TOOLS = new Set([
   'browser_click',
   'browser_type',
+  'browser_focus',
+  'browser_upload',
   'browser_press',
   'browser_scroll',
   'browser_navigate',
   'browser_back',
   'browser_forward',
   'browser_reload',
 ])
+const ELEMENT_TARGET_TOOLS = new Set([
+  'browser_click',
+  'browser_type',
+  'browser_focus',
+  'browser_upload',
+])
 /** Tools that operate on the browser tab collection rather than one page document. */
 export const TAB_MANAGEMENT_TOOL_NAMES = new Set([
   'browser_list_tabs',
@@ -807,7 +817,7 @@ function validateFrameTarget(call: ToolCall, frames: TabFrame[]): ToolAnswer | u
 }
 
 function validateElementTarget(call: ToolCall, tabId: number, frames: TabFrame[]): ToolAnswer | undefined {
-  if (call.name !== 'browser_click' && call.name !== 'browser_type') return undefined
+  if (!ELEMENT_TARGET_TOOLS.has(call.name)) return undefined
   const frameId = requestedFrame(call.args)
   const frame = frames.find((candidate) => candidate.frameId === frameId)
   const snapshotted = snapshotDocumentsByTab.get(tabId)?.get(frameId)