Skip to content

fix(browser): forward non-printable keys to the embedded browser (#1374) - #1850

Open
JasonBuildAI wants to merge 1 commit into
TencentCloud:developfrom
JasonBuildAI:codex/browser-keyboard-input
Open

JasonBuildAI wants to merge 1 commit into
TencentCloud:developfrom
JasonBuildAI:codex/browser-keyboard-input

Conversation

@JasonBuildAI

@JasonBuildAI JasonBuildAI commented Oct 9, 2026 •

Copy link
Copy Markdown

Summary

修复 #1374 的键位部分:工作台内嵌浏览器此前只处理 Enter,其余非可打印按键(退格 / Delete / 方向键 / Tab 等)被静默丢弃,页面输入框只能追加字符、无法删除。

仪表盘对每个非可打印按键都会发送 {"type": "keydown", "key": ...}(见 dashboard/src/components/BrowserWorkspace/index.tsx:243、dashboard/src/hooks/useDesktopCanvasInteraction.ts:47),但服务端 _handle_client_event 只对 Enter 调用 sess.type("\n"),其余键没有任何处理;修复前 rg dispatchKeyEvent src tests dashboard/src 为 0 命中,即从未下发过 CDP 按键事件。

改动:为仪表盘实际会发送的非可打印按键下发 Input.dispatchKeyEvent(rawKeyDown + keyUp),携带 CDP 虚拟键码与 KeyboardEvent.code;键表与 src/octop/api/routers/mobile/stream.py 的 _BROWSER_KEY_TO_ANDROID 覆盖同一组按键。可打印字符仍走原有 type 通道,Enter 仍走原有 type("\n") 通道,其余行为不变。

范围说明:#1374 还报告了「鼠标点击需要左右偏移才能命中」。那是取景框 / 缩放坐标换算的另一个问题,需要真实浏览器才能复现,本 PR 不做任何相关改动,准备单独跟进。

已知限制(既有行为,本 PR 未改变):modifiers 未转发,因此 Ctrl+A/C/V、Shift+方向键选区等组合键仍不生效;前端目前只发送 key,支持组合键需要前端一并改动,属另一条链路。

Target branch

  • Base is develop (feature / fix — default)

Type of change

  • Bug fix

Test plan

先复现(改动前的代码):

$ uv run pytest tests/unit/api/test_browser_stream_keys.py -q
E       assert 0 == 2
E        +  where 0 = len([])
12 failed, 2 passed in 4.92s

失败断言即「Backspace 应下发两次 Input.dispatchKeyEvent,实际 0 次」;两个通过的用例是既有行为的回归锁(未知按键忽略、Enter 仍走 type("\n"))。

改动后:

$ uv run pytest tests/unit/api/test_browser_stream_keys.py -q
14 passed in 0.98s

$ uv run pytest tests/unit/api/test_browser_stream_keys.py tests/unit/api/test_browser_stream_mouse.py \
    tests/unit/api/test_browser_stream_listen.py tests/unit/api/test_browser_stream_normalize_nav.py -q
30 passed in 3.12s

静态检查:

$ uv run ruff check src tests
All checks passed!
$ uv run ruff format --check src tests
1169 files already formatted
$ uv run mypy src/octop
Success: no issues found in 565 source files

全量测试:

$ uv run pytest -n auto -m "not live" -q
53 failed, 4337 passed, 157 skipped, 41 warnings in 887.99s (0:14:47)

这 53 个失败与本改动无关(集中于数据库迁移、agent manager、vendor wheels、登录验证码等),在未改动的 develop 基线上同样失败(已在其上复现同样的 25 个用例)。

关于 make all:本机没有可用的 make(Makefile 需要 git-bash;环境只有 WSL bash 且未安装 git-bash),因此按 make all 的等价步骤逐条执行了 Ruff 格式 / lint、mypy、pytest -m "not live";本 PR 不涉及前端文件,故未跑 Prettier / tsc。

  • make all passes locally — 无法执行,已用上述等价命令逐条替代
  • Added/updated tests

Checklist

  • Updated CHANGELOG.md (if user-facing)
  • README / docs updated (if needed) — 未改动协议文档之外的内容

关联 issue:#1374(仅键盘部分;点击偏移部分另开)

AI disclosure: this change was drafted with AI assistance at my request, under my direction and review. 本次改动由人类设定范围(只修键盘部分、最小改动、不夹带无关变更)并评审后提交,验证命令与输出均如实附上。

The dashboard forwards every non-printable key as
``{"type": "keydown", "key": ...}``, but the browser stream only acted on
Enter and silently dropped the rest. Editing text in the embedded browser could
therefore only append characters -- Backspace, Delete and the arrow keys did
nothing.

Dispatch ``Input.dispatchKeyEvent`` (rawKeyDown + keyUp) with the CDP virtual
key code and ``KeyboardEvent.code`` for the keys the dashboard actually sends.
Printable input keeps using the existing ``type`` path and Enter keeps its
``type("\n")`` path, so nothing else changes.

Refs TencentCloud#1374
@JasonBuildAI
JasonBuildAI force-pushed the codex/browser-keyboard-input branch from 8243b34 to 0eb98d0 Compare October 15, 2026 17:39

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant