feat(browser): discover and drive nonstandard controls

[zzalancelot]

Summary

Whitelist-only snapshots could not discover or drive design-system controls (div buttons, date/select triggers, overlay year/month cells). This change adds pointer heuristics with depth, hidden form inventory, full pointer click sequences, overlay leaf discovery, selector/text addressing, browser_focus / browser_upload, and safer Enter handling while a floating panel is open.

Evidence (ByteDance referral apply form):

• Before: ~21 inventory items, no omit note → discovery gap, not budget truncation (maxInteractiveItems default 60)
• After artificial click on 「添加」: new standard inputs appear and browser_type works → gap was collection, not execution
• Date picker: panel can open on deep heuristic targets; year/month cells need overlay discovery or text:"2024" / text:"01"
• Writing controlled hidden inputs does not sync React picker state

Root causes

Layer	Cause
Discovery	Fixed CSS whitelist missed div/pointer controls
Discovery	Outermost-only nesting hid real picker triggers
Discovery	Overlay cells lacked cursor:pointer / roles
Action	HTMLElement.click() alone skips mousedown/focus open paths
Addressing	Index-only targeting could not reach panel cells reliably
Safety	Synthetic Enter→form.submit risked submitting apply forms

Changes

• Heuristic clickables ([heuristic]) with nested depth; filter common field-title labels; heuristic budget cap
• hiddenForms for visually hidden but focusable inputs (readable; typeable)
• browser_click: index | selector | text; full pointerdown→mousedown→focus→mouseup→click sequence
• Overlay short-text leaves as option "…" [overlay] inside panel/picker/dropdown roots
• browser_focus, browser_upload (Host reads absolute path → base64)
• region scopes main text and inventory; missing region errors
• Open Shadow DOM walk; no new permissions
• Do not synthesize form submit on Enter while an overlay is open
• Docs: press/IME limits, controlled hidden inputs, prefer higher depth

Compatibility

• Existing browser_click/browser_type index usage unchanged
• New parameters optional; approval model unchanged (upload still requires approval)
• Extension 0.1.5, bridge @yuxianglin/dsh-bridge-browser@0.0.6

Test plan

• pnpm --filter dsh-browser-extension run test (403)
• pnpm --filter @yuxianglin/dsh-bridge-browser run test (142)
• typecheck + build both packages
• fixture tests/fixtures/nonstandard-controls.html
• Live: click 「添加」; open date panel (high depth); text:"2024" / text:"01"; confirm row shows concrete month
• Regression: links/buttons, existing snapshot/type flows

Known limits

• Closed Shadow DOM still unread
• browser_press is not real Tab focus / IME
• Typing hidden inputs will not drive controlled design-system pickers
• Overlay heuristics depend on class/position/z-index + text click escape hatch

Confidence Score: 5/5

The PR appears safe to merge; no new actionable defects remain, and every previous thread is resolved.

Summary

This PR expands browser automation support for nonstandard design-system controls and addresses all previously reviewed issues.

• Discovers pointer-based controls, nested heuristic targets, overlay options, hidden form controls, and open Shadow DOM elements.
• Adds selector/text addressing, full pointer click sequences, focus and upload tools, region-scoped inventories, and safer Enter handling.
• Masks hidden form values and strengthens sensitive-field detection.
• Accumulates short file reads while preserving the upload-size boundary.
• Updates extension and bridge versions, documentation, fixtures, and tests.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[browser_snapshot] --> B[Visible controls]
  A --> C[Pointer heuristics]
  A --> D[Overlay leaves]
  A --> E[Masked hidden forms]
  B --> F[Stable element registry]
  C --> F
  D --> F
  E --> F
  F --> G[browser_click / type / focus / upload]
  H[Host file path] --> I[Bounded file read]
  I --> J[Base64 payload]
  J --> G

Loading

_{Reviews (3) · Last reviewed commit: "feat(browser): discover and drive nonsta..."}

[Yukikaze20170315]

Thanks — I ran into this while working with a real Web Components / Shadow DOM page, then reproduced one remaining naming gap in an isolated fixture against this PR.

snapshot.ts names controls through two paths:

const name = hidden ? formControlLabel(element) : accessibleName(element)

Both accessibleName() and formControlLabel() currently resolve
aria-labelledby and label[for] through the outer document.

That breaks for controls inside an open shadow root: attribute ID references are
scoped to the element's own DOM tree / shadow tree, and HTML label/control
associations are tree-scoped as well.

In a representative fixture containing an open shadow root, with the label and a
0x0 textarea in that same shadow tree:

#94 as-is   hiddenForms → textarea(textarea) [hidden]
+ fix       hiddenForms → 标题(textarea) [hidden]

The inventory, indices and 60-item budget are otherwise identical, so this is a
naming fix rather than a discovery change.

The small resolver I tested does three things:

1. resolves aria-labelledby only in el.getRootNode() — no fallback across a
   shadow boundary;
2. resolves every space-separated ID in reference order and ignores duplicate
   references after their first occurrence;
3. resolves label[for] in the control's own tree as well.

The same resolver is used by both accessibleName() and formControlLabel(), so
visible and hidden-form paths stay consistent.

Tests cover:

• shadow-scoped aria-labelledby on visible and hidden controls
• shadow-scoped label[for] on both paths
• multiple ID references in order
• duplicated ID references
• an identically named ID in the outer document that must not be borrowed
• light-DOM regression behavior

The extension suite passes with the patch (422/422), along with typecheck and
build.

Scope note: this keeps the project's existing name model (textContent of the referenced
node); it fixes tree-scoped ID reference resolution inside that model rather than
implementing the full accessible-name computation.

I did not change traversal, pointer heuristics, hiddenForms,
maxInteractiveItems, or snapshotMaxChars.

Happy to send this as a small follow-up after this PR lands, or provide the patch
here if you'd prefer to fold it into #94.