Skip to content

feat(browser): read a page's text and HTML without screenshots - #39361

Open
buluoray wants to merge 1 commit into
mainfrom
crew/slack/browser-snapshot-text
Open

buluoray wants to merge 1 commit into
mainfrom
crew/slack/browser-snapshot-text

Conversation

@buluoray

@buluoray buluoray commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Let the agent read what a web page says as text, instead of scrolling and screenshotting it one screen at a time.

The browser tool's snapshot lists only controls and headings, and screenshot returns pixels. A user testing tutorial pages reported that the outline was missing almost all of the page: no paragraphs, no code blocks. Their agent fell back to scroll-and-screenshot loops of about 550 px each, which were slow and which the host flagged as a loop.

Why it matters

Each frame costs image tokens, and text read off pixels cannot be quoted or searched exactly. A page that only renders in a signed-in browser cannot fall back to web_fetch, so today the screenshot loop is the only way to read it. The reflex loop's text is no substitute: it needs the opt-in decision model, reads only the screen and returns about 1,500 characters.

Not a goal

What changed (motivation → approach → change)

This carries #36027 by @smeyffret onto current main, with credit in the commit trailer. It adds one op, read, with a format of text or html. Text is the whole page's rendered text, plus open shadow roots and same-origin frames, which innerText skips. HTML is a scrubbed copy: typed values, secret attributes, credential URL parameters, scripts and comments are blanked, and the live DOM is untouched.

The walker side is in browser_harness/snapshot_walker.js, the handler in browser_harness/ops.py, and the shim rendering in mcp_tools/browser.py. The op runs through run_op, so the URL gate and the takeover fence apply as they do for snapshot. It needs no preview setting. Results page at 18,000 characters, with next_offset and a digest that changes when the page changes. Redaction runs on the full content before paging, so a credential cannot be split across two pages, and it follows the owner's redaction switch like the other owner-only paths. A new fence pair in context_assembly/markers.py wraps the page text as untrusted data. security_posture.py lists the handler as an inbound redaction site.

The tool description (mcp_tools/catalog/browser.py), config/prompt.md, the built-in skills builtin_skills/web-browse/SKILL.md and builtin_skills/kirocrew-commands/SKILL.md, and the browser spec tell agents to use read instead of screenshots and scrolling. The tool description, the web-browse skill and the spec also say that text holds only what is shown. An unselected tab and a closed details section are left out, so the agent clicks or opens them first, or reads format: html, which keeps them. A note was chosen over an opt-in "include hidden text" mode: it adds no new read path, and hidden text is often stale or for another platform.

Changes since #36027: the description conflict is merged with main's type steps text. The redaction tests now use a synthetic GitHub token with the switch on, because AWS key IDs stay visible since #37408. A new walker test reads a tutorial-like page.

Backwards compatibility

Compatible: a new op and a new optional walker call; every existing op, argument and result is unchanged.

Removes nothing: the diff adds an op, a fence pair and an optional scanFrames parameter whose default keeps today's walk.

Tests

  • website/src/test/snapshotWalker.read.test.ts runs the real walker source in happy-dom. The new case uses a tutorial page with headings, prose, a pre code block, tabs, a collapsed details section and a same-origin frame. snapshot leaves out the prose and the code there. read returns the prose, the code, the active tab's text, the summary and the frame's text. The other cases cover text below the fold, shadow text, hidden roots and frames giving nothing, the HTML scrub, and the live DOM staying unchanged.
  • test/test_browser_read_op.py: paging gives back the full content. A credential that spans a page boundary is redacted. Bad format, offset and ref are refused, and a stale ref gives ref_stale. A refused URL refuses read. The digest is stable for the same page and changes with it. A forged fence stays inside the fence. The result fits the shim's 20,000-character cap.
  • test/test_browser_guard_isolated_count.py: the source pins match the new scanFrames( signature.
  • Local: every test/*browser*.py file (2,173 passed) and the walker vitest files (754 passed).

Proof: not provable -- other: a new op; its tests have nothing to fail against before it exists

Manual verification

On #36027, the walker's read ran in real headless Chromium on a page with an open shadow root, a direct shadow text node, a same-origin iframe, a hidden iframe, a hidden script and a textarea. text returned the light, shadow and frame text once each, and nothing from the hidden parts. On this branch, the walker's read ran in headless Chromium 154 (--dump-dom on a local file, no network) on a page with an active tab, a hidden tab panel, a display:none tab panel, a closed details and an open one. text returned the active tab, both summaries and the open section, and left out both hidden panels and the closed body. html kept all of them. No end-to-end run through a live gateway was done.

Real-machine verification

N/A -- a new read-only op with no base behaviour to show red; the walker and handler are covered by the tests above, and no gateway was started on this shared host.

Screenshots / video

Why no screenshot: the only file under website/ is a unit test; the op has no UI and changes nothing the dashboard renders.

Related Issues

Closes #35742
Carries #36027

Checklist

  • A Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...); PR Hygiene does not count commits
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Sensitive Change Review

Sensitive files changed:

  • src/kiro_crew/security_posture.py

  • No reasoning template for 7fc1f2b yet.

How to clear this check:

  1. The PR author posts the template below as a PR comment, with
    every field filled (N/A needs a reason). Keep the
    Head commit line: it must name the current head SHA.
  2. The check turns green within a minute, with no push.
  3. A new push resets this check: post it again for the new head.

Only the author's comment counts this way. A reviewer can instead
paste the filled template into an "Approve" review on this head.
Merging still needs a reviewer's approval.

## Sensitive change review
- Head commit: 7fc1f2b074a04a2797ef579d9ff01dc6c806c1f6
- What rule changed:
- Before -> after (what was allowed/blocked, now):
- Worst case if wrong (breaks commands / leaks secret / locks user out):
- Evidence checked (which sandbox result proves it):
- No regression:
  - Tools still work (which tools/commands were run, result):
  - Backward compatible (old config / old data / old callers still work, how checked):
  - Existing tests (which suites passed on this head):
- How to undo:

@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Nov 19, 2026
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ✅ PASS

UX-level review of 7fc1f2b074a04a2797ef579d9ff01dc6c806c1f6 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

UX-Verdict: PASS

Nothing to check. The only file under website/ is a unit test, and the new read op is an agent tool, so the dashboard shows no new or changed control.

[UX-REVIEWED] 7fc1f2b

Description read

[DESCRIPTION-READ] 87c5a5862745e96cae5663c22a9153a8093ffb33a7f4a825ceab9835dabdb694

That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 7fc1f2b074a04a2797ef579d9ff01dc6c806c1f6 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: read reuses the run_op gate, takeover fence and redactors, adds no UI, and takes nothing away.

Notes

  • ops.py now imports the private _neutralize_fence_markers from context_assembly/markers.py, which ties the browser harness to a context-assembly internal. Make it public, or move the neutralizing step into the shim, before a third caller copies the pattern.

[DESIGN-REVIEWED] 7fc1f2b

Description read

[DESCRIPTION-READ] 4a52eb98b52a6d6f2b2b6c266e7ecd23e3ff499e2cd905a01102061ae247cac3

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 7fc1f2b074a04a2797ef579d9ff01dc6c806c1f6 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The new read op answers the screenshot-loop report in #35742, and it fences its output the same way as the other untrusted-content paths. The HTML format reaches past a Goal that asks only for text.

Notes

  • Item 2 — oversized: the Goal is "read what a web page says as text", but readHtml/scrubCopy/scrubUrl/secretParam/publicMeta add about 100 lines of in-page scrubbing that only format: "html" uses. The only harm named for it is hidden tabs and closed details, and the description already handles those with "click or open them first". Subtraction: delete format: html and READ_FORMATS, and keep the click/open guidance. Clears when: the mechanism is removed, or a human amends the Goal.

What this change ships

Inventory (8 items) — 7 justified

Intent: let the agent read a page's full rendered text without scroll-and-screenshot loops. ADDITION.

  1. Agent can read a whole page's rendered text, including shadow roots and same-origin frames — justified
  2. Agent can read a scrubbed HTML copy of the page — oversized
  3. A read can be scoped to one element by ref — justified
  4. Long pages come back in 18,000-character pages with next_offset and a change digest — justified
  5. Page text arrives inside a new untrusted-page fence, with forged fences neutralized — justified
  6. Credentials are redacted across the whole content before paging — justified
  7. read counts as read-only after a hand-back — undeclared
  8. The system prompt, skills and tool description tell agents to use read instead of screenshots — justified

[FIRST-PRINCIPLES-REVIEWED] 7fc1f2b

Description read

[DESCRIPTION-READ] f17feb0ffb04c3c3755ac76b6dfd6c1c821ae6882a701f8fcdbcccb2da192e6d

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ⚠️ review incomplete

GPT 6.1 did not produce a complete verdict for 7d795564dacd15015b302840001acd445571f098; inspect the workflow logs and re-run it.

This comment is updated in place on each push.

See the GPT 6.1 Review job logs; this commit has no completed GPT verdict.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 7d795564dacd15015b302840001acd445571f098 false-positive: <reason, 20 to 500 characters>

Add a `read` op to the browser tool. It returns the page's text for the
whole page (not only the screen), plus open shadow roots and same-origin
frames, or a scrubbed copy of its HTML. Results page at 18,000 characters
with `next_offset` and a `digest`. It runs through `run_op`, so the URL gate
and the takeover fence apply as for `snapshot`, and it needs no preview
setting. The tool description tells agents to prefer it over screenshots
and scrolling.

`snapshot` lists only controls and headings, so a reading-heavy page
(prose, code blocks) came back almost empty and agents fell back to
screenshot loops.

Carried from #36027 onto current main: the description conflict is merged
with main's `type` steps text, the redaction tests use a synthetic GitHub
token with the owner's switch on (AWS key IDs stay visible since #37408),
and a walker test reads a tutorial-like page.

Co-authored-by: smeyffret <smeyffret@gmail.com>
@buluoray
buluoray force-pushed the crew/slack/browser-snapshot-text branch from 7d79556 to 7fc1f2b Compare November 19, 2026 02:34
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: checking Automated validation is still running

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Browser tool: read a page's text and HTML instead of screenshotting it

1 participant