Skip to content

feat(browser): read a page's text and HTML without screenshots - #36027

Draft
smeyffret wants to merge 1 commit into
mainfrom
feat/browser-read-text
Draft

smeyffret wants to merge 1 commit into
mainfrom
feat/browser-read-text

Conversation

@smeyffret

@smeyffret smeyffret commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Let the agent read what a web page says as text or HTML, instead of scrolling and screenshotting it one screen at a time.

The browser tool's snapshot lists only controls and headings, and screenshot returns pixels. To read a documentation page, the agent pays one scroll and one screenshot per screen. On a signed-in docs site, nine pages took 60 to 100 screenshots, and the host flagged the repeated scroll calls as a loop.

Why it matters

Each frame costs image tokens, and text read off pixels cannot be quoted or searched exactly. A page that only renders in a logged-in browser cannot fall back to web_fetch, so today the screenshot loop is the only way to read it.

Not a goal

What changed (motivation → approach → change)

One op, read, with a format argument, so text and HTML share every gate and the paging. The reflex loop's visibleText() was not reused because it reads only the viewport.

  • Walker (snapshot_walker.js): text is the root's innerText (the whole page, not only the viewport), then the text of each open shadow root and same-origin frame, which innerText skips; real Chromium confirms that shadow text is not duplicated. A root with no box returns no text, because innerText would fall back to textContent and its script bodies; a display: contents shadow host counts as rendered. The page side caps the content by its JSON-escaped size, so a long non-ASCII page comes back truncated rather than failing. html serializes a copy imported into a document with no window, so the live DOM is untouched and nothing in the copy runs. The copy blanks every typed-field value (React mirrors typed values into the attribute), secret meta content and secret-named attributes, credential parameters in URLs (the rest of the link stays), template content, iframe srcdoc and comments, and empties script and style bodies. scanFrames gains an optional start and root callback, so read counts skipped frames the same way the screenshot guard does; its default call is unchanged.
  • Handler (ops.py): validates format, offset and ref, and redacts and fence-neutralizes the full content before paging, so a credential cannot be split across two pages and no page grows past the shim's cap. One result carries 18,000 characters plus total, next_offset and a digest that changes when the page changed between calls. It runs through run_op, so the URL gate and the takeover fence apply as for snapshot.
  • Shim (mcp_tools/browser.py, description in mcp_tools/catalog/browser.py): fences the page content and the page title with a new pair in the shared marker registry (context_assembly/markers.py), so a page cannot forge the shim's note or hand-back lines. read counts as a read-only op after a hand-back. security_posture.py lists browser_harness/ops.py as inbound page-text redaction, beside the shim.
  • Docs: spec row in docs/system-specs/modules/browser.md, op lists in config/prompt.md and the kirocrew-commands/SKILL.md and web-browse/SKILL.md builtin skills.

Backwards compatibility

Compatible: a new op and a new optional walker call; every existing op, argument and result is unchanged.

Removes nothing: the diff only adds an op, a fence pair and an optional scanFrames parameter whose default keeps today's walk.

Tests

  • test/test_browser_read_op.py: paging reproduces the full content, and the last page has no next_offset; a credential straddling a page boundary is redacted; bad_format, bad_offset and bad_ref refusals; a stale ref returns ref_stale; a refused URL refuses read like snapshot; the digest is stable for the same page and changes with it; a forged end fence or hand-back line stays inside the fence; the header and content fit the shim's 20,000 cap.
  • website/src/test/snapshotWalker.read.test.ts (the real walker source in happy-dom): text below the fold (viewport reads stubbed so a viewport reader would stop); shadow-root and direct shadow text; hidden roots and hidden frames give nothing; the HTML scrub (typed values, secret meta and attributes, URL parameters, template content, comments); the live DOM is unchanged.
  • test/test_browser_guard_isolated_count.py: its source pins now match the new scanFrames( signature and stop before the read section.

Manual verification

The walker's read was run in real headless Chromium (Playwright's headless shell) on a page with an open shadow root, a direct shadow text node, a same-origin iframe, a hidden iframe, a hidden script and a textarea. text returned the light, shadow and frame text once each and nothing from the hidden parts; html reported 2 skipped frames and no shadow text. An end-to-end run of the op through a live gateway is not done yet.

Screenshots / video

Why no screenshot: the only file under website/ is a unit test; the op has no UI and changes nothing the dashboard renders.

Related Issues

Closes #35742

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Sensitive Change Review

Sensitive files changed:

  • src/kiro_crew/security_posture.py

  • No reasoning template for 594ce0d yet.

How to clear this check:

  1. The PR author posts the template below as a PR comment, with
    every field filled (N/A needs a reason). Keep the
    Head commit line: it must name the current head SHA.
  2. The check turns green within a minute, with no push.
  3. A new push resets this check: post it again for the new head.

Only the author's comment counts this way. A reviewer can instead
paste the filled template into an "Approve" review on this head.
Merging still needs a reviewer's approval.

## Sensitive change review
- Head commit: 594ce0d88ea2d1ab69f21b1e4b755fc405399aca
- What rule changed:
- Before -> after (what was allowed/blocked, now):
- Worst case if wrong (breaks commands / leaks secret / locks user out):
- Evidence checked (which sandbox result proves it):
- No regression:
  - Tools still work (which tools/commands were run, result):
  - Backward compatible (old config / old data / old callers still work, how checked):
  - Existing tests (which suites passed on this head):
- How to undo:

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ✅ PASS

UX-level review of 594ce0d88ea2d1ab69f21b1e4b755fc405399aca — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

UX-Verdict: PASS

Nothing to check. This PR adds an agent-facing browser op and changes no dashboard UI. The only file under website/ is a unit test, so the missing screenshots leave no gap.

[UX-REVIEWED] 594ce0d

Description read

[DESCRIPTION-READ] 0b16c0b476f917b3cfff941bb372f80787fb1b69981dd1838798a70eb5ff7e2a

That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 12, 2026
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 594ce0d88ea2d1ab69f21b1e4b755fc405399aca — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

No design-level concerns: one gated, paged read op replaces the scroll-and-screenshot loop, and it reuses the existing run_op URL gate, takeover fence, redaction and marker registry instead of adding a parallel path.

Notes

  • ops.py and the shim both import the private _neutralize_fence_markers from context_assembly/markers.py, so two layers now depend on that helper. If neutralization isn't idempotent, the shim's second pass could grow a page past _RESULT_CAP and cut text that no next_offset points to. Making the helper public, with a pinned idempotence test, would settle both.

[DESIGN-REVIEWED] 594ce0d

Description read

[DESCRIPTION-READ] 7eabef8da6429249fe6cab40396a4d036712eb43ff33ab336a98f31c4ae07ba7

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 594ce0d88ea2d1ab69f21b1e4b755fc405399aca — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The new read op sits beside snapshot and screenshot and changes no default. Its fence, redaction and frame-count pieces each protect the external-content boundary.

Notes

  • Item 2 — oversized: "Why it matters" names only text harms ("image tokens", "cannot be quoted or searched exactly"). No harm is named that html removes and text does not. Yet html ships its own JS scrub (scrubCopy, scrubUrl, secretParam, SECRET_ATTR_RE, PUBLIC_META_RE), which is permanent redaction surface. Subtraction: drop format: "html" and its scrub from snapshot_walker.js and READ_FORMATS, and ship text only. Clears when: the mechanism is removed, or a human amends the Goal.

What this change ships

Inventory (9 items) — 8 justified

Intent: let the agent read a whole web page as text in a few calls instead of a scroll-and-screenshot loop. ADDITION.

  1. The browser tool gains read, which returns the whole page's text, including open shadow roots and same-origin frames — justified
  2. read with format: html returns a scrubbed copy of the top document's light DOM — oversized
  3. A long page comes back in 18,000-character pages with total and next_offset — justified
  4. A digest shows whether the page changed between page calls — justified
  5. Page content and title sit inside a new untrusted-page fence pair in the shared marker registry — justified
  6. The whole content is redacted for credentials before paging — justified
  7. read is allowed after a hand-back as a read-only op — justified
  8. read can be scoped to one ref; a cross-origin frame ref is refused — justified
  9. The tool description, prompt and builtin skills list the op — justified

[FIRST-PRINCIPLES-REVIEWED] 594ce0d

Description read

[DESCRIPTION-READ] f52df22d42164b433eca3423812a4aea9c06f5aaf7c3501e996da6759139300b

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 12, 2026
@smeyffret
smeyffret force-pushed the feat/browser-read-text branch from 12505b6 to 4b5bda0 Compare November 12, 2026 21:03
@smeyffret

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: read result header in the browser shim

fixed in 4b5bda0: the page title now goes inside the untrusted-page fence as a Title: line, neutralized with the content; the header keeps only gateway fields.
The URL, also page-controlled, is whitespace-collapsed and neutralized in the header.
Tests: test_a_forged_fence_or_hand_back_line_in_the_title_stays_inside_the_fence, test_a_fence_marker_in_the_url_is_neutralized_in_the_header.

@smeyffret

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: HTML scrub of the copied DOM in the walker

fixed in 4b5bda0: scrubCopy blanks every srcdoc attribute, so markup embedded in an iframe never reaches the HTML read, matching the rule that frame documents are out of html.
Test: snapshotWalker.read.test.ts "blanks an iframe srcdoc, whose markup the scrub never visits" (hidden csrf value in srcdoc is absent, srcdoc="" present).

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 14, 2026
@smeyffret
smeyffret force-pushed the feat/browser-read-text branch from 4b5bda0 to d6c5c88 Compare November 14, 2026 11:15
@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 594ce0d88ea2d1ab69f21b1e4b755fc405399aca and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 594ce0d

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 594ce0d88ea2d1ab69f21b1e4b755fc405399aca false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 594ce0d88ea2d1ab69f21b1e4b755fc405399aca — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 594ce0d

Verdict parsed from the review's SHA-scoped output markers for commit 594ce0d88ea2d1ab69f21b1e4b755fc405399aca.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 594ce0d88ea2d1ab69f21b1e4b755fc405399aca false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 14, 2026
@smeyffret
smeyffret force-pushed the feat/browser-read-text branch from d6c5c88 to e3270e9 Compare November 14, 2026 13:18
@smeyffret

Copy link
Copy Markdown
Contributor Author
  • span=cdf4c6a83fe3 Boxless shadow hosts discard visibly rendered articles
    self-added: yes
    mechanism: rendered() check on a shadow root's host in the walker's text read

fixed in e3270e9: rendered() treats a display: contents element as rendered when its parent is, so a shadow root on such a host is read; a host under a hidden parent still gives no text.
Test: snapshotWalker.read.test.ts "reads a shadow root whose host is display: contents, but not one under a hidden parent".

@smeyffret

Copy link
Copy Markdown
Contributor Author
  • span=a9917a9eb999 Fence expansion silently drops content without a continuation
    self-added: yes
    mechanism: fence neutralization order between the gateway's paging and the shim's cap

fixed in e3270e9: ops._read neutralizes fence look-alikes in the whole content before paging, so total, next_offset and the digest count the neutralized text; the shim's own pass is then a no-op (the neutralizer is idempotent) and never grows a page past its cap.
Test: test_fence_look_alikes_are_neutralized_before_paging_so_no_text_is_lost (every page's content survives the shim, the last page ends with the page's final text).

@smeyffret

Copy link
Copy Markdown
Contributor Author
  • span=293ddae6ad43 read fails outright on long non-ASCII pages
    self-added: yes
    mechanism: page-side size cap in the walker's read

fixed in e3270e9: the walker cuts the content by its JSON-escaped size (6 bytes per non-ASCII UTF-16 unit, 2 per quote or backslash, never splitting a surrogate pair) at 3.5 MiB, under helper_worker REPLY_CAP_BYTES (4 MiB), and sets truncated; the 2,000,000-character cap stays as the work bound.
Test: snapshotWalker.read.test.ts "cuts non-ASCII content by its JSON-escaped size and says truncated" (700,000 CJK characters).

@smeyffret

Copy link
Copy Markdown
Contributor Author
  • span=cdf4c6a83fe3 Nested display-contents shadow hosts lose visible text
    self-added: yes
    mechanism: rendered() check on a shadow root's host in the walker's text read

fixed in b7462aa: rendered() walks the composed parent, falling back to the shadow root's host when parentElement is null, so a display: contents host that is a direct child of a shadow root is judged by its outer host.
Test: snapshotWalker.read.test.ts "reads a display: contents host nested directly inside another shadow root"; it fails with the fix reverted and passes with it.

@smeyffret
smeyffret force-pushed the feat/browser-read-text branch 2 times, most recently from b7462aa to 22c46b0 Compare November 14, 2026 17:30
@smeyffret

Copy link
Copy Markdown
Contributor Author
  • span=cdf4c6a83fe3 CamelCase secret metadata survives the HTML scrub
    self-added: yes
    mechanism: the meta check tested a word-boundary secret regex against the raw name, so apiToken kept its content
    resolution: removed -- the meta secret-name denylist is replaced by a public-name allowlist, and the ancestor-walking visibility gate by a per-node rule (SEAM REPEAT retrospective)

Fixed in 22c46b0. A meta now keeps content only when every naming attribute (name, property, http-equiv, itemprop) is a page-describing name (description, viewport, og:* and the like), so no spelling of a secret name survives. Attribute names arrive lowercased, so the attribute scrub matches the secret word anywhere in the name (data-apitoken). The same round replaces rendered() with a per-node text read: an element with a box gives innerText, any other element or shadow root is read child by child, and a text node counts only when its range draws a box. That covers hidden ancestors, display: contents and unassigned slottables without the host gate. New tests (apiToken meta, data-apitoken, unassigned slottable under a display: contents host) fail on b7462aa and pass on 22c46b0.

snapshot lists only interactive elements and headings, and screenshot
returns pixels, so an agent reading a docs page paid one scroll and one
screenshot per screen. The new read op returns the page content as text
(innerText of the whole page, plus open shadow roots and same-origin
frames, which Chromium's innerText does not enter) or as HTML, scoped
to a ref when given, and paged by offset.

It runs in the walker's isolated world through run_op, so the URL gate,
the takeover fence and the ref rules apply as for snapshot. A root with
no box gives no text: innerText on it falls back to textContent, script
bodies included. Frames are counted as scanFrames counts them, which
read reuses as its walk, so an object or embed document is never read.

HTML is serialized from a copy imported into a document with no window,
so nothing loads or runs and the live DOM is untouched. React mirrors a
typed value into the value attribute, so every input's value but a
button's, checkbox's or radio's is blanked, and textarea bodies are
emptied. Secret or CSRF meta content and attributes, credential query
and fragment parameters in URLs, template content and comments are
scrubbed too. The html note says it is the light DOM of one document.

The gateway redacts the whole content before paging, so a token cannot
straddle two pages, and returns a digest that changes with the page.
The shim fences the content with a new pair in the shared fence
registry and neutralizes look-alikes, so a page cannot forge a note or
a hand-back line outside the fence.

Rejected: the reflex observe's visibleText() viewport walk, which reads
only what is on screen; reopen if a caller needs what is in view rather
than the whole page. Separate text and html ops: one op with a format
argument shares every gate and the paging; reopen if the two need
different gates. Scoping read to a cross-origin frame ref: refused with
bad_ref for now; reopen if agents need frame text, through FrameOps.
Stripping whole query strings from URLs in html: it breaks doc links;
reopen if a credential parameter name escapes the list.
@smeyffret
smeyffret force-pushed the feat/browser-read-text branch from 22c46b0 to 594ce0d Compare November 14, 2026 21:19
@smeyffret

Copy link
Copy Markdown
Contributor Author
  • span=cdf4c6a83fe3 Separated API-key attribute names bypass credential scrubbing
    self-added: yes
    mechanism: the attribute scrub matched secret words against the raw lowercased name, so api-key and api_key never read as apikey

fixed in 594ce0d: the scrub drops -, _, . and : from the attribute name before the secret-word test, so data-api-key and data-api_key are blanked, in the page and inside template content (the same loop runs there). The test adds all three cases; it fails on 22c46b0 and passes on 594ce0d.

@smeyffret

Copy link
Copy Markdown
Contributor Author
  • span=cdf4c6a83fe3 Text exactly at the cap silently omits additional roots
    self-added: yes
    mechanism: the per-root guard stopped at size >= READ_PAGE_CAP, so content exactly at the cap never exceeded it and truncated stayed unset

fixed in 594ce0d: the guard is size > READ_PAGE_CAP, so one more shadow root or frame pushes the content past the cap and the final cut sets truncated. A new test (2,000,000-char page plus a same-origin frame) fails on 22c46b0 and passes on 594ce0d.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 17, 2026
@buluoray

Copy link
Copy Markdown
Contributor

@smeyffret a user on our internal Slack hit exactly this gap: snapshot leaves out body text and code blocks, so their agent read workshop pages through dozens of scroll-and-screenshot calls. This PR's read op is the fix they need.

The branch is a draft and now conflicts with main (one file, src/kiro_crew/mcp_tools/catalog/browser.py). Do you plan to rebase it and mark it ready soon? If you'd rather not, are you OK with us carrying it on a new branch off main, with you credited as co-author and this PR linked? If we don't hear back by 2026-11-20 03:00 UTC, we'll go ahead that way. We won't push to your branch.

@buluoray

Copy link
Copy Markdown
Contributor

@smeyffret thanks for this. To get the fix to the user sooner, we carried it onto current main in #39361, with you as co-author in the commit. Your branch is untouched.

What changed on the way: the description conflict is merged with main's type steps text, the two redaction tests use a synthetic GitHub token with the owner's redaction switch on (AWS key IDs stay visible since #37408), and one walker test reads a tutorial-like page. Reviews and comments are welcome there.

buluoray added a commit that referenced this pull request Nov 19, 2026
Add a `read` op to the browser tool. It returns the page's text for the
whole page (not only the screen), plus open shadow roots and same-origin
frames, or a scrubbed copy of its HTML. Results page at 18,000 characters
with `next_offset` and a `digest`. It runs through `run_op`, so the URL gate
and the takeover fence apply as for `snapshot`, and it needs no preview
setting. The tool description tells agents to prefer it over screenshots
and scrolling.

`snapshot` lists only controls and headings, so a reading-heavy page
(prose, code blocks) came back almost empty and agents fell back to
screenshot loops.

Carried from #36027 onto current main: the description conflict is merged
with main's `type` steps text, the redaction tests use a synthetic GitHub
token with the owner's switch on (AWS key IDs stay visible since #37408),
and a walker test reads a tutorial-like page.

Co-authored-by: smeyffret <smeyffret@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Browser tool: read a page's text and HTML instead of screenshotting it

2 participants