Skip to content

feat(browser): read a page's text and HTML without screenshots - #35766

Closed
smeyffret wants to merge 1 commit into
kirodotdev:mainfrom
smeyffret:feat/browser-read-text
Closed

smeyffret wants to merge 1 commit into
kirodotdev:mainfrom
smeyffret:feat/browser-read-text

Conversation

@smeyffret

@smeyffret smeyffret commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Let the agent read what a web page says as text or HTML, instead of scrolling and screenshotting it one screen at a time.

The browser tool's snapshot lists only controls and headings, and screenshot returns pixels. To read a documentation page, the agent pays one scroll and one screenshot per screen. On a signed-in docs site, nine pages took 60 to 100 screenshots, and the host flagged the repeated scroll calls as a loop.

Why it matters

Each frame costs image tokens, and text read off pixels cannot be quoted or searched exactly. A page that only renders in a logged-in browser cannot fall back to web_fetch, so today the screenshot loop is the only way to read it.

Not a goal

What changed (motivation → approach → change)

One op, read, with a format argument, so text and HTML share every gate and the paging. The reflex loop's visibleText() was not reused because it reads only the viewport.

  • Walker (snapshot_walker.js): text is the root's innerText (the whole page, not only the viewport), then the text of each open shadow root and same-origin frame, which innerText skips; real Chromium confirms that shadow text is not duplicated. A root with no box returns no text, because innerText would fall back to textContent and its script bodies. html serializes a copy imported into a document with no window, so the live DOM is untouched and nothing in the copy runs. The copy blanks every typed-field value (React mirrors typed values into the attribute), secret meta content and secret-named attributes, credential parameters in URLs (the rest of the link stays), template content and comments, and empties script and style bodies. scanFrames gains an optional start and root callback, so read counts skipped frames the same way the screenshot guard does; its default call is unchanged.
  • Handler (ops.py): validates format, offset and ref, and redacts the full content before paging, so a credential cannot be split across two pages. One result carries 18,000 characters plus total, next_offset and a digest that changes when the page changed between calls. It runs through run_op, so the URL gate and the takeover fence apply as for snapshot.
  • Shim (mcp_tools/browser.py, description in mcp_tools/catalog/browser.py): fences the page content with a new pair in the shared marker registry (context_assembly/markers.py), so a page cannot forge the shim's note or hand-back lines. read counts as a read-only op after a hand-back. security_posture.py lists browser_harness/ops.py as inbound page-text redaction, beside the shim.
  • Docs: spec row in docs/system-specs/modules/browser.md, op lists in config/prompt.md and the kirocrew-commands/SKILL.md and web-browse/SKILL.md builtin skills.

Backwards compatibility

Compatible: a new op and a new optional walker call; every existing op, argument and result is unchanged.

Removes nothing: the diff only adds an op, a fence pair and an optional scanFrames parameter whose default keeps today's walk.

Tests

  • test/test_browser_read_op.py: paging reproduces the full content, and the last page has no next_offset; a credential straddling a page boundary is redacted; bad_format, bad_offset and bad_ref refusals; a stale ref returns ref_stale; a refused URL refuses read like snapshot; the digest is stable for the same page and changes with it; a forged end fence or hand-back line stays inside the fence; the header and content fit the shim's 20,000 cap.
  • website/src/test/snapshotWalker.read.test.ts (the real walker source in happy-dom): text below the fold (viewport reads stubbed so a viewport reader would stop); shadow-root and direct shadow text; hidden roots and hidden frames give nothing; the HTML scrub (typed values, secret meta and attributes, URL parameters, template content, comments); the live DOM is unchanged.
  • test/test_browser_guard_isolated_count.py: its source pins now match the new scanFrames( signature and stop before the read section.

Manual verification

The walker's read was run in real headless Chromium (Playwright's headless shell) on a page with an open shadow root, a direct shadow text node, a same-origin iframe, a hidden iframe, a hidden script and a textarea. text returned the light, shadow and frame text once each and nothing from the hidden parts; html reported 2 skipped frames and no shadow text. An end-to-end run of the op through a live gateway is not done yet.

Screenshots / video

Why no screenshot: the only file under website/ is a unit test; the op has no UI and changes nothing the dashboard renders.

Related Issues

Closes #35742

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Nov 12, 2026
snapshot lists only interactive elements and headings, and screenshot
returns pixels, so an agent reading a docs page paid one scroll and one
screenshot per screen. The new read op returns the page content as text
(innerText of the whole page, plus open shadow roots and same-origin
frames, which Chromium's innerText does not enter) or as HTML, scoped
to a ref when given, and paged by offset.

It runs in the walker's isolated world through run_op, so the URL gate,
the takeover fence and the ref rules apply as for snapshot. A root with
no box gives no text: innerText on it falls back to textContent, script
bodies included. Frames are counted as scanFrames counts them, which
read reuses as its walk, so an object or embed document is never read.

HTML is serialized from a copy imported into a document with no window,
so nothing loads or runs and the live DOM is untouched. React mirrors a
typed value into the value attribute, so every input's value but a
button's, checkbox's or radio's is blanked, and textarea bodies are
emptied. Secret or CSRF meta content and attributes, credential query
and fragment parameters in URLs, template content and comments are
scrubbed too. The html note says it is the light DOM of one document.

The gateway redacts the whole content before paging, so a token cannot
straddle two pages, and returns a digest that changes with the page.
The shim fences the content with a new pair in the shared fence
registry and neutralizes look-alikes, so a page cannot forge a note or
a hand-back line outside the fence.

Rejected: the reflex observe's visibleText() viewport walk, which reads
only what is on screen; reopen if a caller needs what is in view rather
than the whole page. Separate text and html ops: one op with a format
argument shares every gate and the paging; reopen if the two need
different gates. Scoping read to a cross-origin frame ref: refused with
bad_ref for now; reopen if agents need frame text, through FrameOps.
Stripping whole query strings from URLs in html: it breaks doc links;
reopen if a credential parameter name escapes the list.
@smeyffret
smeyffret force-pushed the feat/browser-read-text branch from 6bcdc3f to 12505b6 Compare November 12, 2026 17:02
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Nov 12, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ✅ PASS

UX-level review of 12505b64d2b73ffbcb43c20107131ad0f957a61e via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

UX-Verdict: PASS

Nothing to check: the change adds an agent-facing browser op, and the only file under website/ is a unit test, so no dashboard control, label or state changes.

[UX-REVIEWED] 12505b6

Description read

[DESCRIPTION-READ] 4385bcd356e9880ac96bb62813eecbc4f64453208540e66a8391770b9bd4d47a

That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 12505b64d2b73ffbcb43c20107131ad0f957a61e via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

No design-level concerns. The new read op goes through the same run_op URL gate and takeover fence as snapshot. Content is redacted as a whole before it is paged, and the change removes nothing.

Notes

  • The live-gateway run hasn't been done yet. Run read once on a signed-in docs page before merge to confirm the stated goal: fewer screenshots per page.

[DESIGN-REVIEWED] 12505b6

Description read

[DESCRIPTION-READ] a12add38597e73df9ca58eb4e3c17621a018c31a6dd5d26b84e5cdfe1b32a6df

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 12505b64d2b73ffbcb43c20107131ad0f957a61e via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

The read op is aimed at the real cause: no op returns the page's full text. The one weak spot is the HTML format, a large scrubbing surface the description never ties to a harm that text does not already remove.

Notes

  • Item 2 — oversized: the Goal names HTML, but the harm in the description is the screenshot loop ("one scroll and one screenshot per screen"), and format: text already removes it. readHtml/scrubCopy/scrubUrl/secretParam/scrubParams, the srcset branch and URL_ATTRS add about 85 walker lines of credential-scrubbing heuristics that must keep up with every new secret-bearing attribute. Nothing in the description names a job that only HTML can do.
    Subtraction: delete the html branch, READ_FORMATS, the bad_format refusal and the html note, and ship text only until someone reports a job that needs markup.

What this change ships

Inventory (9 items) — 8 justified

Intent: let the agent read a whole web page, including one behind a login, as text instead of looping on scroll and screenshot. ADDITION.

  1. New browser op read returns the whole page's text, including open shadow roots and same-origin frames — justified
  2. read can return a scrubbed copy of the page's HTML (format: html) — oversized
  3. read can be limited to one element by ref (ref_stale / bad_ref) — justified
  4. A long page comes back in pages of 18,000 characters, with total and next_offset — justified
  5. Each result carries a digest that changes when the page changed between calls — justified
  6. The gateway redacts credentials across the whole content before splitting it into pages — justified
  7. Page content sits inside a new untrusted-page fence pair, with forged fences neutralized — justified
  8. read is allowed after a hand-back, like the other read-only ops — justified
  9. The prompt and builtin skills now tell the agent to prefer read over screenshots — justified

[FIRST-PRINCIPLES-REVIEWED] 12505b6

Description read

[DESCRIPTION-READ] 5b8c3ab9c75b822a349d082bd331921b08b4ff06f6bc30aabfeecfec86252959

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 12505b64d2b73ffbcb43c20107131ad0f957a61e via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 12505b6

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — 🔴 changes requested (blocking)

Reviewed 12505b64d2b73ffbcb43c20107131ad0f957a61e via the fork AI-review pipeline; updated in place on each push.

2 of 2 blocking finding(s) are security-class and were withheld from adjudication, so the blocking verdict stands.

BLOCKING -- src/kiro_crew/mcp_tools/browser.py:301 -- Page-controlled titles bypass the untrusted-content fence

head += f" -- {title}"
Trigger: A page title contains a browser fence marker and a forged handed_back claim; walker read → _read → _read_text copies it into the header.
Outcome: Page-authored markers and purported user authorization reach the agent outside the untrusted-page fence; _redacted preserves them.
Anchor: residual/security
Fix: Neutralize title markers and place the title inside the untrusted fence, keeping page-authored text out of trusted framing.

BLOCKING -- src/kiro_crew/browser_harness/snapshot_walker.js:736 -- HTML reads expose credentials embedded in iframe srcdoc

else if (URL_ATTRS[name]) el.setAttribute(attrs[a].name, scrubUrl(attrs[a].value));
Trigger: A page embeds a prefilled password or hidden CSRF field in an iframe’s srcdoc; read(format="html") → readHtml → scrubCopy preserves the embedded HTML attribute.
Outcome: Nested field values reach the agent unchanged: the DOM scrub never visits attribute-contained inputs, and downstream redactors preserve arbitrary password and CSRF values.
Anchor: residual/security
Fix: Blank srcdoc in the copied DOM, consistent with excluding frame documents from HTML reads.

[BLOCK-MERGE] 12505b6
[GPT-REVIEWED] 12505b6

Adjudication (Opus 5.5) — is blocking on each finding proportionate?

The adjudicable block is empty, so nothing could be downgraded. Both fenced findings keep blocking, and I flagged neither for override.

F1 (src/kiro_crew/mcp_tools/browser.py:301, the patch's _read_text): unbounded harm, a prompt-injection path outside the untrusted-page fence.

  • The page sets document.title (patch snapshot_walker.js +333). _read_text puts it in the header with only a 200-character cap and _redacted (patch browser.py +467, +472–473, +485), with no _neutralize_fence_markers. Only the body is neutralized (+487).
  • A page can therefore write a forged closing fence, (note: …) or a hand-back claim into the "trusted" header. Every page sets its own title, so the condition is common. There is no recovery path.
  • The real fix is small: move the title inside the fence, or neutralize it.

F2 (src/kiro_crew/browser_harness/snapshot_walker.js:736, the patch's scrubCopy): unbounded harm, credential exposure.

  • scrubCopy blanks only attributes that match a secret name or the CSRF pattern, attributes in URL_ATTRS, and srcset (patch +284–294). srcdoc is none of these, so its embedded HTML is copied verbatim by readHtml (+309–312). Hidden fields and token-bearing URLs inside it reach the agent unscrubbed.

  • Webmail and widget hosts commonly render server HTML through srcdoc, so the condition isn't so extreme that it would justify a FLAG. There is no recovery path.

  • The fix is one line: blank srcdoc in the copy.

    [ADJUDICATION] 12505b6 total=0 uphold=0 downgrade=0
    [GPT-ADJUDICATED] 12505b6

    [ADJUDICATION-FENCED] 12505b6 fenced=2 flagged=0
    UPHOLD-FENCED F1 src/kiro_crew/mcp_tools/browser.py:301 -- Every page controls its own title, which _read_text places un-neutralized in the trusted header outside the untrusted-page fence, so a forged fence or hand-back claim is plausible on ordinary pages.
    UPHOLD-FENCED F2 src/kiro_crew/browser_harness/snapshot_walker.js:736 -- scrubCopy never touches srcdoc, and server-rendered srcdoc frames (webmail, widgets) routinely carry hidden fields and token URLs, so the credential-exposure condition is realistic, not contrived.
    [GPT-ADJUDICATED-FENCED] 12505b6

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 12, 2026
@smeyffret

Copy link
Copy Markdown
Contributor Author

Superseded by #36027, the same commit on a branch of this repository.

@smeyffret smeyffret closed this Nov 12, 2026
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Browser tool: read a page's text and HTML instead of screenshotting it

1 participant