Skip to content

feat(browser): screenshot a whole page, one element, or a zoomed region - #36307

Merged
javenciu merged 2 commits into
mainfrom
feat/browser-screenshot-options
Nov 14, 2026
Merged

javenciu merged 2 commits into
mainfrom
feat/browser-screenshot-options

Conversation

@javenciu

@javenciu javenciu commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The agent's screenshot op can capture a whole page, one element, or a magnified region, and can cap the image size, so it reads small text and long pages in one call at a size it chooses.

Today screenshot takes no arguments: it always saves the visible viewport at the browser's full resolution. The helper behind it already accepts full and max_dim (browser_harness/helper_worker.py), and the decision hooks already pass max_dim to keep their own frames small (browser_harness/decide.py), but the op never forwards either. A page longer than the viewport takes a scroll-and-shoot loop, one call per screen. A single control or a line of small text arrives as a whole viewport, so the agent cannot read it at a useful size and pays for pixels it does not need.

Why it matters

Each extra screenshot is an image the model re-reads on every later turn. A long page costs several screens where one capture would do, and a 60 by 20 pixel label read from a full viewport is hard to make out. A size cap lets the agent keep images small when it only needs layout, and an element or region capture shows the part that matters at a readable size.

Not a goal

  • Changing what a screenshot with no arguments does.
  • Relaxing any screenshot guard. Every form runs the same checks, before and after the capture, as today's screenshot.
  • Returning the image inside the tool result.

What changed (motivation → approach → change)

Symptom → cause: the op ignores its arguments, and the helper has no way to photograph one box.

  • browser_harness/ops.py: screenshot reads four optional arguments. full (whole scrollable page) and max_dim (longest side, 64 to 4096 pixels) go to the existing helper. ref (one element, measured the way click measures it) and region ([x, y, width, height], the pixels click uses; a list because the browser tool passes only scalars and lists of them) go to a new box helper. At most one of full, ref and region picks what is photographed. An option that is not valid, or that conflicts with another, is left out and named in a warning on the result, and the screenshot is taken as without it (the visible page by default); a ref whose element is gone or has no size falls back the same way. A region is magnified up to 4 times, fitted to max_dim (default 1568 for a region, 4096 for an element).
  • browser_harness/helper_worker.py and supervisor.py: a new screenshot_clip helper writes one box of the page with Page.captureScreenshot, adds the page scroll offset, and scales so the longest device-pixel side stays within max_dim. run_op checks the current URL right before screenshot_clip runs, as it does before screenshot and every other page helper.
  • browser_harness/snapshot_walker.js: resolve also answers the element's width and height.
  • mcp_tools/browser.py: the browser tool adds the op's warning, when there is one, to the text the model reads.
  • mcp_tools/catalog/browser.py, docs/browser-control.md, docs/system-specs/modules/browser.md: the arguments are described.

Backwards compatibility

All four arguments are optional. With none, the helper is called with the path only, exactly as before (a test pins the keyword set), and the result is the same {ok, path}. No option makes a call fail where main succeeds: a bad or conflicting option, or a ref that cannot be used, still saves the visible page, with a warning naming what was left out. The walker's resolve gains two keys; its callers read x, y and tag and ignore the rest. No setting is added because nothing changes unless the agent asks.

Tests

  • Before: on main bd08975e0a, 29 of the new tests fail: the op ignores full, max_dim, ref and region and names no option it left out, the tool shows no warning, a region screenshot sent through the browser tool is not captured, and the box helper and the walker's size do not exist.
  • After: they pass. The four test files for the op, the tool, the helper worker and the supervisor allowlist (which pins both helper tables equal): 209 passed. All test/test_browser*.py files and test/test_mcp_browser_tool.py: 1518 passed, 1 skipped, 1 failed (test_browser_decisions.py::test_the_image_header_is_sent_only_with_an_image_or_a_box, ModuleNotFoundError: httpx in this environment; it does not touch this change).
  • flake8, isort, mypy pass on the changed files; the black, comment-history, sync-io and flake-pattern gates pass.
  • Reproduce: python -m pytest test/test_browser_harness_ops.py test/test_browser_harness_helper_worker_cov.py -k "screenshot" -n0
  • Proof: PROVEN test/test_browser_harness_ops.py::test_a_screenshot_by_ref_photographs_the_element_box_at_its_own_size, test/test_browser_harness_ops.py::test_a_screenshot_of_a_region_magnifies_it_to_the_zoom_size, test/test_browser_harness_ops.py::test_a_region_screenshot_through_the_browser_tool_reaches_the_capture, test/test_browser_harness_ops.py::test_a_bad_or_conflicting_option_takes_the_visible_page_and_warns, test/test_mcp_browser_tool.py::test_a_screenshot_warning_reaches_the_model, test/test_browser_harness_ops.py::test_a_page_refused_just_before_a_box_capture_is_not_captured, test/test_browser_harness_ops.py::test_a_full_page_screenshot_with_a_size_cap_passes_both_to_the_helper, test/test_browser_harness_helper_worker_cov.py::test_screenshot_clip_fits_the_longest_device_pixel_side_to_max_dim (head abe1b52)

Manual verification

The new helper was run against a headless Chromium at device scale 2 on a page scrolled 100 pixels, with a 200 by 80 element and 9px text:

  • element at its own size: 400 by 160 pixels (200 by 80 CSS pixels at scale 2);
  • the same element with max_dim 200: 200 by 80 pixels;
  • a 60 by 20 region of the 9px text, magnified 4 times: 480 by 160 pixels, and the text is legible and sits in the right place despite the scroll;
  • the same region with max_dim 300: 300 by 100 pixels.

Sibling sweep

Sibling: src/kiro_crew/browser_harness/decide.py -- not affected -- it calls the helper directly with max_dim and is unchanged
Sibling: src/kiro_crew/computer_use/ -- not affected -- native-app screenshots use a different path
Sibling: src/kiro_crew/browser_harness/screencast.py -- not affected -- the live view stream does not use the op

Pattern harvest

Rule candidate: review-prompt
Pattern: an op's wrapper drops arguments its helper already supports, so a capability exists but the agent cannot reach it.

Related Issues

Complements #36166 (return the image in the tool result), which does not overlap: this change decides what is captured, that one decides how the result is delivered. #35738 edits the same files for a different fix.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

@javenciu
javenciu requested review from a team and KiroCrewCaptain as code owners November 13, 2026 02:16
@javenciu
javenciu requested a review from mbriones98 November 13, 2026 02:16
@javenciu
javenciu enabled auto-merge (squash) November 13, 2026 02:17
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 13, 2026
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision abe1b5244d0f6566eaec8d65ccf02e152754beb3 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of abe1b5244d0f6566eaec8d65ccf02e152754beb3 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. The op now forwards options its helper already supports, and the new clip helper is a small, separate addition. Every capture form runs the same guards before and after. Calls with no arguments behave exactly as before, and a test pins that.

Notes

  • A conflict between full, ref and region falls back to the visible page with a warning instead of refusing the call. The point is that no call fails where main succeeds. The trade-off is that the agent may act on an image it did not ask for unless it reads the warning, so the warning text has to stay prominent in the tool output.

[DESIGN-REVIEWED] abe1b52

Description read

[DESCRIPTION-READ] 522cb24d175fa8fcf528ec3093cefbe205274217695a5b92d501bbcd640c2f02

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of abe1b5244d0f6566eaec8d65ccf02e152754beb3 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks: every item serves the stated Goal, the no-argument call is unchanged, and no existing in-repo mechanism already captures one element or region in the browser harness.

Notes

  • Item 5 — inherited: SHOT_ZOOM_DIM = 1568 gives a region a second default cap, separate from the element default (SHOT_MAX_DIM), and nothing in the description says why the number is 1568. Subtraction: delete SHOT_ZOOM_DIM and use options.get("max_dim", SHOT_MAX_DIM) for both forms; magnify already limits a small region.

What this change ships

Inventory (7 items) — 6 justified

Intent: let the agent capture a whole page, one element or a zoomed region, at a size it picks, in one call. This is an ADDITION.

  1. screenshot now passes full and max_dim on to the helper that already accepted them — justified
  2. screenshot {ref} captures just one element's box — justified
  3. screenshot {region} captures a box and magnifies it up to 4× — justified
  4. An invalid or conflicting option is skipped and named in a warning, and the visible page is captured instead — justified
  5. A region capture is capped at 1568px when no max_dim is given (an element at 4096) — inherited
  6. The browser tool appends the op's warning to the text the model reads — justified
  7. The walker's resolve now also returns the element's width and height — justified

[FIRST-PRINCIPLES-REVIEWED] abe1b52

Description read

[DESCRIPTION-READ] a26634203b381987080152f5f78ac911324121b3cf24b95b069bfec7a56f5f00

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed abe1b5244d0f6566eaec8d65ccf02e152754beb3 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] abe1b52

Verdict parsed from the review's SHA-scoped output markers for commit abe1b5244d0f6566eaec8d65ccf02e152754beb3.

False positive or not applicable? A repository writer can comment:
/ai-review override fable abe1b5244d0f6566eaec8d65ccf02e152754beb3 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of abe1b5244d0f6566eaec8d65ccf02e152754beb3 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] abe1b52

False positive or not applicable? A repository writer can comment:
/ai-review override gpt abe1b5244d0f6566eaec8d65ccf02e152754beb3 false-positive: <reason, 20 to 500 characters>

@KiroCrewCaptain

KiroCrewCaptain commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 528 lines, more than the 400 it may approve on its own.

This PR makes screenshot forward full and max_dim to the existing helper, and adds a screenshot_clip box helper for ref and region. The new helper is listed in both helper tables (worker and supervisor), and run_op's checked_helper re-checks the current URL right before screenshot_clip runs, the same check screenshot gets. Every form still runs _screenshot_blocked before and after the capture. Those guards count frames, closed secrets, values filled through ask and secret values across the whole document, not just the viewport. So capturing past the viewport with full or a region shows nothing the guards don't already cover. A screenshot with no arguments still calls the helper with the path only, and a test pins that. A bad or conflicting option is left out and named in a fixed warning string. The only page-derived value in that warning is a ref that has already passed the ref regex, so no page text reaches it. The walker's resolve gains w and h. Existing callers read only x, y and tag. The decision hooks call the helper directly and are unchanged. Tests cover each new path: forwarding the options, the element box, region magnification, the ar

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 90f7db126a · 528 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@javenciu
javenciu force-pushed the feat/browser-screenshot-options branch from 3db2c7d to df40bdc Compare November 13, 2026 04:06
@javenciu

Copy link
Copy Markdown
Contributor Author

Rebased: 3db2c7def8 -> df40bdc4f2 onto main 27e9359419, with no change to the diff. Shard 8 failed on test_reply_format_once.py::...::test_the_table_names_every_selectable_harness, which also failed on plain main at af5ca42876 and passes on 27e9359419.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Nov 13, 2026
@javenciu

Copy link
Copy Markdown
Contributor Author

The GPT finding is right, and 8b004a625d fixes it. The browser tool's argument gate (_args_are_scalar) passes only scalars and lists of them, so the object-valued region was refused before it reached the op. region is now a list, [x, y, width, height], which the gate already admits, so the gate keeps its rule that no op takes a nested object instead of gaining an exception for one op. A new test drives a region screenshot through the tool into the op, with the gateway's answer shape, and it fails on df40bdc4f2. An object sent straight to the op is refused with bad_args. The tool description and browser-control.md now show the list form. The op, helper worker and supervisor test files pass (165 tests).

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Nov 13, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I read the whole diff on this head. The feature is useful, and most of it is good to keep:

  • Nothing new leaves the host. screenshot_clip is one CDP Page.captureScreenshot in the local helper worker, and the PNG goes to the same staged path under the snapshots folder as today.
  • The secret-field and filled-field guards run before and after every form. test_every_screenshot_form_keeps_the_secret_field_guard and the discard test cover that.
  • A screenshot with no arguments makes the same helper call as main (set(kw) == {"path"}).
  • The size is bounded: max_dim is 64 to 4096, magnify is at most 4x, and the clip scale is fitted to max_dim in device pixels.
  • The helper and supervisor tables stay equal, and there is no new regex in product code.

I am requesting changes on one part: the new refusals in _shot_options and _shot_clip.

On main, _screenshot ignores its arguments, so every screenshot call with any args succeeds and saves the visible page. On this head, the same call now fails with bad_args or bad_ref when an option is malformed or two options conflict. That covers full: "yes", max_dim: 10 or 99999, ref: "banana", a bad region, and full with ref (test_screenshot_refuses_bad_options_before_touching_the_page). A valid ref whose element is gone, or has no size, now fails with ref_stale. This repo does not take a new condition that makes a call fail where main passes, even for argument shape.

Please keep main's outcome for those cases. Take the screenshot the way main does (the visible page), and add a warning to the result that names the option that was ignored and why. The accepted forms (full, max_dim, ref, region) stay exactly as they are. The checks in _region_box can stay, as long as a failure there falls back with a warning instead of refusing.

I will approve once a malformed or conflicting option no longer turns a call main answers into a refusal.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Nov 13, 2026
@javenciu
javenciu force-pushed the feat/browser-screenshot-options branch from 8b004a6 to 32958b4 Compare November 13, 2026 15:16
@javenciu

Copy link
Copy Markdown
Contributor Author

Done in 32958b4002. No screenshot option turns a call that main answers into a refusal now:

  • A full, max_dim, ref or region value that is not valid is left out, and so are full, ref and region when more than one is given. The screenshot is taken as without them (the visible page by default), and the result carries a warning that names each option left out and why. A valid option next to a bad one still applies: {"full": "yes", "max_dim": 900} saves the visible page at 900 pixels.
  • A valid ref whose element is gone or has no size falls back to the visible page the same way, with a warning, instead of ref_stale.
  • The browser tool adds that warning to the text the model reads, so the agent learns what was skipped.
  • The accepted forms are unchanged, and _region_box keeps its checks; a failure there now falls back with a warning.

The refusal test became test_a_bad_or_conflicting_option_takes_the_visible_page_and_warns with the same 13 cases. Each one asserts a saved image, the plain helper call with the path only, and the warning. 17 tests fail on 8b004a625d and pass here, and the op, tool, helper worker and supervisor test files pass (207 tests). It is still two commits; the second one carries this change.

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Nov 13, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 13, 2026
@javenciu

Copy link
Copy Markdown
Contributor Author

Rebased: onto main 188d1b9069, which has #36652. Backend shard 3 (Linux and Windows) failed on the old head only in test_mate_conductor_parity.py, which main fixed in that PR; it passes here. The two commits change the same lines as before, and the 29 new tests still fail on this main and pass here.

@javenciu
javenciu force-pushed the feat/browser-screenshot-options branch from 32958b4 to fce936a Compare November 13, 2026 17:25
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 13, 2026
@javenciu
javenciu force-pushed the feat/browser-screenshot-options branch from fce936a to 90f7db1 Compare November 13, 2026 18:15
@javenciu

Copy link
Copy Markdown
Contributor Author

The approver's point is right, and 90f7db126a fixes it. screenshot_clip was missing from the helpers that run_op checks against the current URL right before they run, so a box capture did not get the check that today's screenshot gets. It is in that set now, so every form runs the same checks before and after the capture, as the description says. A new test, test_a_page_refused_just_before_a_box_capture_is_not_captured, sends the tab to a refused page after the guards and before the capture. On fce936aae9 the region form was photographed and only the check after the capture refused it; the ref form was already stopped, by the walker's own URL check while it measured the element. Now neither form is captured. The op, tool, helper worker and supervisor test files pass (209 tests).

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 13, 2026
The browser tool's argument gate passes only scalars and lists of them, so an
object-valued region was refused before it reached the op. The region is now a
list of four numbers, which the gate already admits; the gate is unchanged. A
test drives a region screenshot through the tool into the op.

An option that is not valid, or that conflicts with another, no longer refuses
the call. It is left out, the screenshot is taken as without it (the visible page
by default), and the result carries a warning that names it, which the tool shows
the model. A ref whose element is gone or has no size falls back the same way.
@javenciu
javenciu force-pushed the feat/browser-screenshot-options branch from 90f7db1 to abe1b52 Compare November 13, 2026 21:15
@javenciu

Copy link
Copy Markdown
Contributor Author

Rebased: onto main bd08975e0a, which has #36804. The Coverage Gate failed on the old head only on src/kiro_crew/agent_ipc_client.py (75.7%), a file this PR does not touch; #36804 added that file's missing coverage. The two commits change the same lines as before.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 13, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed the full diff at abe1b52. The change I asked for is in place, so I'm approving.

  • _shot_options no longer refuses. A malformed full, max_dim, ref or region, or two of full/ref/region together, is left out and named in the result's warning. The screenshot is still saved, and the parametrized test_a_bad_or_conflicting_option_takes_the_visible_page_and_warns pins that with the helper called on {"path"} only.
  • _shot_clip falls back to the visible page with a warning when a ref is stale or has no size, so ref_stale no longer fails the call.
  • A call with no options still makes the same helper call as main.
  • The secret-field and filled-field guards, the URL check before the box capture, and the after-capture discard cover every form, and tests cover each one.
  • screenshot_clip stays in the local helper worker with the same staged path. The helper and supervisor tables match, and there's no new regex in product code.

The cancelled Issue Gate, GPT 6.1 Review, PR Policy, PR Hygiene and Sensitive Change Review runs were superseded by passing runs on this head. Auto-merge is left as you set it.

@javenciu
javenciu merged commit 887fb43 into main Nov 14, 2026
154 of 163 checks passed
@javenciu
javenciu deleted the feat/browser-screenshot-options branch November 14, 2026 00:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants