Repository navigation
feat(browser): screenshot a whole page, one element, or a zoomed region - #36307
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks. The op now forwards options its helper already supports, and the new clip helper is a small, separate addition. Every capture form runs the same guards before and after. Calls with no arguments behave exactly as before, and a test pins that. Notes
[DESIGN-REVIEWED] abe1b52 Description read[DESCRIPTION-READ] 522cb24d175fa8fcf528ec3093cefbe205274217695a5b92d501bbcd640c2f02 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks: every item serves the stated Goal, the no-argument call is unchanged, and no existing in-repo mechanism already captures one element or region in the browser harness. Notes
What this change shipsInventory (7 items) — 6 justifiedIntent: let the agent capture a whole page, one element or a zoomed region, at a size it picks, in one call. This is an ADDITION.
[FIRST-PRINCIPLES-REVIEWED] abe1b52 Description read[DESCRIPTION-READ] a26634203b381987080152f5f78ac911324121b3cf24b95b069bfec7a56f5f00 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Code change approver recommendation: A person should decideWhy: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 528 lines, more than the 400 it may approve on its own. This PR makes An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
3db2c7d to
df40bdc
Compare
|
Rebased: |
|
The GPT finding is right, and |
bolichen97
left a comment
There was a problem hiding this comment.
Thanks. I read the whole diff on this head. The feature is useful, and most of it is good to keep:
- Nothing new leaves the host.
screenshot_clipis one CDPPage.captureScreenshotin the local helper worker, and the PNG goes to the same staged path under the snapshots folder as today. - The secret-field and filled-field guards run before and after every form.
test_every_screenshot_form_keeps_the_secret_field_guardand the discard test cover that. - A screenshot with no arguments makes the same helper call as main (
set(kw) == {"path"}). - The size is bounded:
max_dimis 64 to 4096, magnify is at most 4x, and the clip scale is fitted tomax_dimin device pixels. - The helper and supervisor tables stay equal, and there is no new regex in product code.
I am requesting changes on one part: the new refusals in _shot_options and _shot_clip.
On main, _screenshot ignores its arguments, so every screenshot call with any args succeeds and saves the visible page. On this head, the same call now fails with bad_args or bad_ref when an option is malformed or two options conflict. That covers full: "yes", max_dim: 10 or 99999, ref: "banana", a bad region, and full with ref (test_screenshot_refuses_bad_options_before_touching_the_page). A valid ref whose element is gone, or has no size, now fails with ref_stale. This repo does not take a new condition that makes a call fail where main passes, even for argument shape.
Please keep main's outcome for those cases. Take the screenshot the way main does (the visible page), and add a warning to the result that names the option that was ignored and why. The accepted forms (full, max_dim, ref, region) stay exactly as they are. The checks in _region_box can stay, as long as a failure there falls back with a warning instead of refusing.
I will approve once a malformed or conflicting option no longer turns a call main answers into a refusal.
8b004a6 to
32958b4
Compare
|
Done in
The refusal test became |
|
Rebased: onto main |
32958b4 to
fce936a
Compare
fce936a to
90f7db1
Compare
|
The approver's point is right, and |
The browser tool's argument gate passes only scalars and lists of them, so an object-valued region was refused before it reached the op. The region is now a list of four numbers, which the gate already admits; the gate is unchanged. A test drives a region screenshot through the tool into the op. An option that is not valid, or that conflicts with another, no longer refuses the call. It is left out, the screenshot is taken as without it (the visible page by default), and the result carries a warning that names it, which the tool shows the model. A ref whose element is gone or has no size falls back the same way.
90f7db1 to
abe1b52
Compare
bolichen97
left a comment
There was a problem hiding this comment.
Re-reviewed the full diff at abe1b52. The change I asked for is in place, so I'm approving.
_shot_optionsno longer refuses. A malformedfull,max_dim,reforregion, or two offull/ref/regiontogether, is left out and named in the result'swarning. The screenshot is still saved, and the parametrizedtest_a_bad_or_conflicting_option_takes_the_visible_page_and_warnspins that with the helper called on{"path"}only._shot_clipfalls back to the visible page with a warning when arefis stale or has no size, soref_staleno longer fails the call.- A call with no options still makes the same helper call as main.
- The secret-field and filled-field guards, the URL check before the box capture, and the after-capture discard cover every form, and tests cover each one.
screenshot_clipstays in the local helper worker with the same staged path. The helper and supervisor tables match, and there's no new regex in product code.
The cancelled Issue Gate, GPT 6.1 Review, PR Policy, PR Hygiene and Sensitive Change Review runs were superseded by passing runs on this head. Auto-merge is left as you set it.
Problem / Motivation
Goal: The agent's
screenshotop can capture a whole page, one element, or a magnified region, and can cap the image size, so it reads small text and long pages in one call at a size it chooses.Today
screenshottakes no arguments: it always saves the visible viewport at the browser's full resolution. The helper behind it already acceptsfullandmax_dim(browser_harness/helper_worker.py), and the decision hooks already passmax_dimto keep their own frames small (browser_harness/decide.py), but the op never forwards either. A page longer than the viewport takes a scroll-and-shoot loop, one call per screen. A single control or a line of small text arrives as a whole viewport, so the agent cannot read it at a useful size and pays for pixels it does not need.Why it matters
Each extra screenshot is an image the model re-reads on every later turn. A long page costs several screens where one capture would do, and a 60 by 20 pixel label read from a full viewport is hard to make out. A size cap lets the agent keep images small when it only needs layout, and an element or region capture shows the part that matters at a readable size.
Not a goal
What changed (motivation → approach → change)
Symptom → cause: the op ignores its arguments, and the helper has no way to photograph one box.
browser_harness/ops.py:screenshotreads four optional arguments.full(whole scrollable page) andmax_dim(longest side, 64 to 4096 pixels) go to the existing helper.ref(one element, measured the wayclickmeasures it) andregion([x, y, width, height], the pixelsclickuses; a list because thebrowsertool passes only scalars and lists of them) go to a new box helper. At most one offull,refandregionpicks what is photographed. An option that is not valid, or that conflicts with another, is left out and named in awarningon the result, and the screenshot is taken as without it (the visible page by default); arefwhose element is gone or has no size falls back the same way. A region is magnified up to 4 times, fitted tomax_dim(default 1568 for a region, 4096 for an element).browser_harness/helper_worker.pyandsupervisor.py: a newscreenshot_cliphelper writes one box of the page withPage.captureScreenshot, adds the page scroll offset, and scales so the longest device-pixel side stays withinmax_dim.run_opchecks the current URL right beforescreenshot_clipruns, as it does beforescreenshotand every other page helper.browser_harness/snapshot_walker.js:resolvealso answers the element's width and height.mcp_tools/browser.py: thebrowsertool adds the op'swarning, when there is one, to the text the model reads.mcp_tools/catalog/browser.py,docs/browser-control.md,docs/system-specs/modules/browser.md: the arguments are described.Backwards compatibility
All four arguments are optional. With none, the helper is called with the path only, exactly as before (a test pins the keyword set), and the result is the same
{ok, path}. No option makes a call fail where main succeeds: a bad or conflicting option, or arefthat cannot be used, still saves the visible page, with awarningnaming what was left out. The walker'sresolvegains two keys; its callers readx,yandtagand ignore the rest. No setting is added because nothing changes unless the agent asks.Tests
bd08975e0a, 29 of the new tests fail: the op ignoresfull,max_dim,refandregionand names no option it left out, the tool shows no warning, a region screenshot sent through thebrowsertool is not captured, and the box helper and the walker's size do not exist.test/test_browser*.pyfiles andtest/test_mcp_browser_tool.py: 1518 passed, 1 skipped, 1 failed (test_browser_decisions.py::test_the_image_header_is_sent_only_with_an_image_or_a_box,ModuleNotFoundError: httpxin this environment; it does not touch this change).flake8,isort,mypypass on the changed files; the black, comment-history, sync-io and flake-pattern gates pass.python -m pytest test/test_browser_harness_ops.py test/test_browser_harness_helper_worker_cov.py -k "screenshot" -n0Manual verification
The new helper was run against a headless Chromium at device scale 2 on a page scrolled 100 pixels, with a 200 by 80 element and 9px text:
max_dim200: 200 by 80 pixels;max_dim300: 300 by 100 pixels.Sibling sweep
Sibling:
src/kiro_crew/browser_harness/decide.py-- not affected -- it calls the helper directly withmax_dimand is unchangedSibling:
src/kiro_crew/computer_use/-- not affected -- native-app screenshots use a different pathSibling:
src/kiro_crew/browser_harness/screencast.py-- not affected -- the live view stream does not use the opPattern harvest
Rule candidate: review-prompt
Pattern: an op's wrapper drops arguments its helper already supports, so a capability exists but the agent cannot reach it.
Related Issues
Complements #36166 (return the image in the tool result), which does not overlap: this change decides what is captured, that one decides how the result is delivered. #35738 edits the same files for a different fix.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement