Skip to content

Describe controls scrolled out of a scroll box as off-screen - #1352

Open
giordano-lucas wants to merge 3 commits into
mainfrom
fix-observe-inner-scroller-offscreen
Open

giordano-lucas wants to merge 3 commits into
mainfrom
fix-observe-inner-scroller-offscreen

Conversation

@giordano-lucas

@giordano-lucas giordano-lucas commented Oct 26, 2026 •

Copy link
Copy Markdown
Member

What

observe() describes the controls above and below the viewport without an id, so a reader knows they exist and can scroll to them. That only looked at the page viewport. A control scrolled out of a small scroll box that is itself in view was neither in view nor below the fold, so it vanished. On uitestingplayground.com/overlapped, "Subject" sits below the visible part of a 100 px overflow-y: scroll box and was missing from the observation.

Controls whose center is above or below the box of an ancestor that scrolls vertically (and that they don't escape by positioning) are now described like the controls below the fold: no id, offered once scrolled into view. Only vertical scrolling counts, as for the page. Boxes that clip without scrolling keep hiding what they clip, as before.

Text rows scrolled out of a box (not controls) are unchanged.

Example

<div style="height: 100px; width: 300px; overflow-y: scroll">
  <input placeholder="Id" style="display: block">
  <div style="height: 150px"></div>
  <input placeholder="Subject">
</div>

Before (main), the Subject field below the box's visible area is missing:

I1[:]<input placeholder="Id"></input>

After, it is described without an id, like a control below the fold:

I1[:]<input placeholder="Id"></input>
_[:]<input placeholder="Subject"></input>

Tests

  • tests/browser/test_dom_offscreen_controls.py: 9 passed on Python 3.15.1 with locked Patchright 1.64.1. Covers scrolling shadow hosts, controls behind both borders of plain and scaled scroll boxes, and becoming actionable after scrolling.
  • Pre-commit passed for the changed files, including its pinned type checker.
  • No golden or characterization snapshots changed. GitHub CI is validating the rebased head.

Bench

The bench ran before the last rebase, on a local build of feat-session-act with the coverage tasks and all the observe fixes of this series merged (one build; each task below exercises one fix). Base is the same build without the fixes. Jev, 2 runs each, base and head back to back, concurrency 2, load average 28–40 on a shared machine.

task / probe base head
uitap-overlapped 0/2 2/2, with the covered-field fix; this PR adds Subject as an off-screen field

🤖 Generated with Claude Code


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@coderabbitai

coderabbitai Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 3 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 8 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration
Commits

Reviewing files that changed from the base of the PR and between b122d73 and 89687cf.

Files selected for processing: 3
  • AGENTS.md
  • packages/notte-browser/src/notte_browser/js/buildDomNode.js
  • tests/browser/test_dom_offscreen_controls.py

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@greptile-apps

greptile-apps Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The reviewed changes appear safe to merge.

Summary

Controls scrolled above or below a scroll box now remain in observations without action IDs.

  • Scroll-box controls appear in the page description before they are visible.

All three earlier findings are addressed in the current code. No new actionable issues were found.

Reviews (3) · Last reviewed commit: "fix(observe): preserve actions in escapi..." · Reviewed by Greptile

Comment thread packages/notte-browser/src/notte_browser/js/buildDomNode.js
Comment thread packages/notte-browser/src/notte_browser/js/buildDomNode.js Outdated
@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Coverage

Warning

Your comment is too long (maximum is 65536 characters), so the coverage report was not added. See the job log for how to reduce it.

Tests Skipped Failures Errors Time
3430 6 💤 0 ❌ 0 🔥 5m 19s ⏱️

giordano-lucas and others added 2 commits October 26, 2026 18:39
Off-screen controls were found against the page viewport only. A field scrolled out of a
small scroll box in view was neither in view nor below the fold, so it vanished from the
observation. Controls above or below the box of a scrollable ancestor are now described
like the controls below the fold, without an id until scrolled to.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@giordano-lucas
giordano-lucas force-pushed the fix-observe-inner-scroller-offscreen branch from 6af70d9 to dbafbcf Compare October 26, 2026 17:50
@giordano-lucas

Copy link
Copy Markdown
Member Author

@greptile review

Comment thread packages/notte-browser/src/notte_browser/js/buildDomNode.js
@giordano-lucas

Copy link
Copy Markdown
Member Author

@greptile review

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant