Skip to content

feat(macos): add set-of-marks and zoom to get_app_state - #122

Open
hyprcat wants to merge 1 commit into
iFurySt:mainfrom
hyprcat:feat/set-of-marks-zoom
Open

hyprcat wants to merge 1 commit into
iFurySt:mainfrom
hyprcat:feat/set-of-marks-zoom

Conversation

@hyprcat

@hyprcat hyprcat commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

What

get_app_state gains two optional macOS parameters:

  • marks: true outlines each enabled, actionable element (press/pick/confirm/open/increment actions, text fields, rows) on the screenshot and labels it with its element_index, the same index the tree prints and click, set_value and the other element actions take. No new index space; the tree text is unchanged apart from one note.
  • zoom: {x, y, width, height} (fractions of the window) returns only that box of the window, cropped from the capture at full resolution. Click and drag x/y then refer to pixels of the zoomed image; the result notes the region in window points.

Marks are drawn after the resize and before encoding, so the image format, maxLongEdgePixels and the other image settings apply as before, labels stay readable at any size, and the delivered size and coordinate mapping are unchanged. marks requires include_ax and include_screenshot; zoom requires include_screenshot; invalid boxes are rejected before the app is resolved. The JS REPL forwards both; Linux and Windows return "marks/zoom is not supported".

Why

A vision model can read a number off the screenshot and act on it with the existing element actions, without matching the picture to the tree. Zoom gives a closer look at small or dense parts of a window without a larger full screenshot.

Verification

  • swift test: 390 tests, 7 skipped, 0 failures (main: 382/7/0). New tests cover which elements get marks, label placement, drawing orientation, unchanged delivered size, zoom and coordinate mapping, validation, dispatcher parsing and schema.
  • Node REPL and script tests: 50 passed. Windows go test passes; the Linux rejection test passes. make check-docs passes.
  • Live, on a test AppKit app that never activates (window behind all others), default PNG settings: marked and plain images are both 1280×898 and the tree text is identical apart from the note; the disabled button gets no mark. Median get_app_state over 15 rounds: 61.0 ms plain, 62.2 ms marked. Zooming the top-right quarter gave a 900×632 native-resolution image, and a coordinate click on it pressed the intended button.
  • Size: PNG 165 → 179 KB; JPEG at an 800 px cap 43 → 94 KB (the 1 px outlines are expensive for JPEG).

Known limits

  • A table row partly hidden below its scroll view's edge still gets a mark over blank space (marks are not clipped to the enclosing scroll area).

marks: true outlines each enabled, actionable element on the screenshot and labels it with its element_index, the index the tree prints and the element actions take. zoom {x, y, width, height} returns that part of the window at capture resolution; coordinate actions map pixels of the zoomed image back to the window. Marks are drawn after resize and before encode, so the image config, size and coordinate mapping are unchanged. Linux and Windows reject both options; the JS REPL forwards them.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant