Repository navigation
Conversation
marks: true outlines each enabled, actionable element on the screenshot and labels it with its element_index, the index the tree prints and the element actions take. zoom {x, y, width, height} returns that part of the window at capture resolution; coordinate actions map pixels of the zoomed image back to the window. Marks are drawn after resize and before encode, so the image config, size and coordinate mapping are unchanged. Linux and Windows reject both options; the JS REPL forwards them.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
get_app_stategains two optional macOS parameters:marks: trueoutlines each enabled, actionable element (press/pick/confirm/open/increment actions, text fields, rows) on the screenshot and labels it with itselement_index, the same index the tree prints andclick,set_valueand the other element actions take. No new index space; the tree text is unchanged apart from one note.zoom: {x, y, width, height}(fractions of the window) returns only that box of the window, cropped from the capture at full resolution. Click and dragx/ythen refer to pixels of the zoomed image; the result notes the region in window points.Marks are drawn after the resize and before encoding, so the image format,
maxLongEdgePixelsand the other image settings apply as before, labels stay readable at any size, and the delivered size and coordinate mapping are unchanged.marksrequiresinclude_axandinclude_screenshot;zoomrequiresinclude_screenshot; invalid boxes are rejected before the app is resolved. The JS REPL forwards both; Linux and Windows return "marks/zoom is not supported".Why
A vision model can read a number off the screenshot and act on it with the existing element actions, without matching the picture to the tree. Zoom gives a closer look at small or dense parts of a window without a larger full screenshot.
Verification
swift test: 390 tests, 7 skipped, 0 failures (main: 382/7/0). New tests cover which elements get marks, label placement, drawing orientation, unchanged delivered size, zoom and coordinate mapping, validation, dispatcher parsing and schema.go testpasses; the Linux rejection test passes.make check-docspasses.get_app_stateover 15 rounds: 61.0 ms plain, 62.2 ms marked. Zooming the top-right quarter gave a 900×632 native-resolution image, and a coordinate click on it pressed the intended button.Known limits