diff --git a/.agents/skills/uloop-screenshot/references/annotated-elements.md b/.agents/skills/uloop-screenshot/references/annotated-elements.md index 688ffe58a5..1ec7d61539 100644 --- a/.agents/skills/uloop-screenshot/references/annotated-elements.md +++ b/.agents/skills/uloop-screenshot/references/annotated-elements.md @@ -1,10 +1,10 @@ # Annotated Elements and Coordinates -Read this when using `uloop screenshot --capture-mode rendering --annotate-elements true` or clustered `--annotate-raycast-grid true --raycast-layer-mask ` output to find coordinates for `simulate-mouse-ui`, `simulate-mouse-input`, or `raycast`. +Read this when using `uloop screenshot --capture-mode rendering --annotate-elements true` or `--annotate-raycast-grid true` output to find coordinates for `simulate-mouse-ui`, `simulate-mouse-input`, or `raycast`. ## AnnotatedElements Fields -`AnnotatedElements` is empty unless `--annotate-elements true` is used, or unless `--annotate-raycast-grid true --raycast-layer-mask ` adds clustered 3D collider candidates. UI entries are sorted by z-order, frontmost first. Each item contains: +`AnnotatedElements` is empty unless `--annotate-elements true` is used, or unless `--annotate-raycast-grid true` adds clustered 3D collider candidates (with or without `--raycast-layer-mask`). UI entries are sorted by z-order, frontmost first. Each item contains: - `Label`: Index label in JSON (`A` = frontmost, `B` = next, ...). Screenshot labels also include the interaction hint, such as `A / CLICK` or `B / DRAG`. - `Name`: Element name @@ -18,21 +18,9 @@ Read this when using `uloop screenshot --capture-mode rendering --annotate-eleme - `SortingOrder`: Canvas sorting order. Higher values are in front. - `SiblingIndex`: Transform sibling index under the element's direct parent. Do not use it as a reliable z-order signal across nested UI hierarchies. -## RaycastGridPoints Fields - -`RaycastGridPoints` is populated when `--annotate-raycast-grid true` is used without `--raycast-layer-mask`. It is a coarse 5x5 grid preview of what each sample point would hit. - -- `Label`: Grid point label -- `Hit`: Whether the raycast hit a collider -- `InputX`, `InputY`: Top-left Game View coordinates for this sample point. Pass directly to `simulate-mouse-input --x/--y` or `raycast --x/--y`. -- `InjectedUnityPositionX`, `InjectedUnityPositionY`: The same point converted to bottom-left Unity coordinates (what `Mouse.current.position` would read) -- `HitGameObjectName`, `HitGameObjectPath`: Identifies the hit object when `Hit` is true, null otherwise -- `HitLayer`, `HitLayerIndex`: Physics layer of the hit object, null otherwise -- `Distance`: Distance from `Camera.main` to the hit point, null otherwise - ## RaycastLayerSummaries Fields -`RaycastLayerSummaries` is populated when `--annotate-raycast-grid true` is used without `--raycast-layer-mask`. It is built from dense raycast samples, while `RaycastGridPoints` remains the coarse 5x5 annotated grid. +`RaycastLayerSummaries` is always populated when `--annotate-raycast-grid true` is used, regardless of `--raycast-layer-mask`. It is built from a dense 40x40 raycast sample pass over `Physics.DefaultRaycastLayers` (fixed, independent of `--raycast-layer-mask`), so it always tells you what else is hittable across every default-visible layer, even when you narrowed `AnnotatedElements` down to one layer with `--raycast-layer-mask`. - `Layer`: Physics layer name to pass to `--raycast-layer-mask` - `LayerIndex`: Unity physics layer index @@ -41,6 +29,12 @@ Read this when using `uloop screenshot --capture-mode rendering --annotate-eleme Entries are sorted by `HitCount` descending, then `LayerIndex` ascending. +## RaycastLayerNamesChecked Fields + +`RaycastLayerNamesChecked` is populated when `--annotate-raycast-grid true` is used. It lists the physics layer names that were actually eligible to produce `AnnotatedElements` `PhysicsCollider` entries in this response: the clustering mask (`--raycast-layer-mask` if set, otherwise `Physics.DefaultRaycastLayers`) intersected with `Camera.main.cullingMask`. Use it to diagnose why an expected layer produced no `PhysicsCollider` entries — if the layer name is missing here, the active camera cannot see it this frame regardless of `--raycast-layer-mask`. + +This is a different mask than `RaycastLayerSummaries`, which always reports against the fixed `Physics.DefaultRaycastLayers` set. `RaycastLayerNamesChecked` tracks what was actually clustered; `RaycastLayerSummaries` is a constant discovery aid for "what else could I filter to next." + ## Coordinate Conversion When `ImageCoordinateSystem` is `"top-left-game-view"`, convert raw image pixel coordinates from `screenshot --capture-mode rendering` with the formula returned in `ScreenshotToInputFormula`: @@ -50,7 +44,7 @@ simulate_mouse_x = image_x / resolutionScale simulate_mouse_y = image_y / resolutionScale + imageToInputOffsetY ``` -When `ResolutionScale` is `1.0` and `imageToInputOffsetY` is `0` for rendering captures, raw image pixel coordinates already match mouse-input coordinates. `AnnotatedElements[].SimX/SimY` and `RaycastGridPoints[].InputX/InputY` are already mouse-input coordinates in that mode, so pass those values directly. +When `ResolutionScale` is `1.0` and `imageToInputOffsetY` is `0` for rendering captures, raw image pixel coordinates already match mouse-input coordinates. `AnnotatedElements[].SimX/SimY` is already a mouse-input coordinate in that mode, so pass it directly. For `PhysicsCollider` entries, `SimX/SimY` is a real sampled raycast hit nearest to the reachable cluster centroid. This avoids synthetic center points that may fall into empty space for L-shaped or ring-shaped collider coverage. Always use `SimX/SimY` for clicking; use `BoundsMinX/Y` and `BoundsMaxX/Y` only as a sampled coverage guide. diff --git a/.agents/skills/uloop-simulate-mouse-input/SKILL.md b/.agents/skills/uloop-simulate-mouse-input/SKILL.md index 887c18b908..4e704499fc 100644 --- a/.agents/skills/uloop-simulate-mouse-input/SKILL.md +++ b/.agents/skills/uloop-simulate-mouse-input/SKILL.md @@ -29,8 +29,8 @@ uloop simulate-mouse-input --action [options] | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `--action` | enum | `Click` | `Click`, `LongPress`, `MoveDelta`, `SmoothDelta`, `Scroll` | -| `--x` | number | `0` | Target X position in Game View pixels (origin: top-left). Used by Click and LongPress. Use `AnnotatedElements[].SimX`, `RaycastGridPoints[].InputX`, or raw image pixels converted with `ScreenshotToInputFormula`. | -| `--y` | number | `0` | Target Y position in Game View pixels (origin: top-left). Used by Click and LongPress. Use `AnnotatedElements[].SimY`, `RaycastGridPoints[].InputY`, or raw image pixels converted with `ScreenshotToInputFormula`. | +| `--x` | number | `0` | Target X position in Game View pixels (origin: top-left). Used by Click and LongPress. Use `AnnotatedElements[].SimX`, or raw image pixels converted with `ScreenshotToInputFormula`. | +| `--y` | number | `0` | Target Y position in Game View pixels (origin: top-left). Used by Click and LongPress. Use `AnnotatedElements[].SimY`, or raw image pixels converted with `ScreenshotToInputFormula`. | | `--button` | enum | `Left` | Mouse button: `Left`, `Right`, `Middle`. Used by Click and LongPress. | | `--duration` | number | `0` | Hold duration for LongPress, or interpolation duration for SmoothDelta (seconds). For Click, 0 = one-shot tap. | | `--delta-x` | number | `0` | Delta X in pixels for MoveDelta/SmoothDelta. Positive = right. | @@ -103,7 +103,7 @@ uloop simulate-mouse-input --action SmoothDelta --delta-x 300 --delta-y 0 --dura - `--x` / `--y` use **top-left Game View coordinates**. - Raw image pixels from `uloop screenshot --capture-mode rendering` must be converted with `ScreenshotToInputFormula`. -- `AnnotatedElements[].SimX/SimY` and `RaycastGridPoints[].InputX/InputY` can be passed directly to this tool. +- `AnnotatedElements[].SimX/SimY` can be passed directly to this tool. - Do not flip Y in the caller. The tool converts internally for Unity Input System: ```text diff --git a/.claude/skills/uloop-screenshot/references/annotated-elements.md b/.claude/skills/uloop-screenshot/references/annotated-elements.md index 688ffe58a5..1ec7d61539 100644 --- a/.claude/skills/uloop-screenshot/references/annotated-elements.md +++ b/.claude/skills/uloop-screenshot/references/annotated-elements.md @@ -1,10 +1,10 @@ # Annotated Elements and Coordinates -Read this when using `uloop screenshot --capture-mode rendering --annotate-elements true` or clustered `--annotate-raycast-grid true --raycast-layer-mask ` output to find coordinates for `simulate-mouse-ui`, `simulate-mouse-input`, or `raycast`. +Read this when using `uloop screenshot --capture-mode rendering --annotate-elements true` or `--annotate-raycast-grid true` output to find coordinates for `simulate-mouse-ui`, `simulate-mouse-input`, or `raycast`. ## AnnotatedElements Fields -`AnnotatedElements` is empty unless `--annotate-elements true` is used, or unless `--annotate-raycast-grid true --raycast-layer-mask ` adds clustered 3D collider candidates. UI entries are sorted by z-order, frontmost first. Each item contains: +`AnnotatedElements` is empty unless `--annotate-elements true` is used, or unless `--annotate-raycast-grid true` adds clustered 3D collider candidates (with or without `--raycast-layer-mask`). UI entries are sorted by z-order, frontmost first. Each item contains: - `Label`: Index label in JSON (`A` = frontmost, `B` = next, ...). Screenshot labels also include the interaction hint, such as `A / CLICK` or `B / DRAG`. - `Name`: Element name @@ -18,21 +18,9 @@ Read this when using `uloop screenshot --capture-mode rendering --annotate-eleme - `SortingOrder`: Canvas sorting order. Higher values are in front. - `SiblingIndex`: Transform sibling index under the element's direct parent. Do not use it as a reliable z-order signal across nested UI hierarchies. -## RaycastGridPoints Fields - -`RaycastGridPoints` is populated when `--annotate-raycast-grid true` is used without `--raycast-layer-mask`. It is a coarse 5x5 grid preview of what each sample point would hit. - -- `Label`: Grid point label -- `Hit`: Whether the raycast hit a collider -- `InputX`, `InputY`: Top-left Game View coordinates for this sample point. Pass directly to `simulate-mouse-input --x/--y` or `raycast --x/--y`. -- `InjectedUnityPositionX`, `InjectedUnityPositionY`: The same point converted to bottom-left Unity coordinates (what `Mouse.current.position` would read) -- `HitGameObjectName`, `HitGameObjectPath`: Identifies the hit object when `Hit` is true, null otherwise -- `HitLayer`, `HitLayerIndex`: Physics layer of the hit object, null otherwise -- `Distance`: Distance from `Camera.main` to the hit point, null otherwise - ## RaycastLayerSummaries Fields -`RaycastLayerSummaries` is populated when `--annotate-raycast-grid true` is used without `--raycast-layer-mask`. It is built from dense raycast samples, while `RaycastGridPoints` remains the coarse 5x5 annotated grid. +`RaycastLayerSummaries` is always populated when `--annotate-raycast-grid true` is used, regardless of `--raycast-layer-mask`. It is built from a dense 40x40 raycast sample pass over `Physics.DefaultRaycastLayers` (fixed, independent of `--raycast-layer-mask`), so it always tells you what else is hittable across every default-visible layer, even when you narrowed `AnnotatedElements` down to one layer with `--raycast-layer-mask`. - `Layer`: Physics layer name to pass to `--raycast-layer-mask` - `LayerIndex`: Unity physics layer index @@ -41,6 +29,12 @@ Read this when using `uloop screenshot --capture-mode rendering --annotate-eleme Entries are sorted by `HitCount` descending, then `LayerIndex` ascending. +## RaycastLayerNamesChecked Fields + +`RaycastLayerNamesChecked` is populated when `--annotate-raycast-grid true` is used. It lists the physics layer names that were actually eligible to produce `AnnotatedElements` `PhysicsCollider` entries in this response: the clustering mask (`--raycast-layer-mask` if set, otherwise `Physics.DefaultRaycastLayers`) intersected with `Camera.main.cullingMask`. Use it to diagnose why an expected layer produced no `PhysicsCollider` entries — if the layer name is missing here, the active camera cannot see it this frame regardless of `--raycast-layer-mask`. + +This is a different mask than `RaycastLayerSummaries`, which always reports against the fixed `Physics.DefaultRaycastLayers` set. `RaycastLayerNamesChecked` tracks what was actually clustered; `RaycastLayerSummaries` is a constant discovery aid for "what else could I filter to next." + ## Coordinate Conversion When `ImageCoordinateSystem` is `"top-left-game-view"`, convert raw image pixel coordinates from `screenshot --capture-mode rendering` with the formula returned in `ScreenshotToInputFormula`: @@ -50,7 +44,7 @@ simulate_mouse_x = image_x / resolutionScale simulate_mouse_y = image_y / resolutionScale + imageToInputOffsetY ``` -When `ResolutionScale` is `1.0` and `imageToInputOffsetY` is `0` for rendering captures, raw image pixel coordinates already match mouse-input coordinates. `AnnotatedElements[].SimX/SimY` and `RaycastGridPoints[].InputX/InputY` are already mouse-input coordinates in that mode, so pass those values directly. +When `ResolutionScale` is `1.0` and `imageToInputOffsetY` is `0` for rendering captures, raw image pixel coordinates already match mouse-input coordinates. `AnnotatedElements[].SimX/SimY` is already a mouse-input coordinate in that mode, so pass it directly. For `PhysicsCollider` entries, `SimX/SimY` is a real sampled raycast hit nearest to the reachable cluster centroid. This avoids synthetic center points that may fall into empty space for L-shaped or ring-shaped collider coverage. Always use `SimX/SimY` for clicking; use `BoundsMinX/Y` and `BoundsMaxX/Y` only as a sampled coverage guide. diff --git a/.claude/skills/uloop-simulate-mouse-input/SKILL.md b/.claude/skills/uloop-simulate-mouse-input/SKILL.md index 887c18b908..4e704499fc 100644 --- a/.claude/skills/uloop-simulate-mouse-input/SKILL.md +++ b/.claude/skills/uloop-simulate-mouse-input/SKILL.md @@ -29,8 +29,8 @@ uloop simulate-mouse-input --action [options] | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `--action` | enum | `Click` | `Click`, `LongPress`, `MoveDelta`, `SmoothDelta`, `Scroll` | -| `--x` | number | `0` | Target X position in Game View pixels (origin: top-left). Used by Click and LongPress. Use `AnnotatedElements[].SimX`, `RaycastGridPoints[].InputX`, or raw image pixels converted with `ScreenshotToInputFormula`. | -| `--y` | number | `0` | Target Y position in Game View pixels (origin: top-left). Used by Click and LongPress. Use `AnnotatedElements[].SimY`, `RaycastGridPoints[].InputY`, or raw image pixels converted with `ScreenshotToInputFormula`. | +| `--x` | number | `0` | Target X position in Game View pixels (origin: top-left). Used by Click and LongPress. Use `AnnotatedElements[].SimX`, or raw image pixels converted with `ScreenshotToInputFormula`. | +| `--y` | number | `0` | Target Y position in Game View pixels (origin: top-left). Used by Click and LongPress. Use `AnnotatedElements[].SimY`, or raw image pixels converted with `ScreenshotToInputFormula`. | | `--button` | enum | `Left` | Mouse button: `Left`, `Right`, `Middle`. Used by Click and LongPress. | | `--duration` | number | `0` | Hold duration for LongPress, or interpolation duration for SmoothDelta (seconds). For Click, 0 = one-shot tap. | | `--delta-x` | number | `0` | Delta X in pixels for MoveDelta/SmoothDelta. Positive = right. | @@ -103,7 +103,7 @@ uloop simulate-mouse-input --action SmoothDelta --delta-x 300 --delta-y 0 --dura - `--x` / `--y` use **top-left Game View coordinates**. - Raw image pixels from `uloop screenshot --capture-mode rendering` must be converted with `ScreenshotToInputFormula`. -- `AnnotatedElements[].SimX/SimY` and `RaycastGridPoints[].InputX/InputY` can be passed directly to this tool. +- `AnnotatedElements[].SimX/SimY` can be passed directly to this tool. - Do not flip Y in the caller. The tool converts internally for Unity Input System: ```text diff --git a/Assets/Tests/Editor/RaycastGridAnnotatorTests.cs b/Assets/Tests/Editor/RaycastGridAnnotatorTests.cs index b79876d212..ee716cc466 100644 --- a/Assets/Tests/Editor/RaycastGridAnnotatorTests.cs +++ b/Assets/Tests/Editor/RaycastGridAnnotatorTests.cs @@ -16,68 +16,31 @@ namespace io.github.hatayama.UnityCliLoop.Tests.Editor public class RaycastGridAnnotatorTests { [Test] - public void CalculateGridInputPosition_WhenRenderingHasTopOffset_ShouldSampleInsideCapturedImage() + public void CalculateGridInputPositionForGrid_WhenRenderingHasTopOffset_ShouldSampleInsideCapturedImage() { // Tests that a grid input position samples inside the captured image when rendering has a top offset. Vector2 renderingImageSize = new Vector2(1200f, 1080f); Vector2 inputPosition = - RaycastGridAnnotator.CalculateGridInputPosition(renderingImageSize, 303, 1, 3); + RaycastGridAnnotator.CalculateGridInputPositionForGrid(renderingImageSize, 303, 5, 5, 1, 3); Assert.That(inputPosition.x, Is.EqualTo(600f)); Assert.That(inputPosition.y, Is.EqualTo(483f)); } [Test] - public void CalculateGridInputPosition_WhenRenderingHasTopOffset_ShouldKeepBottomRowVisible() + public void CalculateGridInputPositionForGrid_WhenRenderingHasTopOffset_ShouldKeepBottomRowVisible() { // Tests that the bottom grid row stays visible within the rendering image when a top offset is applied. Vector2 renderingImageSize = new Vector2(1200f, 1080f); Vector2 inputPosition = - RaycastGridAnnotator.CalculateGridInputPosition(renderingImageSize, 303, 5, 3); + RaycastGridAnnotator.CalculateGridInputPositionForGrid(renderingImageSize, 303, 5, 5, 5, 3); Assert.That(inputPosition.x, Is.EqualTo(600f)); Assert.That(inputPosition.y, Is.EqualTo(1203f)); } - [Test] - public void CreateOverlayElements_WhenPointsIncludeMisses_ShouldAnnotateOnlyHits() - { - // Tests that overlay elements are created only for grid points that registered a physics hit. - List points = new List - { - new RaycastGridPointInfo - { - Label = "R1", - Hit = true, - InputX = 100f, - InputY = 200f, - InjectedUnityPositionX = 100f, - InjectedUnityPositionY = 880f, - HitGameObjectName = "Cube", - HitGameObjectPath = "Cube" - }, - new RaycastGridPointInfo - { - Label = "R2", - Hit = false, - InputX = 200f, - InputY = 300f, - InjectedUnityPositionX = 200f, - InjectedUnityPositionY = 780f - } - }; - - List overlayElements = RaycastGridAnnotator.CreateOverlayElements(points); - - Assert.That(overlayElements.Count, Is.EqualTo(1)); - Assert.That(overlayElements[0].Label, Is.EqualTo("R1")); - Assert.That(overlayElements[0].Name, Is.EqualTo("Cube")); - Assert.That(overlayElements[0].Type, Is.EqualTo("RaycastHit")); - Assert.That(overlayElements[0].Interaction, Is.EqualTo("Raycast")); - } - [Test] public void Resolve_WhenLayerNamesAreCommaSeparated_ShouldBuildMaskFromKnownLayers() { @@ -117,6 +80,54 @@ public void Resolve_WhenLayerNameDoesNotExist_ShouldReturnInvalidNamesAndValidNa Assert.That(resolution.ValidLayerNames, Is.EqualTo(new List { "Default", "Ground" })); } + [Test] + public void CreateLayerNamesFromMask_WhenMaskMatchesSpecificLayers_ShouldReturnOnlyThoseLayerNames() + { + // Tests that only the layer names whose bits are set in the mask are returned. + List availableLayers = new List + { + new RaycastLayerDefinition { Name = "Default", Index = 0 }, + new RaycastLayerDefinition { Name = "Ground", Index = 8 }, + new RaycastLayerDefinition { Name = "Clickable", Index = 9 } + }; + int mask = (1 << 8) | (1 << 9); + + List layerNames = RaycastLayerMaskResolver.CreateLayerNamesFromMask(mask, availableLayers); + + Assert.That(layerNames, Is.EqualTo(new List { "Ground", "Clickable" })); + } + + [Test] + public void CreateLayerNamesFromMask_WhenMaskIsDefaultRaycastLayers_ShouldExcludeIgnoreRaycastLayer() + { + // Tests that Physics.DefaultRaycastLayers excludes the built-in Ignore Raycast layer (index 2). + List availableLayers = new List + { + new RaycastLayerDefinition { Name = "Default", Index = 0 }, + new RaycastLayerDefinition { Name = "Ignore Raycast", Index = 2 }, + new RaycastLayerDefinition { Name = "Ground", Index = 8 } + }; + + List layerNames = RaycastLayerMaskResolver.CreateLayerNamesFromMask( + Physics.DefaultRaycastLayers, availableLayers); + + Assert.That(layerNames, Is.EqualTo(new List { "Default", "Ground" })); + } + + [Test] + public void CreateLayerNamesFromMask_WhenMaskHasNoMatchingLayers_ShouldReturnEmptyList() + { + // Tests that a mask with no bits overlapping the available layers returns an empty list. + List availableLayers = new List + { + new RaycastLayerDefinition { Name = "Ground", Index = 8 } + }; + + List layerNames = RaycastLayerMaskResolver.CreateLayerNamesFromMask(0, availableLayers); + + Assert.That(layerNames, Is.Empty); + } + [Test] public void CreateClusters_WhenSamplesHitSameCollider_ShouldReturnOneCluster() { @@ -566,7 +577,7 @@ public void IsUiOcclusionRaycastResult_WhenPhysicsRaycasterHit_ShouldReturnFalse public void CreateLayerSummaries_ShouldCountHitsByLayerAndSortByHitCount() { // Tests that layer summaries count hits per layer and sort by descending hit count. - List points = new List + List points = new List { CreateLayerHitPoint("Ground", 8, "Ground/A"), CreateLayerHitPoint("Ground", 8, "Ground/B"), @@ -586,7 +597,7 @@ public void CreateLayerSummaries_ShouldCountHitsByLayerAndSortByHitCount() public void CreateLayerSummaries_WhenLayerCountsTie_ShouldSortByLayerIndex() { // Tests that tied hit counts fall back to sorting by ascending layer index. - List points = new List + List points = new List { CreateLayerHitPoint("Clickable", 9, "Clickable/A"), CreateLayerHitPoint("Ground", 8, "Ground/A") @@ -602,7 +613,7 @@ public void CreateLayerSummaries_WhenLayerCountsTie_ShouldSortByLayerIndex() public void CreateLayerSummaries_ShouldUseMostFrequentObjectPathAsRepresentative() { // Tests that the representative object path is the most frequently hit path within the layer. - List points = new List + List points = new List { CreateLayerHitPoint("Ground", 8, "Ground/A"), CreateLayerHitPoint("Ground", 8, "Ground/A"), @@ -618,7 +629,7 @@ public void CreateLayerSummaries_ShouldUseMostFrequentObjectPathAsRepresentative public void CreateLayerSummaries_WhenObjectCountsTie_ShouldUseAlphabeticalPath() { // Tests that a tie in per-object hit counts is broken by picking the alphabetically first path. - List points = new List + List points = new List { CreateLayerHitPoint("Ground", 8, "Ground/B"), CreateLayerHitPoint("Ground", 8, "Ground/A") @@ -633,9 +644,9 @@ public void CreateLayerSummaries_WhenObjectCountsTie_ShouldUseAlphabeticalPath() public void CreateLayerSummaries_WhenNoHits_ShouldReturnEmptyList() { // Tests that no layer summaries are produced when none of the grid points registered a hit. - List points = new List + List points = new List { - new RaycastGridPointInfo + new RaycastLayerHitSample { Hit = false } @@ -697,12 +708,12 @@ private static void AssertSegment( Assert.That(segment.EndY, Is.EqualTo(endY)); } - private static RaycastGridPointInfo CreateLayerHitPoint( + private static RaycastLayerHitSample CreateLayerHitPoint( string layer, int layerIndex, string objectPath) { - return new RaycastGridPointInfo + return new RaycastLayerHitSample { Hit = true, HitLayer = layer, diff --git a/Packages/src/Editor/FirstPartyTools/Screenshot/RaycastAnnotation/RaycastGridAnnotator.cs b/Packages/src/Editor/FirstPartyTools/Screenshot/RaycastAnnotation/RaycastGridAnnotator.cs index 9c17da8025..a21c5f079f 100644 --- a/Packages/src/Editor/FirstPartyTools/Screenshot/RaycastAnnotation/RaycastGridAnnotator.cs +++ b/Packages/src/Editor/FirstPartyTools/Screenshot/RaycastAnnotation/RaycastGridAnnotator.cs @@ -13,43 +13,28 @@ namespace io.github.hatayama.UnityCliLoop.FirstPartyTools /// internal static class RaycastGridAnnotator { - private const int GRID_COLUMNS = 5; - private const int GRID_ROWS = 5; private const int CLUSTERED_GRID_COLUMNS = 40; private const int CLUSTERED_GRID_ROWS = 40; - private const float MARKER_SIZE = 18f; - - internal static List CollectRaycastGridPoints( - Vector2 renderingImageSize, - int imageToInputOffsetY) - { - return CollectRaycastGridPointsForGrid( - renderingImageSize, - imageToInputOffsetY, - GRID_ROWS, - GRID_COLUMNS); - } internal static List CollectRaycastLayerSummaries( Vector2 renderingImageSize, int imageToInputOffsetY) { - List points = CollectRaycastGridPointsForGrid( + List samples = CollectLayerHitSamples( renderingImageSize, imageToInputOffsetY, CLUSTERED_GRID_ROWS, CLUSTERED_GRID_COLUMNS); - return CreateLayerSummaries(points); + return CreateLayerSummaries(samples); } - private static List CollectRaycastGridPointsForGrid( + private static List CollectLayerHitSamples( Vector2 renderingImageSize, int imageToInputOffsetY, int rowCount, int columnCount) { - List points = new List(); - int labelIndex = 1; + List samples = new List(); // Sync once for the whole grid; each candidate raycast then reads the same current physics state. Physics.SyncTransforms(); @@ -70,27 +55,11 @@ private static List CollectRaycastGridPointsForGrid( Physics.DefaultRaycastLayers, false); - points.Add(CreatePointInfo($"R{labelIndex}", inputPosition, raycastResult)); - labelIndex++; + samples.Add(CreateLayerHitSample(raycastResult)); } } - return points; - } - - internal static Vector2 CalculateGridInputPosition( - Vector2 renderingImageSize, - int imageToInputOffsetY, - int row, - int column) - { - return CalculateGridInputPositionForGrid( - renderingImageSize, - imageToInputOffsetY, - GRID_ROWS, - GRID_COLUMNS, - row, - column); + return samples; } internal static Vector2 CalculateGridInputPositionForGrid( @@ -153,90 +122,47 @@ internal static List CollectPhysicsColliderElements( return elements; } - internal static List CreateOverlayElements(List points) - { - List overlayElements = new List(); - - foreach (RaycastGridPointInfo point in points) - { - if (!point.Hit) - { - continue; - } - - float halfSize = MARKER_SIZE / 2f; - overlayElements.Add(new UIElementInfo - { - Label = point.Label, - Name = point.HitGameObjectName ?? "", - Path = point.HitGameObjectPath ?? "", - Type = "RaycastHit", - Interaction = "Raycast", - SimX = point.InputX, - SimY = point.InputY, - BoundsMinX = point.InjectedUnityPositionX - halfSize, - BoundsMinY = point.InjectedUnityPositionY - halfSize, - BoundsMaxX = point.InjectedUnityPositionX + halfSize, - BoundsMaxY = point.InjectedUnityPositionY + halfSize, - SortingOrder = 0, - SiblingIndex = 0 - }); - } - - return overlayElements; - } - - private static RaycastGridPointInfo CreatePointInfo( - string label, - Vector2 inputPosition, - GameViewRaycastResult raycastResult) + private static RaycastLayerHitSample CreateLayerHitSample(GameViewRaycastResult raycastResult) { - RaycastGridPointInfo pointInfo = new RaycastGridPointInfo + RaycastLayerHitSample sample = new RaycastLayerHitSample { - Label = label, - Hit = raycastResult.Hits.Length > 0, - InputX = inputPosition.x, - InputY = inputPosition.y, - InjectedUnityPositionX = raycastResult.Conversion.InjectedUnityPosition.x, - InjectedUnityPositionY = raycastResult.Conversion.InjectedUnityPosition.y + Hit = raycastResult.Hits.Length > 0 }; - if (!pointInfo.Hit) + if (!sample.Hit) { - return pointInfo; + return sample; } RaycastHit hit = raycastResult.Hits[0]; - pointInfo.HitGameObjectName = hit.collider.gameObject.name; - pointInfo.HitGameObjectPath = GameObjectPathUtility.GetFullPath(hit.collider.gameObject); - pointInfo.HitLayerIndex = hit.collider.gameObject.layer; - pointInfo.HitLayer = LayerMask.LayerToName(hit.collider.gameObject.layer); - pointInfo.Distance = hit.distance; - return pointInfo; + sample.HitGameObjectPath = GameObjectPathUtility.GetFullPath(hit.collider.gameObject); + sample.HitLayerIndex = hit.collider.gameObject.layer; + sample.HitLayer = LayerMask.LayerToName(hit.collider.gameObject.layer); + return sample; } - internal static List CreateLayerSummaries(List points) + internal static List CreateLayerSummaries(List samples) { Dictionary accumulatorsByLayerIndex = new Dictionary(); - foreach (RaycastGridPointInfo point in points) + foreach (RaycastLayerHitSample sample in samples) { - if (!point.Hit || point.HitLayerIndex == null) + if (!sample.Hit || sample.HitLayerIndex == null) { continue; } - int layerIndex = point.HitLayerIndex.Value; + int layerIndex = sample.HitLayerIndex.Value; if (!accumulatorsByLayerIndex.ContainsKey(layerIndex)) { accumulatorsByLayerIndex.Add( layerIndex, - new RaycastLayerSummaryAccumulator(point.HitLayer ?? "", layerIndex)); + new RaycastLayerSummaryAccumulator(sample.HitLayer ?? "", layerIndex)); } RaycastLayerSummaryAccumulator accumulator = accumulatorsByLayerIndex[layerIndex]; - accumulator.AddHit(point.HitGameObjectPath ?? ""); + accumulator.AddHit(sample.HitGameObjectPath ?? ""); } List summaries = new List(); @@ -578,4 +504,15 @@ public RaycastSampleBounds(float minX, float minY, float maxX, float maxY) } } } + + /// + /// Carries the raycast hit layer/object of one dense grid sample, for internal layer-summary aggregation only. + /// + internal sealed class RaycastLayerHitSample + { + public bool Hit { get; set; } + public string? HitGameObjectPath { get; set; } + public string? HitLayer { get; set; } + public int? HitLayerIndex { get; set; } + } } diff --git a/Packages/src/Editor/FirstPartyTools/Screenshot/RaycastAnnotation/RaycastLayerMaskResolver.cs b/Packages/src/Editor/FirstPartyTools/Screenshot/RaycastAnnotation/RaycastLayerMaskResolver.cs index 9664eef577..ee422d97a7 100644 --- a/Packages/src/Editor/FirstPartyTools/Screenshot/RaycastAnnotation/RaycastLayerMaskResolver.cs +++ b/Packages/src/Editor/FirstPartyTools/Screenshot/RaycastAnnotation/RaycastLayerMaskResolver.cs @@ -129,6 +129,29 @@ private static List CreateValidLayerNames( return layerNames; } + internal static List CreateLayerNamesFromMask( + int mask, + IReadOnlyList availableLayers) + { + List layerNames = new List(); + foreach (RaycastLayerDefinition layer in availableLayers) + { + if (!IsValidLayer(layer)) + { + continue; + } + + if ((mask & (1 << layer.Index)) == 0) + { + continue; + } + + layerNames.Add(layer.Name); + } + + return layerNames; + } + private static bool IsValidLayer(RaycastLayerDefinition layer) { return !string.IsNullOrEmpty(layer.Name) && diff --git a/Packages/src/Editor/FirstPartyTools/Screenshot/ScreenshotUseCase.cs b/Packages/src/Editor/FirstPartyTools/Screenshot/ScreenshotUseCase.cs index 2c7bdff4d8..3317a49e65 100644 --- a/Packages/src/Editor/FirstPartyTools/Screenshot/ScreenshotUseCase.cs +++ b/Packages/src/Editor/FirstPartyTools/Screenshot/ScreenshotUseCase.cs @@ -63,9 +63,8 @@ private async Task CaptureRenderingAsync( List annotatedElements = new(); List physicsColliderElements = new(); Vector2 gameViewSize = GameViewCoordinateUtility.GetMainGameViewSize(); - List raycastGridPoints = new(); List raycastLayerSummaries = new(); - List raycastGridOverlayElements = new(); + List raycastLayerNamesChecked = new(); GameRenderingImageInfo? raycastGridRenderingInfo = null; if (request.AnnotateElements) @@ -94,24 +93,24 @@ private async Task CaptureRenderingAsync( raycastGridRenderingInfo = renderingImageInfo; gameViewSize = renderingImageInfo.GameViewSize; RaycastLayerMaskResolution raycastLayerMaskResolution = ResolveRaycastLayerMask(request); - - if (raycastLayerMaskResolution.HasLayerNames) - { - physicsColliderElements = RaycastGridAnnotator.CollectPhysicsColliderElements( - renderingImageInfo.RenderingImageSize, - renderingImageInfo.ImageToInputOffsetY, - raycastLayerMaskResolution.Mask); - } - else - { - raycastGridPoints = RaycastGridAnnotator.CollectRaycastGridPoints( - renderingImageInfo.RenderingImageSize, - renderingImageInfo.ImageToInputOffsetY); - raycastLayerSummaries = RaycastGridAnnotator.CollectRaycastLayerSummaries( - renderingImageInfo.RenderingImageSize, - renderingImageInfo.ImageToInputOffsetY); - raycastGridOverlayElements = RaycastGridAnnotator.CreateOverlayElements(raycastGridPoints); - } + List availableLayerDefinitions = GetAvailableLayerDefinitions(); + int effectiveLayerMask = raycastLayerMaskResolution.HasLayerNames + ? raycastLayerMaskResolution.Mask + : Physics.DefaultRaycastLayers; + + physicsColliderElements = RaycastGridAnnotator.CollectPhysicsColliderElements( + renderingImageInfo.RenderingImageSize, + renderingImageInfo.ImageToInputOffsetY, + effectiveLayerMask); + raycastLayerSummaries = RaycastGridAnnotator.CollectRaycastLayerSummaries( + renderingImageInfo.RenderingImageSize, + renderingImageInfo.ImageToInputOffsetY); + + Camera mainCamera = Camera.main; + int checkedLayerMask = mainCamera != null ? effectiveLayerMask & mainCamera.cullingMask : 0; + raycastLayerNamesChecked = RaycastLayerMaskResolver.CreateLayerNamesFromMask( + checkedLayerMask, + availableLayerDefinitions); } if (request.ElementsOnly) @@ -123,8 +122,8 @@ private async Task CaptureRenderingAsync( int elementsOnlyImageToInputOffsetY = raycastGridRenderingInfo?.ImageToInputOffsetY ?? 0; ApplyRenderingCoordinateMetadata(elementsOnlyInfo, gameViewSize, elementsOnlyImageToInputOffsetY); elementsOnlyInfo.AnnotatedElements = elementsOnlyAnnotatedElements; - elementsOnlyInfo.RaycastGridPoints = raycastGridPoints; elementsOnlyInfo.RaycastLayerSummaries = raycastLayerSummaries; + elementsOnlyInfo.RaycastLayerNamesChecked = raycastLayerNamesChecked; return new ScreenshotResponse { Screenshots = new List { elementsOnlyInfo } @@ -140,7 +139,6 @@ private async Task CaptureRenderingAsync( if (request.AnnotateElements || request.AnnotateRaycastGrid) { List overlayElements = new(annotatedElements); - overlayElements.AddRange(raycastGridOverlayElements); overlayElements.AddRange(physicsColliderElements); annotationOverlay = UIElementAnnotator.CreateAnnotationOverlay( overlayElements, @@ -221,8 +219,8 @@ private async Task CaptureRenderingAsync( }; ApplyRenderingCoordinateMetadata(info, captureRenderingInfo.GameViewSize, captureRenderingInfo.ImageToInputOffsetY); info.AnnotatedElements = responseAnnotatedElements; - info.RaycastGridPoints = raycastGridPoints; info.RaycastLayerSummaries = raycastLayerSummaries; + info.RaycastLayerNamesChecked = raycastLayerNamesChecked; screenshots.Add(info); } catch (Exception ex) diff --git a/Packages/src/Editor/FirstPartyTools/Screenshot/Skill/references/annotated-elements.md b/Packages/src/Editor/FirstPartyTools/Screenshot/Skill/references/annotated-elements.md index 688ffe58a5..1ec7d61539 100644 --- a/Packages/src/Editor/FirstPartyTools/Screenshot/Skill/references/annotated-elements.md +++ b/Packages/src/Editor/FirstPartyTools/Screenshot/Skill/references/annotated-elements.md @@ -1,10 +1,10 @@ # Annotated Elements and Coordinates -Read this when using `uloop screenshot --capture-mode rendering --annotate-elements true` or clustered `--annotate-raycast-grid true --raycast-layer-mask ` output to find coordinates for `simulate-mouse-ui`, `simulate-mouse-input`, or `raycast`. +Read this when using `uloop screenshot --capture-mode rendering --annotate-elements true` or `--annotate-raycast-grid true` output to find coordinates for `simulate-mouse-ui`, `simulate-mouse-input`, or `raycast`. ## AnnotatedElements Fields -`AnnotatedElements` is empty unless `--annotate-elements true` is used, or unless `--annotate-raycast-grid true --raycast-layer-mask ` adds clustered 3D collider candidates. UI entries are sorted by z-order, frontmost first. Each item contains: +`AnnotatedElements` is empty unless `--annotate-elements true` is used, or unless `--annotate-raycast-grid true` adds clustered 3D collider candidates (with or without `--raycast-layer-mask`). UI entries are sorted by z-order, frontmost first. Each item contains: - `Label`: Index label in JSON (`A` = frontmost, `B` = next, ...). Screenshot labels also include the interaction hint, such as `A / CLICK` or `B / DRAG`. - `Name`: Element name @@ -18,21 +18,9 @@ Read this when using `uloop screenshot --capture-mode rendering --annotate-eleme - `SortingOrder`: Canvas sorting order. Higher values are in front. - `SiblingIndex`: Transform sibling index under the element's direct parent. Do not use it as a reliable z-order signal across nested UI hierarchies. -## RaycastGridPoints Fields - -`RaycastGridPoints` is populated when `--annotate-raycast-grid true` is used without `--raycast-layer-mask`. It is a coarse 5x5 grid preview of what each sample point would hit. - -- `Label`: Grid point label -- `Hit`: Whether the raycast hit a collider -- `InputX`, `InputY`: Top-left Game View coordinates for this sample point. Pass directly to `simulate-mouse-input --x/--y` or `raycast --x/--y`. -- `InjectedUnityPositionX`, `InjectedUnityPositionY`: The same point converted to bottom-left Unity coordinates (what `Mouse.current.position` would read) -- `HitGameObjectName`, `HitGameObjectPath`: Identifies the hit object when `Hit` is true, null otherwise -- `HitLayer`, `HitLayerIndex`: Physics layer of the hit object, null otherwise -- `Distance`: Distance from `Camera.main` to the hit point, null otherwise - ## RaycastLayerSummaries Fields -`RaycastLayerSummaries` is populated when `--annotate-raycast-grid true` is used without `--raycast-layer-mask`. It is built from dense raycast samples, while `RaycastGridPoints` remains the coarse 5x5 annotated grid. +`RaycastLayerSummaries` is always populated when `--annotate-raycast-grid true` is used, regardless of `--raycast-layer-mask`. It is built from a dense 40x40 raycast sample pass over `Physics.DefaultRaycastLayers` (fixed, independent of `--raycast-layer-mask`), so it always tells you what else is hittable across every default-visible layer, even when you narrowed `AnnotatedElements` down to one layer with `--raycast-layer-mask`. - `Layer`: Physics layer name to pass to `--raycast-layer-mask` - `LayerIndex`: Unity physics layer index @@ -41,6 +29,12 @@ Read this when using `uloop screenshot --capture-mode rendering --annotate-eleme Entries are sorted by `HitCount` descending, then `LayerIndex` ascending. +## RaycastLayerNamesChecked Fields + +`RaycastLayerNamesChecked` is populated when `--annotate-raycast-grid true` is used. It lists the physics layer names that were actually eligible to produce `AnnotatedElements` `PhysicsCollider` entries in this response: the clustering mask (`--raycast-layer-mask` if set, otherwise `Physics.DefaultRaycastLayers`) intersected with `Camera.main.cullingMask`. Use it to diagnose why an expected layer produced no `PhysicsCollider` entries — if the layer name is missing here, the active camera cannot see it this frame regardless of `--raycast-layer-mask`. + +This is a different mask than `RaycastLayerSummaries`, which always reports against the fixed `Physics.DefaultRaycastLayers` set. `RaycastLayerNamesChecked` tracks what was actually clustered; `RaycastLayerSummaries` is a constant discovery aid for "what else could I filter to next." + ## Coordinate Conversion When `ImageCoordinateSystem` is `"top-left-game-view"`, convert raw image pixel coordinates from `screenshot --capture-mode rendering` with the formula returned in `ScreenshotToInputFormula`: @@ -50,7 +44,7 @@ simulate_mouse_x = image_x / resolutionScale simulate_mouse_y = image_y / resolutionScale + imageToInputOffsetY ``` -When `ResolutionScale` is `1.0` and `imageToInputOffsetY` is `0` for rendering captures, raw image pixel coordinates already match mouse-input coordinates. `AnnotatedElements[].SimX/SimY` and `RaycastGridPoints[].InputX/InputY` are already mouse-input coordinates in that mode, so pass those values directly. +When `ResolutionScale` is `1.0` and `imageToInputOffsetY` is `0` for rendering captures, raw image pixel coordinates already match mouse-input coordinates. `AnnotatedElements[].SimX/SimY` is already a mouse-input coordinate in that mode, so pass it directly. For `PhysicsCollider` entries, `SimX/SimY` is a real sampled raycast hit nearest to the reachable cluster centroid. This avoids synthetic center points that may fall into empty space for L-shaped or ring-shaped collider coverage. Always use `SimX/SimY` for clicking; use `BoundsMinX/Y` and `BoundsMaxX/Y` only as a sampled coverage guide. diff --git a/Packages/src/Editor/FirstPartyTools/SimulateMouseInput/Skill/SKILL.md b/Packages/src/Editor/FirstPartyTools/SimulateMouseInput/Skill/SKILL.md index 887c18b908..4e704499fc 100644 --- a/Packages/src/Editor/FirstPartyTools/SimulateMouseInput/Skill/SKILL.md +++ b/Packages/src/Editor/FirstPartyTools/SimulateMouseInput/Skill/SKILL.md @@ -29,8 +29,8 @@ uloop simulate-mouse-input --action [options] | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `--action` | enum | `Click` | `Click`, `LongPress`, `MoveDelta`, `SmoothDelta`, `Scroll` | -| `--x` | number | `0` | Target X position in Game View pixels (origin: top-left). Used by Click and LongPress. Use `AnnotatedElements[].SimX`, `RaycastGridPoints[].InputX`, or raw image pixels converted with `ScreenshotToInputFormula`. | -| `--y` | number | `0` | Target Y position in Game View pixels (origin: top-left). Used by Click and LongPress. Use `AnnotatedElements[].SimY`, `RaycastGridPoints[].InputY`, or raw image pixels converted with `ScreenshotToInputFormula`. | +| `--x` | number | `0` | Target X position in Game View pixels (origin: top-left). Used by Click and LongPress. Use `AnnotatedElements[].SimX`, or raw image pixels converted with `ScreenshotToInputFormula`. | +| `--y` | number | `0` | Target Y position in Game View pixels (origin: top-left). Used by Click and LongPress. Use `AnnotatedElements[].SimY`, or raw image pixels converted with `ScreenshotToInputFormula`. | | `--button` | enum | `Left` | Mouse button: `Left`, `Right`, `Middle`. Used by Click and LongPress. | | `--duration` | number | `0` | Hold duration for LongPress, or interpolation duration for SmoothDelta (seconds). For Click, 0 = one-shot tap. | | `--delta-x` | number | `0` | Delta X in pixels for MoveDelta/SmoothDelta. Positive = right. | @@ -103,7 +103,7 @@ uloop simulate-mouse-input --action SmoothDelta --delta-x 300 --delta-y 0 --dura - `--x` / `--y` use **top-left Game View coordinates**. - Raw image pixels from `uloop screenshot --capture-mode rendering` must be converted with `ScreenshotToInputFormula`. -- `AnnotatedElements[].SimX/SimY` and `RaycastGridPoints[].InputX/InputY` can be passed directly to this tool. +- `AnnotatedElements[].SimX/SimY` can be passed directly to this tool. - Do not flip Y in the caller. The tool converts internally for Unity Input System: ```text diff --git a/Packages/src/Editor/ToolContracts/RaycastGridPointInfo.cs b/Packages/src/Editor/ToolContracts/RaycastGridPointInfo.cs deleted file mode 100644 index 3fef9ff4a1..0000000000 --- a/Packages/src/Editor/ToolContracts/RaycastGridPointInfo.cs +++ /dev/null @@ -1,22 +0,0 @@ -#nullable enable - -namespace io.github.hatayama.UnityCliLoop.ToolContracts -{ - /// - /// Describes one coarse raycast grid sample point used to preview 3D physics hits on a screenshot. - /// - public class RaycastGridPointInfo - { - public string Label { get; set; } = ""; - public bool Hit { get; set; } - public float InputX { get; set; } - public float InputY { get; set; } - public float InjectedUnityPositionX { get; set; } - public float InjectedUnityPositionY { get; set; } - public string? HitGameObjectName { get; set; } - public string? HitGameObjectPath { get; set; } - public string? HitLayer { get; set; } - public int? HitLayerIndex { get; set; } - public float? Distance { get; set; } - } -} diff --git a/Packages/src/Editor/ToolContracts/RaycastGridPointInfo.cs.meta b/Packages/src/Editor/ToolContracts/RaycastGridPointInfo.cs.meta deleted file mode 100644 index cc11daf730..0000000000 --- a/Packages/src/Editor/ToolContracts/RaycastGridPointInfo.cs.meta +++ /dev/null @@ -1,11 +0,0 @@ -fileFormatVersion: 2 -guid: 21d97469b484c437aa867ba4141d02f9 -MonoImporter: - externalObjects: {} - serializedVersion: 2 - defaultReferences: [] - executionOrder: 0 - icon: {instanceID: 0} - userData: - assetBundleName: - assetBundleVariant: diff --git a/Packages/src/Editor/ToolContracts/ScreenshotResponse.cs b/Packages/src/Editor/ToolContracts/ScreenshotResponse.cs index 80548624d5..a9a6d54986 100644 --- a/Packages/src/Editor/ToolContracts/ScreenshotResponse.cs +++ b/Packages/src/Editor/ToolContracts/ScreenshotResponse.cs @@ -25,8 +25,8 @@ public class ScreenshotInfo public string ScreenshotToInputFormula { get; set; } = UnityCliLoopConstants.SCREENSHOT_WINDOW_TO_INPUT_FORMULA_UNAVAILABLE; public string UnityInputFormula { get; set; } = ""; public List AnnotatedElements { get; set; } = new List(); - public List RaycastGridPoints { get; set; } = new List(); public List RaycastLayerSummaries { get; set; } = new List(); + public List RaycastLayerNamesChecked { get; set; } = new List(); } /// diff --git a/README.md b/README.md index d7f8eb2ea1..d7fe4e76af 100644 --- a/README.md +++ b/README.md @@ -344,7 +344,7 @@ Great for keeping visual feedback in sync after other apps steal focus. (Linux i Take a screenshot of any EditorWindow as a PNG. Specify the window name (the text displayed in the title bar/tab) to capture. When multiple windows of the same type are open (e.g., 3 Inspector windows), all windows are saved with numbered filenames. Supports three matching modes: `exact` (default), `prefix`, and `contains` - all case-insensitive. -Use `CaptureMode: rendering` when you need coordinates for `simulate-mouse-input`, `simulate-mouse-ui`, or `raycast`; convert raw image pixels with `ScreenshotToInputFormula`, or pass annotated `SimX/SimY` and raycast-grid `InputX/InputY` values directly. +Use `CaptureMode: rendering` when you need coordinates for `simulate-mouse-input`, `simulate-mouse-ui`, or `raycast`; convert raw image pixels with `ScreenshotToInputFormula`, or pass annotated `SimX/SimY` values directly. ```text → screenshot (WindowName: "Console") → screenshot (CaptureMode: rendering, AnnotateRaycastGrid: true) diff --git a/cli/common/tools/default-tools.json b/cli/common/tools/default-tools.json index 93d5b7f09a..e434429cca 100644 --- a/cli/common/tools/default-tools.json +++ b/cli/common/tools/default-tools.json @@ -271,12 +271,12 @@ }, "AnnotateRaycastGrid": { "type": "boolean", - "description": "Annotate 3D physics raycast candidate points on rendering screenshots. Uses Camera.main, Camera.main.cullingMask visibility, and the same top-left Game View coordinates as simulate-mouse-input.", + "description": "Annotate clustered 3D physics raycast candidates (PhysicsCollider entries in AnnotatedElements) on rendering screenshots. Uses Camera.main, Camera.main.cullingMask visibility, and the same top-left Game View coordinates as simulate-mouse-input.", "default": false }, "RaycastLayerMask": { "type": "string", - "description": "Comma-separated physics layer names used by AnnotateRaycastGrid. Hits are limited to layers also visible to Camera.main.cullingMask. When set, returns clustered PhysicsCollider entries in AnnotatedElements instead of the coarse grid points.", + "description": "Comma-separated physics layer names used by AnnotateRaycastGrid to narrow which layers are clustered. Hits are limited to layers also visible to Camera.main.cullingMask. When omitted, clusters against Physics.DefaultRaycastLayers.", "default": "" } } @@ -470,12 +470,12 @@ }, "X": { "type": "number", - "description": "Target X position in Game View pixels (origin: top-left). Used by Click and LongPress. Use AnnotatedElements[].SimX, RaycastGridPoints[].InputX, or raw image pixels converted with ScreenshotToInputFormula.", + "description": "Target X position in Game View pixels (origin: top-left). Used by Click and LongPress. Use AnnotatedElements[].SimX, or raw image pixels converted with ScreenshotToInputFormula.", "default": 0 }, "Y": { "type": "number", - "description": "Target Y position in Game View pixels (origin: top-left). Used by Click and LongPress. Use AnnotatedElements[].SimY, RaycastGridPoints[].InputY, or raw image pixels converted with ScreenshotToInputFormula.", + "description": "Target Y position in Game View pixels (origin: top-left). Used by Click and LongPress. Use AnnotatedElements[].SimY, or raw image pixels converted with ScreenshotToInputFormula.", "default": 0 }, "Button": {