From 1af35b93dbf2e92cd15c7c2b4a23ce07f3986a9f Mon Sep 17 00:00:00 2001 From: Yan Jin Date: Tue, 15 Sep 2026 09:30:51 -0700 Subject: [PATCH 1/6] feat: add the polyspatial-playtest skill and recording-analyst agent Prove gameplay changes by recording a Play session as a PolySpatial .qrec, driving the game with simulated input, and answering from the recorded per-frame scene state. Resolves annotation references (#) left on recordings. The recording-analyst sub-agent measures strictly from the recording; the Codex equivalent ships as a toml under references/. --- agents/recording-analyst.md | 56 +++++ skills/polyspatial-playtest/SKILL.md | 199 ++++++++++++++++++ .../references/codex-recording-analyst.toml | 24 +++ .../references/commands.md | 53 +++++ 4 files changed, 332 insertions(+) create mode 100644 agents/recording-analyst.md create mode 100644 skills/polyspatial-playtest/SKILL.md create mode 100644 skills/polyspatial-playtest/references/codex-recording-analyst.toml create mode 100644 skills/polyspatial-playtest/references/commands.md diff --git a/agents/recording-analyst.md b/agents/recording-analyst.md new file mode 100644 index 0000000..002519f --- /dev/null +++ b/agents/recording-analyst.md @@ -0,0 +1,56 @@ +--- +name: recording-analyst +description: Answers quantitative questions about a PolySpatial recording (.qrec) strictly from the recorded per-frame scene state — how many times something rotated or changed color, how far it moved, when it appeared or disappeared, what changed around an annotation. Use it whenever a question about a recording or an annotation reference (`#`) can be settled by numbers; it never reads project source or scenes, so its answer is a measurement, not an interpretation. +tools: Bash +model: sonnet +--- + +You measure. You are handed a Unity project path, a recording (or an annotation reference that resolves +to one) and a question. You answer the question with numbers taken from the recording, with the frames +and seconds they came from, and nothing else. + +## Tools you use + +Only these, always with `--project-path `: + +- `unity command polyspatial_annotation_show --ref "" [--window N]` — resolve a reference: text, + frame, entity path, state at that frame, changes around it. +- `unity command polyspatial_annotation_list`, `polyspatial_recording_list`, `polyspatial_recording_metadata`. +- `unity command polyspatial_scene_state --recording R --entity "" --start-frame A --end-frame B [--properties p1,p2] [--summarize true]` +- `unity command polyspatial_scene_changes --recording R --start-frame A --end-frame B [--entity X] [--properties ...] [--group-depth 2]` +- `unity command polyspatial_entity_timeline --recording R --entity X --property position|worldPosition|rotation|worldRotation|scale --step 1` +- `unity command polyspatial_scene_export --out Temp/.ndjson ...` — same as scene_state, to a file. +- `python3` on files you exported or redirected into your scratch directory. + +Never read `Assets/`, `Packages/`, `ProjectSettings/`, `.unity`, `.prefab` or `.cs` files, never run +`grep` over the project, never `editor_play`, `simulate_*`, `capture_game_view` or anything that changes +the Editor. If the recording cannot answer the question, say exactly which data is missing. + +## How you work + +1. Resolve the reference first; note `entityPath`, `frame`, `frameEnd`, `recordingFrames`. +2. Decide which entity and property answer the question. A property that is constant on the annotated + entity usually lives on an ancestor: walk up the `entityPath` one level at a time. Names repeat; when + a command reports an ambiguous name, pass a path suffix such as `Parent/Child`. +3. Pull every frame you need (`--step 1`, or `--properties` to keep only the property you care about) + and redirect anything longer than a screen to a file. Compute on the file: + - rotation: for consecutive quaternions q0,q1 take delta = q1 * inverse(q0), convert to angle-axis, + accumulate signed angle about the dominant axis; net turns = sum/360, total turns = sum|angle|/360; + runs of |delta| > 0 are bursts. Report net and total separately. + - distance: sum |worldPosition(i) - worldPosition(i-1)|. + - counts: the length of a `[[frame,value],...]` keyframe array minus one. + - presence: `lifecycle` keyframes. +4. Convert frames to seconds with the `time` fields of `polyspatial_annotation_list` or by + `polyspatial_scene_changes --start-time/--end-time`. + +## What you return + +A short report, nothing else: + +- **Answer**: the number(s), with unit, and the frame range and seconds they cover. +- **Where it lives**: entity path and property that produced the answer (e.g. "rotation is on the + parent `MMCupcake`, the annotated `Cup` is rigid"). +- **Evidence**: 2–5 lines of the raw values or the bursts you found. +- **Commands**: the exact commands you ran, so the caller can reproduce. +- **Not measurable**: anything the question asked that the recording cannot show (pixels, script + variables, why something happened). diff --git a/skills/polyspatial-playtest/SKILL.md b/skills/polyspatial-playtest/SKILL.md new file mode 100644 index 0000000..25c22dc --- /dev/null +++ b/skills/polyspatial-playtest/SKILL.md @@ -0,0 +1,199 @@ +--- +name: polyspatial-playtest +description: Use when you must prove that gameplay, UI, or behavior in a Unity project actually works or is actually broken — "verify my change in Play mode", "does the button do X", "play-test this", "why does the cupcake stop spinning", "check the fix", or whenever someone hands you a PolySpatial annotation reference like `MyRecording-2026-9-11-101947#7d29bfdc…`. Records the Play session as a PolySpatial .qrec, drives the game with simulated input, then answers from the recorded per-frame scene state instead of eyeballing a screenshot. Requires a running Editor with com.unity.pipeline and the PolySpatial recording commands (`unity command --query polyspatial`). +allowed-tools: + - Bash + - Read +--- + +# PolySpatial play-test and debug + +Screenshots tell you what a frame looked like; a PolySpatial recording tells you what every +entity *was* on every frame — transforms, component properties, lifecycle, audio — and you can +query it after Play mode has ended. This skill is the loop: understand the request (often an +annotation someone left on a recording), change the code, then **play the game for real, record +it, and read the numbers back**. Use `capture_game_view` only to confirm what a frame looks like +once you already know from the data which frame matters. + +All commands below are `unity command [--flag value]`. Always pass `--project-path ` +when more than one Editor may be open. Read [references/commands.md](references/commands.md) for +every flag and the shape of each result. + +## 0. Preconditions (check once per session) + +```bash +unity status # an Editor for this project, state "ready" +unity command --query polyspatial --detail compact # the polyspatial_* commands must be listed +unity command set_autotick --enable true # an unfocused Editor barely advances frames otherwise +``` + +If `polyspatial_*` commands are missing, the project lacks the PolySpatial recording packages: say +so, and fall back to `editor_play` plus `capture_game_view`. Do not hand-edit `.unity`/`.prefab` +files while an Editor is reachable (see the `unity-cli` skill). + +The game must use the **Input System** for simulated input to reach it; legacy `Input.GetKey` code +cannot be driven. Check `ProjectSettings/ProjectSettings.asset` → `activeInputHandler` (1 or 2), or +just try `simulate_key` in Play mode and see whether the game reacts. + +## 1. Start from an annotation reference + +A reference looks like `#` (people copy it with the `Ref` button in +Window ▸ PolySpatial ▸ Annotations). It may also arrive as a bare id or as a path to the +annotation's `.json`. Resolve it first; never guess what it points at: + +```bash +unity command polyspatial_annotation_list # every annotation, one JSON line each +unity command polyspatial_annotation_show --ref "#" --window 30 +``` + +`polyspatial_annotation_show` returns: + +- `annotation.text` — what the person said, and `frame`/`time` — when. `kind` is `entity` when they + right-clicked an object (then `entityPath`, `worldPosition`, `worldBoundsSize`, `hitPoint` are set) + or `moment` when they annotated the whole frame. +- `state` — the entity's subtree at that frame: world transform, components and their properties. +- `changes` — every property of that subtree that varied within ±`window` frames, with `from`, `to`, + `firstChangeFrame`, `lastChangeFrame`. For a moment annotation you get `changedEntities` grouped by + the top two hierarchy levels instead; drill in with `polyspatial_scene_changes --entity`. + +Read text and data together. "The cupcake stopped spinning" at frame 400 plus `world.rotation` +changing through frame 430 means it did *not* stop — the person is describing the expected +behavior, or noticed something else. "The guy is looking back" as a moment annotation plus +`changedEntities` naming only `DudeContainer` tells you which object to inspect. + +### Delegate the measuring + +Raw recording data is large and the arithmetic is mechanical, so keep both out of the main +conversation. In Claude Code, hand every quantitative question to the `recording-analyst` subagent +that ships with this plugin (`Agent` with `subagent_type: recording-analyst`), passing the project +path, the reference or recording, and the question verbatim. It can only run `polyspatial_*` commands +and compute on their output, so what comes back is a measurement with frames and seconds, and you +decide afterwards whether the *why* needs the code. In Codex there is no plugin subagent: copy +[references/codex-recording-analyst.toml](references/codex-recording-analyst.toml) into the project's +`.codex/agents/` and delegate the same way, or follow the rules below yourself. + +### Answer from the recording first, explain from the code second + +When the question is about what happened ("how many times did it rotate", "did it ever leave the +platform", "how long was the button disabled"), the recording is the source of truth and the +answer is a number you compute from it. Do that **before** opening any scene or script, and report +it as soon as you have it. Only then, if the person asked *why*, read the code that drives the +entity — and say plainly which part of your answer is measured and which is inferred from code. +Do not spend the session grepping `.unity` files while the measured answer sits unreported. + +Worked example, "how many times did the cupcake rotate?". The recording gives you samples; +the arithmetic is yours — dump every frame to a file and compute in a short script, exactly as +you would with any dataset: + +```bash +unity command polyspatial_annotation_show --ref "#" # entityPath: …/Cupcake/Wiggle/MMCupcake/Cup +unity command polyspatial_entity_timeline --recording --entity "MMCupcake/Cup" --property rotation --step 1 --json > cup.json +unity command polyspatial_entity_timeline --recording --entity "Wiggle/MMCupcake" --property rotation --step 1 --json > mmcupcake.json +# python: for consecutive quaternions q0,q1 take delta = q1 * inverse(q0), convert to angle-axis, +# accumulate signed angle about the dominant axis; sum / 360 = turns; runs of |delta| > 0 = bursts. +``` + +Result on the sample recording: `Cup` never turns relative to its parent (one distinct rotation +in 779 frames); `MMCupcake` turns 13.9 times about x in two bursts, frames 336–380 (6.9) and +384–428 (6.9). Answer with those numbers and frames first: "the cupcake spun ~13.9 turns in two +~7-turn bursts at 7.4–8.4 s and 8.5–9.5 s; the rotation is on the parent MMCupcake, the annotated +Cup mesh is rigid." Then, only if asked why: the MMF_Rotation feedback on that object. + +The same pattern answers "how many times did the color change" (`polyspatial_scene_state +--entity X --start-frame 1 --end-frame N --properties Image.color` and count that keyframe array), +"how far did the player travel" (sum `worldPosition` deltas), "was it ever inactive" (the +`lifecycle` keyframes). There is no per-question command; there are samples and your computation. +When a slice is more than a screen of text, `polyspatial_scene_export --out Temp/x.ndjson ...` +writes it to a file: compute on the file, print only the result. + +Rules for measuring: + +- A property that is constant on the annotated entity usually lives on an ancestor: walk up the + `entityPath` one level at a time. `worldRotation`/`worldPosition` give the composed result. +- Net rotation and total rotation differ: a wiggle travels many degrees and nets zero. Report + the one the question asks for, and say which. +- `--step 1` gives every frame; the default samples ~200 points, which is fine for a curve but + not for counting. +- Convert frames to seconds with the `time` fields of `polyspatial_annotation_list` or `--start-time` + on `polyspatial_scene_changes`; quote both in the answer. + +To see the moment, replay and park on the frame, then capture: + +```bash +unity command polyspatial_playback --recording # enters Play mode; wait for it +unity command polyspatial_playback_status # poll until playingBack is true +unity command polyspatial_playback_seek --frame 400 --pause true +unity command capture_game_view --source screen --save_path Temp/annotation-400.png +unity command polyspatial_record_stop # leaves Play mode +``` + +If the task is to change behavior, then go read the code that drives that entity (the hierarchy +path names the GameObjects), fix or implement, and prove it with section 2. + +## 2. Verify by playing: record → drive → stop → query + +Decide **before** recording what "correct" means as numbers: which entity, which property, which +frames or seconds, which value or change. Then: + +```bash +# 1. Open the scene to test (must be a saved scene; recording refuses untitled scenes). +unity command polyspatial_record_start # arms a .qrec and enters Play mode; returns its path +unity command polyspatial_playback_status # poll until inPlayMode and recording are true; note "frame" + +# 2. Drive the game. Timed sequences run over real frames; poll status until completed. +unity command simulate_input_script --script '{"steps":[{"at":0.5,"key":"W","action":"hold","duration":1.0},{"at":2.0,"x":640,"y":360,"action":"click"}]}' +unity command simulate_input_script_status # "fired" lists time and Time.frameCount per event +unity command click_ui_element --name "Play Button" # uGUI by GameObject name; scrolls it into view +unity command polyspatial_playback_status # note "frame" again: the recording frames you drove + +# 3. Stop and wait for the file. +unity command polyspatial_record_stop # returns the .qrec path +unity command polyspatial_recording_metadata --recording # poll until it answers; frameCount + +# 4. Ask the recording. +unity command polyspatial_scene_state --recording --summarize true # what exists +unity command polyspatial_scene_changes --recording --start-frame A --end-frame B --entity "Player" +unity command polyspatial_entity_timeline --recording --entity "Player" --property worldPosition --start-frame A --end-frame B +unity command polyspatial_scene_state --recording --entity "Enemy/HealthBar" --start-frame B --end-frame B +``` + +Rules of evidence: + +- Quote frames and values from the recording in your answer ("frame 1210–1290: `world.position.y` + rose from 0.00 to 2.31, then fell back by frame 1350"), and name the `.qrec` path so a person can + replay it. A screenshot alone is not proof. +- Entity names repeat (a UI scene has hundreds of `Text`). When a command says the name is + ambiguous, pass a hierarchy path suffix such as `Button - Scale/Text`. +- `polyspatial_entity_timeline --property` takes `position`, `worldPosition`, `rotation`, + `worldRotation`, `scale` (the `world.rotation` spelling from scene_state output also works); + frames are 1-based. +- Frames from `polyspatial_playback_status` while recording are recording frames; `Time.frameCount` + in `simulate_input_script_status` is the game's counter. Bracket with status before and after + driving, or convert with `--start-time/--end-time` on `polyspatial_scene_changes`. +- The Editor throttles when unfocused: expect frame rates that differ from a focused run, and use + seconds, not frame counts, when timing input. +- Leave Play mode with `polyspatial_record_stop` (or `editor_stop`); the scene that was open + before is restored. Never leave the Editor in Play mode. + +## 3. Inspect an existing recording without an annotation + +```bash +unity command polyspatial_recording_list +unity command polyspatial_recording_metadata --recording latest +unity command polyspatial_scene_state --recording latest --summarize true +unity command polyspatial_scene_changes --recording latest --start-time 0 --end-time 5 --group-depth 2 --include-components false +``` + +Start wide (`--summarize`, `--group-depth 2`) and narrow to one entity and a short frame range; +whole-scene keyframe dumps run to hundreds of kilobytes. + +## Gotchas + +- Entering or leaving Play mode reloads the domain: `unity command` may fail to connect for a few + seconds. Retry; do not assume the Editor died. +- `polyspatial_record_start` fails when already in Play mode, when the scene is untitled, or when + the scene has unsaved changes at playback time. Save first. +- Input screen coordinates are Game view pixels with the origin bottom-left; `capture_game_view` + reports the size it rendered at. +- Audio needs `UnityEngine.AudioSource` in PolySpatial Settings ▸ Generic Tracking Excluded Types + to be recorded as play/stop events; without it, audio shows no state changes. diff --git a/skills/polyspatial-playtest/references/codex-recording-analyst.toml b/skills/polyspatial-playtest/references/codex-recording-analyst.toml new file mode 100644 index 0000000..8e82842 --- /dev/null +++ b/skills/polyspatial-playtest/references/codex-recording-analyst.toml @@ -0,0 +1,24 @@ +# Codex sub-agent definition. Copy this file to /.codex/agents/recording-analyst.toml +# (plugins cannot ship Codex agents). Field names follow the Codex custom-agent format; check +# `codex agents --help` on your version if it refuses the file. + +name = "recording-analyst" +description = "Answers quantitative questions about a PolySpatial recording (.qrec) strictly from recorded per-frame scene state: rotations, distances, change counts, presence, what changed around an annotation reference. Never reads project source or scenes." + +developer_instructions = """ +You measure. You are handed a Unity project path, a recording or an annotation reference (#) and a question. Answer with numbers from the recording and the frames/seconds they came from. + +Only use, always with --project-path : + unity command polyspatial_annotation_show --ref "" [--window N] + unity command polyspatial_annotation_list | polyspatial_recording_list | polyspatial_recording_metadata + unity command polyspatial_scene_state --recording R --entity "" --start-frame A --end-frame B [--properties p1,p2] [--summarize true] + unity command polyspatial_scene_changes --recording R --start-frame A --end-frame B [--entity X] [--properties ...] [--group-depth 2] + unity command polyspatial_entity_timeline --recording R --entity X --property position|worldPosition|rotation|worldRotation|scale --step 1 + unity command polyspatial_scene_export --out Temp/.ndjson ... + python3 on files you exported or redirected. +Never read Assets/, Packages/, ProjectSettings/, .unity, .prefab or .cs files; never grep the project; never editor_play, simulate_*, capture_game_view. + +Method: resolve the reference; pick the entity and property (constant on the annotated entity usually means it lives on an ancestor, walk up the path); pull every frame with --step 1 or --properties, redirect to a file, compute (rotation: quaternion deltas to angle-axis, net vs total turns, bursts; distance: sum of position deltas; counts: keyframe array length minus one; presence: lifecycle keyframes); convert frames to seconds. + +Return only: Answer (numbers, unit, frame range, seconds); Where it lives (entity path, property); Evidence (2-5 raw lines); Commands run; Not measurable (what the recording cannot show). +""" diff --git a/skills/polyspatial-playtest/references/commands.md b/skills/polyspatial-playtest/references/commands.md new file mode 100644 index 0000000..a995958 --- /dev/null +++ b/skills/polyspatial-playtest/references/commands.md @@ -0,0 +1,53 @@ +# Command reference + +Every command is `unity command [--flag value] [--project-path ]`. Results come +back in the `result` field; several are NDJSON (one JSON object per line) so they stream and grep. + +## Annotations + +| Command | Flags | Result | +|---|---|---| +| `polyspatial_annotation_list` | `--recording all\|latest\|` (default all) | One line per annotation: `reference`, `recording`, `recordingPath`, `frame`, `frameEnd`, `time`, `kind` (`entity`\|`moment`), `text`, `created`, `createdBy`, and for entity annotations `entityId`, `entityName`, `entityPath`, `worldPosition`, `worldBoundsCenter`, `worldBoundsSize`, `hitPoint`; `camera` is the Scene view camera when it was written. | +| `polyspatial_annotation_show` | `--ref # \| \| ` (required), `--window 30`, `--include-components true`, `--max-changes 300` | `annotation` (as above), `recordingFrames`, `changeWindow {from,to}`, `state` (entity annotations: NDJSON lines of the subtree at the frame, parsed into an array), `changes` (entity) or `changedEntities` (moment), `changeCount`, `changesTruncated`, `entityResolvedByName`, `entityMissing`. | + +`kind` is `entity`, `region` (a circle drawn in the Scene view: `entityIds`, `entityPaths`, `entityCount`; show returns `members`, their `state` and only their `changes`), `span` (`frame`–`frameEnd`) or `moment`. + +A `changes` entry: `{ entity, component?, property, from, to, firstChangeFrame, lastChangeFrame, keyframes }`. +A `changedEntities` entry: `{ entity, changedProperties, properties[], firstChangeFrame, lastChangeFrame }`. + +## Recording and playback control + +| Command | Flags | Result | +|---|---|---| +| `polyspatial_record_start` | `--shaders false` | `{ armed, path, shaders }`; enters Play mode. | +| `polyspatial_record_stop` | | `{ stopping, path, note }`; leaves Play mode; poll `polyspatial_recording_metadata` for the path. | +| `polyspatial_playback` | `--recording latest\|\|` | `{ playing, path }`; opens an empty scene and replays. | +| `polyspatial_playback_seek` | `--frame N` or `--time s`, `--pause true` | The status object after the jump. Backward jumps restart the replay in place (a few hundred ms). | +| `polyspatial_playback_status` | | `{ inPlayMode, paused, playingBack, recording, recordingPath, frame, frameCount, time, duration, ended }`; while recording, `frame` is the recorder's frame counter. | +| `polyspatial_recording_list` | | One line per `.qrec`: `path`, `name`, `sizeKB`, `lastWriteUtc`. | +| `polyspatial_recording_metadata` | `--recording` | `{ path, name, version, frameCount, recordingType, commandCount }`. | + +## Recording queries + +| Command | Flags | Result | +|---|---|---| +| `polyspatial_scene_state` | `--recording`, `--entity `, `--start-frame`, `--end-frame`, `--summarize false`, `--include-components true`, `--transform-digits`, `--properties a,b`, `--exclude-properties c`, `--include-assets false`, `--include-inputs false` | NDJSON: a header line with counts, then one line per entity with `path`, `lifecycle`, `world.position/rotation/scale`, `components[]` and `Component.property` values; a property that varies inside the range becomes `[[frame,value],...]`. `--properties` keeps only the named properties (entity headers stay), which is the cheap way to ask "how many times did Image.color change": count that array. | +| `polyspatial_scene_export` | `--out ` plus every `polyspatial_scene_state` flag | Writes the same NDJSON to a file and returns `{ path, lines, bytes }`. Use it whenever the slice is more than a screen of text, then compute on the file. | +| `polyspatial_scene_changes` | `--start-frame/--end-frame` or `--start-time/--end-time`, `--recording`, `--entity`, `--include-components true`, `--max-changes 500`, `--group-depth 0`, `--properties`, `--exclude-properties` | `{ recording, from, to, changeCount, changes[] }` or, with `--group-depth`, `changedEntities[]`. Values that differ only by float rounding are dropped. | +| `polyspatial_entity_timeline` | `--entity` (required), `--recording`, `--property position\|worldPosition\|rotation\|worldRotation\|scale`, `--start-frame`, `--end-frame`, `--step 0` | A header line then `{ frame, x, y, z[, w] }` samples (rotations are quaternions); `--step 0` targets about 200 samples, `--step 1` every frame. Redirect to a file and compute on it. | + +Entity arguments accept a name, or a hierarchy path suffix (`Parent/Child`) when the name is +shared; an ambiguous name returns an error listing the candidate paths. + +## Driving the game (com.unity.pipeline) + +| Command | Flags | Result | +|---|---|---| +| `simulate_key` | `--key Space`, `--action press\|down\|up` | `{ Success, Detail, Error }`. Input System key names. | +| `simulate_pointer` | `--x --y`, `--action click\|move\|down\|up`, `--button left\|right\|middle` | Same. Screen pixels, origin bottom-left. | +| `simulate_input_script` | `--script '{"steps":[…],"releaseAtEnd":true}'` | `{ Status, ScriptId, StepCount, EventCount, EventsFired, StartFrame, EndFrame, StartTime, ElapsedSeconds, Fired[], Released[], Error }`. Steps: `{at, key, action: down\|up\|press\|hold, duration}` or `{at, x, y, button, action: move\|down\|up\|click\|hold, duration}`. Runs over the following frames; one script at a time. | +| `simulate_input_script_status` / `_cancel` | | The same status object. | +| `click_ui_element` | `--name `, `--button left`, `--index 0`, `--hold 0` | Status object; the first `Fired` line names the resolved path and screen position, and whether it scrolled the element into view. Fails with the position when the element is off screen. | +| `editor_play` / `editor_stop` / `editor_pause` | | Plain Play mode control without recording. | +| `capture_game_view` | `--source screen`, `--save_path ` | PNG of the Game view; `screen` includes overlay UI and needs Play mode. | +| `set_autotick` | `--enable true` | Keeps the Editor ticking while unfocused. | From 2c2edf7d14f40799bfcd040796f2d479389d40f8 Mon Sep 17 00:00:00 2001 From: Yan Jin Date: Wed, 16 Sep 2026 15:10:45 -0700 Subject: [PATCH 2/6] Drive recording and playback through eval The polyspatial_record_* and polyspatial_playback_* commands are not shipping; the skill calls RecordingPlaybackScene through unity command eval and leaves Play mode with editor_stop. --- skills/polyspatial-playtest/SKILL.md | 26 ++++++++++--------- .../references/commands.md | 20 +++++++++----- 2 files changed, 28 insertions(+), 18 deletions(-) diff --git a/skills/polyspatial-playtest/SKILL.md b/skills/polyspatial-playtest/SKILL.md index 25c22dc..125c2bc 100644 --- a/skills/polyspatial-playtest/SKILL.md +++ b/skills/polyspatial-playtest/SKILL.md @@ -120,11 +120,11 @@ Rules for measuring: To see the moment, replay and park on the frame, then capture: ```bash -unity command polyspatial_playback --recording # enters Play mode; wait for it -unity command polyspatial_playback_status # poll until playingBack is true -unity command polyspatial_playback_seek --frame 400 --pause true +R=UnityEditor.PolySpatial.Utilities.RecordingPlaybackScene +unity command eval --code "return $R.StartPlaybackAt(\"\", 400, true);" # enters Play mode, replays, parks on frame 400; null on success +unity command eval --code "return \$\"{$R.IsPlayingBack} {$R.CurrentFrame}\";" # poll until "True 400" unity command capture_game_view --source screen --save_path Temp/annotation-400.png -unity command polyspatial_record_stop # leaves Play mode +unity command editor_stop # leaves Play mode ``` If the task is to change behavior, then go read the code that drives that entity (the hierarchy @@ -137,17 +137,18 @@ frames or seconds, which value or change. Then: ```bash # 1. Open the scene to test (must be a saved scene; recording refuses untitled scenes). -unity command polyspatial_record_start # arms a .qrec and enters Play mode; returns its path -unity command polyspatial_playback_status # poll until inPlayMode and recording are true; note "frame" +R=UnityEditor.PolySpatial.Utilities.RecordingPlaybackScene +unity command eval --code "return $R.StartRecording();" # arms a .qrec and enters Play mode; returns its path +unity command eval --code "return \$\"{$R.IsLiveSession} {$R.LiveFrame}\";" # poll until True; note the frame # 2. Drive the game. Timed sequences run over real frames; poll status until completed. unity command simulate_input_script --script '{"steps":[{"at":0.5,"key":"W","action":"hold","duration":1.0},{"at":2.0,"x":640,"y":360,"action":"click"}]}' unity command simulate_input_script_status # "fired" lists time and Time.frameCount per event unity command click_ui_element --name "Play Button" # uGUI by GameObject name; scrolls it into view -unity command polyspatial_playback_status # note "frame" again: the recording frames you drove +unity command eval --code "return $R.LiveFrame;" # note the frame again: the recording frames you drove # 3. Stop and wait for the file. -unity command polyspatial_record_stop # returns the .qrec path +unity command editor_stop # the .qrec finalizes on exit unity command polyspatial_recording_metadata --recording # poll until it answers; frameCount # 4. Ask the recording. @@ -167,12 +168,12 @@ Rules of evidence: - `polyspatial_entity_timeline --property` takes `position`, `worldPosition`, `rotation`, `worldRotation`, `scale` (the `world.rotation` spelling from scene_state output also works); frames are 1-based. -- Frames from `polyspatial_playback_status` while recording are recording frames; `Time.frameCount` +- `LiveFrame` while recording is the recording frame counter; `Time.frameCount` in `simulate_input_script_status` is the game's counter. Bracket with status before and after driving, or convert with `--start-time/--end-time` on `polyspatial_scene_changes`. - The Editor throttles when unfocused: expect frame rates that differ from a focused run, and use seconds, not frame counts, when timing input. -- Leave Play mode with `polyspatial_record_stop` (or `editor_stop`); the scene that was open +- Leave Play mode with `editor_stop`; the scene that was open before is restored. Never leave the Editor in Play mode. ## 3. Inspect an existing recording without an annotation @@ -191,8 +192,9 @@ whole-scene keyframe dumps run to hundreds of kilobytes. - Entering or leaving Play mode reloads the domain: `unity command` may fail to connect for a few seconds. Retry; do not assume the Editor died. -- `polyspatial_record_start` fails when already in Play mode, when the scene is untitled, or when - the scene has unsaved changes at playback time. Save first. +- `StartRecording` returns `Error: ...` when already in Play mode or when the scene is untitled; + `StartPlaybackAt` refuses a scene with unsaved changes. Save first. Recording and playback have no + `polyspatial_*` commands of their own: drive them through `eval` as shown above. - Input screen coordinates are Game view pixels with the origin bottom-left; `capture_game_view` reports the size it rendered at. - Audio needs `UnityEngine.AudioSource` in PolySpatial Settings ▸ Generic Tracking Excluded Types diff --git a/skills/polyspatial-playtest/references/commands.md b/skills/polyspatial-playtest/references/commands.md index a995958..5908bd0 100644 --- a/skills/polyspatial-playtest/references/commands.md +++ b/skills/polyspatial-playtest/references/commands.md @@ -15,15 +15,23 @@ back in the `result` field; several are NDJSON (one JSON object per line) so the A `changes` entry: `{ entity, component?, property, from, to, firstChangeFrame, lastChangeFrame, keyframes }`. A `changedEntities` entry: `{ entity, changedProperties, properties[], firstChangeFrame, lastChangeFrame }`. -## Recording and playback control +## Recording and playback (through `eval`) + +No `polyspatial_*` command enters Play mode. Call the public `UnityEditor.PolySpatial.Utilities.RecordingPlaybackScene` API through `unity command eval --code "..."`: + +| Call | Result | +|---|---| +| `return R.StartRecording();` | The new `.qrec` path, or `Error: ...` (already in Play mode, untitled scene). Enters Play mode. | +| `return $"{R.IsLiveSession} {R.LiveFrame}";` | `True ` once the recorder runs; `LiveFrame` is the recording frame counter. | +| `unity command editor_stop` | Leaves Play mode; the file finalizes. Poll `polyspatial_recording_metadata` for it. | +| `return R.StartPlaybackAt("", , true);` | Replays `` parked on ``; `null` on success. Works from Edit mode, from a live session, or during another replay. | +| `R.SeekTo(, true); return R.CurrentFrame;` | Seeks inside the running replay; backward seeks restart it in place (a few hundred ms). | +| `return $"{R.IsPlayingBack} {R.CurrentFrame} {R.PlaybackEnded}";` | Replay status. | + +`R` stands for the full `UnityEditor.PolySpatial.Utilities.RecordingPlaybackScene`; `eval` has no `using`, so spell it out. | Command | Flags | Result | |---|---|---| -| `polyspatial_record_start` | `--shaders false` | `{ armed, path, shaders }`; enters Play mode. | -| `polyspatial_record_stop` | | `{ stopping, path, note }`; leaves Play mode; poll `polyspatial_recording_metadata` for the path. | -| `polyspatial_playback` | `--recording latest\|\|` | `{ playing, path }`; opens an empty scene and replays. | -| `polyspatial_playback_seek` | `--frame N` or `--time s`, `--pause true` | The status object after the jump. Backward jumps restart the replay in place (a few hundred ms). | -| `polyspatial_playback_status` | | `{ inPlayMode, paused, playingBack, recording, recordingPath, frame, frameCount, time, duration, ended }`; while recording, `frame` is the recorder's frame counter. | | `polyspatial_recording_list` | | One line per `.qrec`: `path`, `name`, `sizeKB`, `lastWriteUtc`. | | `polyspatial_recording_metadata` | `--recording` | `{ path, name, version, frameCount, recordingType, commandCount }`. | From b2e2e668b42aedd324d856384881a1326492130d Mon Sep 17 00:00:00 2001 From: Yan Jin Date: Wed, 16 Sep 2026 17:02:01 -0700 Subject: [PATCH 3/6] Replay runs on the timeline without Play mode --- skills/polyspatial-playtest/SKILL.md | 8 ++++---- skills/polyspatial-playtest/references/commands.md | 6 ++++-- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/skills/polyspatial-playtest/SKILL.md b/skills/polyspatial-playtest/SKILL.md index 125c2bc..769ed67 100644 --- a/skills/polyspatial-playtest/SKILL.md +++ b/skills/polyspatial-playtest/SKILL.md @@ -121,10 +121,10 @@ To see the moment, replay and park on the frame, then capture: ```bash R=UnityEditor.PolySpatial.Utilities.RecordingPlaybackScene -unity command eval --code "return $R.StartPlaybackAt(\"\", 400, true);" # enters Play mode, replays, parks on frame 400; null on success -unity command eval --code "return \$\"{$R.IsPlayingBack} {$R.CurrentFrame}\";" # poll until "True 400" -unity command capture_game_view --source screen --save_path Temp/annotation-400.png -unity command editor_stop # leaves Play mode +unity command eval --code "return $R.StartPlaybackAt(\"\", 400, true);" # rebuilds the recording in the open scene, parked on frame 400; null on success; no Play mode +unity command eval --code "return \$\"{$R.IsPlayingBack} {$R.CurrentFrame}\";" # "True 400" +unity command capture_game_view --save_path Temp/annotation-400.png # camera capture; "screen" needs Play mode +unity command eval --code "$R.StopPlayback(); return $R.IsPlayingBack;" # restores the scene's own objects ``` If the task is to change behavior, then go read the code that drives that entity (the hierarchy diff --git a/skills/polyspatial-playtest/references/commands.md b/skills/polyspatial-playtest/references/commands.md index 5908bd0..cfcae9d 100644 --- a/skills/polyspatial-playtest/references/commands.md +++ b/skills/polyspatial-playtest/references/commands.md @@ -24,9 +24,11 @@ No `polyspatial_*` command enters Play mode. Call the public `UnityEditor.PolySp | `return R.StartRecording();` | The new `.qrec` path, or `Error: ...` (already in Play mode, untitled scene). Enters Play mode. | | `return $"{R.IsLiveSession} {R.LiveFrame}";` | `True ` once the recorder runs; `LiveFrame` is the recording frame counter. | | `unity command editor_stop` | Leaves Play mode; the file finalizes. Poll `polyspatial_recording_metadata` for it. | -| `return R.StartPlaybackAt("", , true);` | Replays `` parked on ``; `null` on success. Works from Edit mode, from a live session, or during another replay. | -| `R.SeekTo(, true); return R.CurrentFrame;` | Seeks inside the running replay; backward seeks restart it in place (a few hundred ms). | +| `return R.StartPlaybackAt("", , true);` | Rebuilds `` on a timeline inside the open scene, parked on ``; `null` on success. Never enters Play mode; the scene's own objects are deactivated until `StopPlayback`. | +| `R.SeekTo(, true); return R.CurrentFrame;` | Rebuilds that frame directly, in either direction. | +| `R.IsPaused = false;` / `R.IsPaused = true;` | Plays in real time / pauses. | | `return $"{R.IsPlayingBack} {R.CurrentFrame} {R.PlaybackEnded}";` | Replay status. | +| `R.StopPlayback();` | Tears the timeline down and reactivates the scene's objects. | `R` stands for the full `UnityEditor.PolySpatial.Utilities.RecordingPlaybackScene`; `eval` has no `using`, so spell it out. From 3ef7e21a12805a86ddd2d898b3a955b40ed13524 Mon Sep 17 00:00:00 2001 From: Yan Jin Date: Thu, 17 Sep 2026 10:39:09 -0700 Subject: [PATCH 4/6] Query recordings through eval instead of the removed polyspatial commands com.unity.polyspatial.annotation keeps only polyspatial_annotation_list and polyspatial_annotation_show. Listing, header, scene state, changes and timelines are SceneStateQuery chains run with eval_file against the public scene-state API (polyspatial #4964), following the Muse play-session-recording skill templates (polyspatial #4963). --- agents/recording-analyst.md | 25 ++-- skills/polyspatial-playtest/SKILL.md | 135 +++++++++++------- .../references/codex-recording-analyst.toml | 15 +- .../references/commands.md | 52 +++++-- 4 files changed, 144 insertions(+), 83 deletions(-) diff --git a/agents/recording-analyst.md b/agents/recording-analyst.md index 002519f..2746b1f 100644 --- a/agents/recording-analyst.md +++ b/agents/recording-analyst.md @@ -15,12 +15,14 @@ Only these, always with `--project-path `: - `unity command polyspatial_annotation_show --ref "" [--window N]` — resolve a reference: text, frame, entity path, state at that frame, changes around it. -- `unity command polyspatial_annotation_list`, `polyspatial_recording_list`, `polyspatial_recording_metadata`. -- `unity command polyspatial_scene_state --recording R --entity "" --start-frame A --end-frame B [--properties p1,p2] [--summarize true]` -- `unity command polyspatial_scene_changes --recording R --start-frame A --end-frame B [--entity X] [--properties ...] [--group-depth 2]` -- `unity command polyspatial_entity_timeline --recording R --entity X --property position|worldPosition|rotation|worldRotation|scale --step 1` -- `unity command polyspatial_scene_export --out Temp/.ndjson ...` — same as scene_state, to a file. -- `python3` on files you exported or redirected into your scratch directory. +- `unity command polyspatial_annotation_list`. +- `unity command eval_file --file --timeout 120` running a read-only `SceneStateQuery` over + `PolySpatialSceneStateRecordingLoader.Load(recordingPath)`: `Summarize()` to resolve names to + `instanceId`, then `FilterBySubtree(id)`, `FilterByFrameRange(a, b)`, `FilterProperties(include: …)`, + `WorldTransforms()` / `LocalTransforms()`, `Diff(a, b)`, `ToJson()`; write the NDJSON to + `Temp/.ndjson`. The exact template is in the `polyspatial-playtest` skill's + `references/commands.md`. The script only reads the recording and writes under `Temp/`. +- `python3` on the files you wrote. Never read `Assets/`, `Packages/`, `ProjectSettings/`, `.unity`, `.prefab` or `.cs` files, never run `grep` over the project, never `editor_play`, `simulate_*`, `capture_game_view` or anything that changes @@ -31,17 +33,18 @@ the Editor. If the recording cannot answer the question, say exactly which data 1. Resolve the reference first; note `entityPath`, `frame`, `frameEnd`, `recordingFrames`. 2. Decide which entity and property answer the question. A property that is constant on the annotated entity usually lives on an ancestor: walk up the `entityPath` one level at a time. Names repeat; when - a command reports an ambiguous name, pass a path suffix such as `Parent/Child`. -3. Pull every frame you need (`--step 1`, or `--properties` to keep only the property you care about) - and redirect anything longer than a screen to a file. Compute on the file: + names collide, pick the `instanceId` whose `path` in the summary ends with `Parent/Child`. +3. Pull the keyframes you need (`FilterProperties` keeps only the property you care about; a value is + stored only where it changed, hold the last value across frames when you need per-frame samples) + and write anything longer than a screen to a file. Compute on the file: - rotation: for consecutive quaternions q0,q1 take delta = q1 * inverse(q0), convert to angle-axis, accumulate signed angle about the dominant axis; net turns = sum/360, total turns = sum|angle|/360; runs of |delta| > 0 are bursts. Report net and total separately. - distance: sum |worldPosition(i) - worldPosition(i-1)|. - counts: the length of a `[[frame,value],...]` keyframe array minus one. - presence: `lifecycle` keyframes. -4. Convert frames to seconds with the `time` fields of `polyspatial_annotation_list` or by - `polyspatial_scene_changes --start-time/--end-time`. +4. Convert frames to seconds with the `time` fields of `polyspatial_annotation_list`, or query by + seconds with `FilterByTimeRange(s0, s1)`. ## What you return diff --git a/skills/polyspatial-playtest/SKILL.md b/skills/polyspatial-playtest/SKILL.md index 769ed67..7f8cd18 100644 --- a/skills/polyspatial-playtest/SKILL.md +++ b/skills/polyspatial-playtest/SKILL.md @@ -1,6 +1,6 @@ --- name: polyspatial-playtest -description: Use when you must prove that gameplay, UI, or behavior in a Unity project actually works or is actually broken — "verify my change in Play mode", "does the button do X", "play-test this", "why does the cupcake stop spinning", "check the fix", or whenever someone hands you a PolySpatial annotation reference like `MyRecording-2026-9-11-101947#7d29bfdc…`. Records the Play session as a PolySpatial .qrec, drives the game with simulated input, then answers from the recorded per-frame scene state instead of eyeballing a screenshot. Requires a running Editor with com.unity.pipeline and the PolySpatial recording commands (`unity command --query polyspatial`). +description: Use when you must prove that gameplay, UI, or behavior in a Unity project actually works or is actually broken — "verify my change in Play mode", "does the button do X", "play-test this", "why does the cupcake stop spinning", "check the fix", or whenever someone hands you a PolySpatial annotation reference like `MyRecording-2026-9-11-101947#7d29bfdc…`. Records the Play session as a PolySpatial .qrec, drives the game with simulated input, then answers from the recorded per-frame scene state instead of eyeballing a screenshot. Requires a running Editor with com.unity.pipeline and com.unity.polyspatial.annotation (`unity command --query polyspatial` lists `polyspatial_annotation_*`). allowed-tools: - Bash - Read @@ -15,21 +15,24 @@ annotation someone left on a recording), change the code, then **play the game f it, and read the numbers back**. Use `capture_game_view` only to confirm what a frame looks like once you already know from the data which frame matters. -All commands below are `unity command [--flag value]`. Always pass `--project-path ` -when more than one Editor may be open. Read [references/commands.md](references/commands.md) for -every flag and the shape of each result. +Two kinds of call. Annotations have commands: `unity command polyspatial_annotation_list` and +`polyspatial_annotation_show`. Everything else about a recording is a C# query you run in the +Editor with `unity command eval_file --file ` against the public `PolySpatialSceneState` +API. Always pass `--project-path ` when more than one Editor may be open. Read +[references/commands.md](references/commands.md) for every flag, the query API and the shape of +each result. ## 0. Preconditions (check once per session) ```bash unity status # an Editor for this project, state "ready" -unity command --query polyspatial --detail compact # the polyspatial_* commands must be listed +unity command --query polyspatial --detail compact # polyspatial_annotation_list / _show must be listed unity command set_autotick --enable true # an unfocused Editor barely advances frames otherwise ``` -If `polyspatial_*` commands are missing, the project lacks the PolySpatial recording packages: say -so, and fall back to `editor_play` plus `capture_game_view`. Do not hand-edit `.unity`/`.prefab` -files while an Editor is reachable (see the `unity-cli` skill). +If the `polyspatial_annotation_*` commands are missing, the project lacks the PolySpatial +annotation package: say so, and fall back to `editor_play` plus `capture_game_view`. Do not +hand-edit `.unity`/`.prefab` files while an Editor is reachable (see the `unity-cli` skill). The game must use the **Input System** for simulated input to reach it; legacy `Input.GetKey` code cannot be driven. Check `ProjectSettings/ProjectSettings.asset` → `activeInputHandler` (1 or 2), or @@ -48,13 +51,14 @@ unity command polyspatial_annotation_show --ref "#" --window 30 `polyspatial_annotation_show` returns: -- `annotation.text` — what the person said, and `frame`/`time` — when. `kind` is `entity` when they - right-clicked an object (then `entityPath`, `worldPosition`, `worldBoundsSize`, `hitPoint` are set) - or `moment` when they annotated the whole frame. +- `annotation.text` — what the person said, and `frame`/`time` — when; `recordingPath` — the + `.qrec` to query. `kind` is `entity` when they right-clicked an object (then `entityId`, + `entityPath`, `worldPosition`, `worldBoundsSize`, `hitPoint` are set), `entities` for a box or + lasso selection (`members`), `span` for a time range, or `moment` for the whole frame. - `state` — the entity's subtree at that frame: world transform, components and their properties. - `changes` — every property of that subtree that varied within ±`window` frames, with `from`, `to`, `firstChangeFrame`, `lastChangeFrame`. For a moment annotation you get `changedEntities` grouped by - the top two hierarchy levels instead; drill in with `polyspatial_scene_changes --entity`. + the top two hierarchy levels instead; drill in with a subtree query (section 4). Read text and data together. "The cupcake stopped spinning" at frame 400 plus `world.rotation` changing through frame 430 means it did *not* stop — the person is describing the expected @@ -66,9 +70,10 @@ behavior, or noticed something else. "The guy is looking back" as a moment annot Raw recording data is large and the arithmetic is mechanical, so keep both out of the main conversation. In Claude Code, hand every quantitative question to the `recording-analyst` subagent that ships with this plugin (`Agent` with `subagent_type: recording-analyst`), passing the project -path, the reference or recording, and the question verbatim. It can only run `polyspatial_*` commands -and compute on their output, so what comes back is a measurement with frames and seconds, and you -decide afterwards whether the *why* needs the code. In Codex there is no plugin subagent: copy +path, the reference or recording, and the question verbatim. It only runs the annotation commands +and read-only scene-state queries and computes on their output, so what comes back is a measurement +with frames and seconds, and you decide afterwards whether the *why* needs the code. In Codex there +is no plugin subagent: copy [references/codex-recording-analyst.toml](references/codex-recording-analyst.toml) into the project's `.codex/agents/` and delegate the same way, or follow the rules below yourself. @@ -82,15 +87,27 @@ entity — and say plainly which part of your answer is measured and which is in Do not spend the session grepping `.unity` files while the measured answer sits unreported. Worked example, "how many times did the cupcake rotate?". The recording gives you samples; -the arithmetic is yours — dump every frame to a file and compute in a short script, exactly as +the arithmetic is yours — write the keyframes to a file and compute in a short script, exactly as you would with any dataset: ```bash -unity command polyspatial_annotation_show --ref "#" # entityPath: …/Cupcake/Wiggle/MMCupcake/Cup -unity command polyspatial_entity_timeline --recording --entity "MMCupcake/Cup" --property rotation --step 1 --json > cup.json -unity command polyspatial_entity_timeline --recording --entity "Wiggle/MMCupcake" --property rotation --step 1 --json > mmcupcake.json -# python: for consecutive quaternions q0,q1 take delta = q1 * inverse(q0), convert to angle-axis, -# accumulate signed angle about the dominant axis; sum / 360 = turns; runs of |delta| > 0 = bursts. +unity command polyspatial_annotation_show --ref "#" # entityId: 4711, entityPath: …/Cupcake/Wiggle/MMCupcake/Cup, recordingPath +cat > /tmp/cup.cs <<'CS' +var state = UnityEditor.PolySpatial.Serialization.SceneState.PolySpatialSceneStateRecordingLoader.Load(""); +string Rotation(long id) => state.Query() + .FilterBySubtree(id) + .FilterProperties(include: new[] { "transform.rotation" }) + .LocalTransforms() + .IncludeComponents(false).IncludeAssets(false).IncludeInputs(false) + .ToJson(); +System.IO.File.WriteAllText("Temp/cup.ndjson", Rotation(4711)); // the annotated Cup +System.IO.File.WriteAllText("Temp/mmcupcake.ndjson", Rotation()); // its parent, from the Cup's path in the summary +return $"frames={state.TotalFrameCount}"; +CS +unity command eval_file --file /tmp/cup.cs --timeout 120 +# python over Temp/*.ndjson: transform.rotation is [[frame,[x,y,z,w]],...]; for consecutive quaternions +# q0,q1 take delta = q1 * inverse(q0), convert to angle-axis, accumulate signed angle about the dominant +# axis; sum / 360 = turns; runs of |delta| > 0 = bursts. ``` Result on the sample recording: `Cup` never turns relative to its parent (one distinct rotation @@ -99,23 +116,23 @@ in 779 frames); `MMCupcake` turns 13.9 times about x in two bursts, frames 336 ~7-turn bursts at 7.4–8.4 s and 8.5–9.5 s; the rotation is on the parent MMCupcake, the annotated Cup mesh is rigid." Then, only if asked why: the MMF_Rotation feedback on that object. -The same pattern answers "how many times did the color change" (`polyspatial_scene_state ---entity X --start-frame 1 --end-frame N --properties Image.color` and count that keyframe array), -"how far did the player travel" (sum `worldPosition` deltas), "was it ever inactive" (the -`lifecycle` keyframes). There is no per-question command; there are samples and your computation. -When a slice is more than a screen of text, `polyspatial_scene_export --out Temp/x.ndjson ...` -writes it to a file: compute on the file, print only the result. +The same pattern answers "how many times did the color change" (`FilterProperties(include: new[] { +"Image.color" })` and count that keyframe array), "how far did the player travel" (`WorldTransforms()` +on `transform.position`, sum the deltas), "was it ever inactive" (the `lifecycle` keyframes). There +is no per-question command; there are keyframes and your computation. Write anything longer than a +screen to `Temp/` and compute on the file, print only the result. Rules for measuring: - A property that is constant on the annotated entity usually lives on an ancestor: walk up the - `entityPath` one level at a time. `worldRotation`/`worldPosition` give the composed result. + `entityPath` one level at a time. `WorldTransforms()` gives the composed result, `LocalTransforms()` + the value relative to the parent. - Net rotation and total rotation differ: a wiggle travels many degrees and nets zero. Report the one the question asks for, and say which. -- `--step 1` gives every frame; the default samples ~200 points, which is fine for a curve but - not for counting. -- Convert frames to seconds with the `time` fields of `polyspatial_annotation_list` or `--start-time` - on `polyspatial_scene_changes`; quote both in the answer. +- Keyframes are stored only where the value changed; a value constant over the range is written + bare. Hold the last value across frames when you need per-frame samples. +- Convert frames to seconds with the `time` field of the annotation, or query by seconds with + `FilterByTimeRange(s0, s1)`; quote both in the answer. To see the moment, replay and park on the frame, then capture: @@ -149,13 +166,18 @@ unity command eval --code "return $R.LiveFrame;" # note the frame again: the re # 3. Stop and wait for the file. unity command editor_stop # the .qrec finalizes on exit -unity command polyspatial_recording_metadata --recording # poll until it answers; frameCount - -# 4. Ask the recording. -unity command polyspatial_scene_state --recording --summarize true # what exists -unity command polyspatial_scene_changes --recording --start-frame A --end-frame B --entity "Player" -unity command polyspatial_entity_timeline --recording --entity "Player" --property worldPosition --start-frame A --end-frame B -unity command polyspatial_scene_state --recording --entity "Enemy/HealthBar" --start-frame B --end-frame B +L=UnityEditor.PolySpatial.Serialization.SceneState.PolySpatialSceneStateRecordingLoader +unity command eval --code "return $L.Load(\"\").TotalFrameCount;" --timeout 120 # poll until it answers + +# 4. Ask the recording (one script, several questions; see references/commands.md). +cat > /tmp/verify.cs <<'CS' +var state = UnityEditor.PolySpatial.Serialization.SceneState.PolySpatialSceneStateRecordingLoader.Load(""); +System.IO.File.WriteAllText("Temp/summary.ndjson", state.Query().Summarize().OrderAlphabetically().ToJson()); // what exists, with instanceIds +System.IO.File.WriteAllText("Temp/player.ndjson", state.Query().FilterBySubtree().FilterByFrameRange(, ).WorldTransforms().ToJson()); +System.IO.File.WriteAllText("Temp/changes.ndjson", state.Query().Diff(, ).WithOutputMode(Unity.PolySpatial.Serialization.SceneState.OutputMode.PropertyPerLine).ToJson()); +return $"frames={state.TotalFrameCount}"; +CS +unity command eval_file --file /tmp/verify.cs --timeout 120 ``` Rules of evidence: @@ -163,14 +185,11 @@ Rules of evidence: - Quote frames and values from the recording in your answer ("frame 1210–1290: `world.position.y` rose from 0.00 to 2.31, then fell back by frame 1350"), and name the `.qrec` path so a person can replay it. A screenshot alone is not proof. -- Entity names repeat (a UI scene has hundreds of `Text`). When a command says the name is - ambiguous, pass a hierarchy path suffix such as `Button - Scale/Text`. -- `polyspatial_entity_timeline --property` takes `position`, `worldPosition`, `rotation`, - `worldRotation`, `scale` (the `world.rotation` spelling from scene_state output also works); - frames are 1-based. +- Entity names repeat (a UI scene has hundreds of `Text`). Resolve by `path` in the summary and + query by `instanceId`, never by name alone. - `LiveFrame` while recording is the recording frame counter; `Time.frameCount` in `simulate_input_script_status` is the game's counter. Bracket with status before and after - driving, or convert with `--start-time/--end-time` on `polyspatial_scene_changes`. + driving, or query by seconds with `FilterByTimeRange`. - The Editor throttles when unfocused: expect frame rates that differ from a focused run, and use seconds, not frame counts, when timing input. - Leave Play mode with `editor_stop`; the scene that was open @@ -179,15 +198,29 @@ Rules of evidence: ## 3. Inspect an existing recording without an annotation ```bash -unity command polyspatial_recording_list -unity command polyspatial_recording_metadata --recording latest -unity command polyspatial_scene_state --recording latest --summarize true -unity command polyspatial_scene_changes --recording latest --start-time 0 --end-time 5 --group-depth 2 --include-components false +ls -t Library/PolySpatialRecordings/*.qrec | head -3 +cat > /tmp/inspect.cs <<'CS' +var state = UnityEditor.PolySpatial.Serialization.SceneState.PolySpatialSceneStateRecordingLoader.Load(""); +System.IO.File.WriteAllText("Temp/summary.ndjson", state.Query().Summarize().OrderAlphabetically().ToJson()); +System.IO.File.WriteAllText("Temp/first5s.ndjson", state.Query().FilterByTimeRange(0, 5).FilterByDepth(2).IncludeComponents(false).ToJson()); +return $"frames={state.TotalFrameCount}"; +CS +unity command eval_file --file /tmp/inspect.cs --timeout 120 ``` -Start wide (`--summarize`, `--group-depth 2`) and narrow to one entity and a short frame range; +Start wide (`Summarize()`, `FilterByDepth(2)`) and narrow to one subtree and a short frame range; whole-scene keyframe dumps run to hundreds of kilobytes. +## 4. Query cheat sheet + +`state.Query()` returns a `SceneStateQuery`; every call chains and `ToJson()` ends it with NDJSON. +`FilterBySubtree(instanceId)`, `FilterByFrameRange(a, b)`, `FilterByTimeRange(s0, s1)`, +`FilterByDepth(n)`, `FilterByComponentType("MeshRenderer")`, `FilterProperties(include, exclude)`, +`Summarize()`, `Diff(a, b)`, `WorldTransforms()` / `LocalTransforms()`, `IncludeComponents(false)`, +`IncludeAssets(false)`, `IncludeInputs(false)`, `WithSignificantTransformDigits(5)`, +`WithOutputMode(OutputMode.PropertyPerLine)`. Loading takes seconds on a long recording: one +script per recording, several questions inside it. + ## Gotchas - Entering or leaving Play mode reloads the domain: `unity command` may fail to connect for a few @@ -195,6 +228,8 @@ whole-scene keyframe dumps run to hundreds of kilobytes. - `StartRecording` returns `Error: ...` when already in Play mode or when the scene is untitled; `StartPlaybackAt` refuses a scene with unsaved changes. Save first. Recording and playback have no `polyspatial_*` commands of their own: drive them through `eval` as shown above. +- `eval` has no `using`: spell out `UnityEditor.PolySpatial.Serialization.SceneState.…` and + `Unity.PolySpatial.Serialization.SceneState.…`. The types are public from polyspatial #4964. - Input screen coordinates are Game view pixels with the origin bottom-left; `capture_game_view` reports the size it rendered at. - Audio needs `UnityEngine.AudioSource` in PolySpatial Settings ▸ Generic Tracking Excluded Types diff --git a/skills/polyspatial-playtest/references/codex-recording-analyst.toml b/skills/polyspatial-playtest/references/codex-recording-analyst.toml index 8e82842..2ad7dc3 100644 --- a/skills/polyspatial-playtest/references/codex-recording-analyst.toml +++ b/skills/polyspatial-playtest/references/codex-recording-analyst.toml @@ -10,15 +10,16 @@ You measure. You are handed a Unity project path, a recording or an annotation r Only use, always with --project-path : unity command polyspatial_annotation_show --ref "" [--window N] - unity command polyspatial_annotation_list | polyspatial_recording_list | polyspatial_recording_metadata - unity command polyspatial_scene_state --recording R --entity "" --start-frame A --end-frame B [--properties p1,p2] [--summarize true] - unity command polyspatial_scene_changes --recording R --start-frame A --end-frame B [--entity X] [--properties ...] [--group-depth 2] - unity command polyspatial_entity_timeline --recording R --entity X --property position|worldPosition|rotation|worldRotation|scale --step 1 - unity command polyspatial_scene_export --out Temp/.ndjson ... - python3 on files you exported or redirected. + unity command polyspatial_annotation_list + unity command eval_file --file --timeout 120, where the script loads the recording with + UnityEditor.PolySpatial.Serialization.SceneState.PolySpatialSceneStateRecordingLoader.Load(path) and runs a + read-only state.Query() chain (Summarize() to resolve names to instanceId; FilterBySubtree(id), + FilterByFrameRange(a, b), FilterProperties(include: ...), WorldTransforms()/LocalTransforms(), Diff(a, b), + ToJson()), writing NDJSON to Temp/.ndjson. Template: the polyspatial-playtest skill, references/commands.md. + python3 on the files you wrote. Never read Assets/, Packages/, ProjectSettings/, .unity, .prefab or .cs files; never grep the project; never editor_play, simulate_*, capture_game_view. -Method: resolve the reference; pick the entity and property (constant on the annotated entity usually means it lives on an ancestor, walk up the path); pull every frame with --step 1 or --properties, redirect to a file, compute (rotation: quaternion deltas to angle-axis, net vs total turns, bursts; distance: sum of position deltas; counts: keyframe array length minus one; presence: lifecycle keyframes); convert frames to seconds. +Method: resolve the reference; pick the entity and property (constant on the annotated entity usually means it lives on an ancestor, walk up the path); pull the keyframes with FilterProperties, hold the last value across frames when you need per-frame samples, write to a file, compute (rotation: quaternion deltas to angle-axis, net vs total turns, bursts; distance: sum of position deltas; counts: keyframe array length minus one; presence: lifecycle keyframes); convert frames to seconds. Return only: Answer (numbers, unit, frame range, seconds); Where it lives (entity path, property); Evidence (2-5 raw lines); Commands run; Not measurable (what the recording cannot show). """ diff --git a/skills/polyspatial-playtest/references/commands.md b/skills/polyspatial-playtest/references/commands.md index cfcae9d..cd5a389 100644 --- a/skills/polyspatial-playtest/references/commands.md +++ b/skills/polyspatial-playtest/references/commands.md @@ -10,7 +10,7 @@ back in the `result` field; several are NDJSON (one JSON object per line) so the | `polyspatial_annotation_list` | `--recording all\|latest\|` (default all) | One line per annotation: `reference`, `recording`, `recordingPath`, `frame`, `frameEnd`, `time`, `kind` (`entity`\|`moment`), `text`, `created`, `createdBy`, and for entity annotations `entityId`, `entityName`, `entityPath`, `worldPosition`, `worldBoundsCenter`, `worldBoundsSize`, `hitPoint`; `camera` is the Scene view camera when it was written. | | `polyspatial_annotation_show` | `--ref # \| \| ` (required), `--window 30`, `--include-components true`, `--max-changes 300` | `annotation` (as above), `recordingFrames`, `changeWindow {from,to}`, `state` (entity annotations: NDJSON lines of the subtree at the frame, parsed into an array), `changes` (entity) or `changedEntities` (moment), `changeCount`, `changesTruncated`, `entityResolvedByName`, `entityMissing`. | -`kind` is `entity`, `region` (a circle drawn in the Scene view: `entityIds`, `entityPaths`, `entityCount`; show returns `members`, their `state` and only their `changes`), `span` (`frame`–`frameEnd`) or `moment`. +`kind` is `entity`, `entities` (a box or lasso selection in the Scene view: `entityIds`, `entityPaths`, `entityCount`; show returns `members`, their `state` and only their `changes`), `span` (`frame`–`frameEnd`) or `moment`. A `changes` entry: `{ entity, component?, property, from, to, firstChangeFrame, lastChangeFrame, keyframes }`. A `changedEntities` entry: `{ entity, changedProperties, properties[], firstChangeFrame, lastChangeFrame }`. @@ -23,7 +23,7 @@ No `polyspatial_*` command enters Play mode. Call the public `UnityEditor.PolySp |---|---| | `return R.StartRecording();` | The new `.qrec` path, or `Error: ...` (already in Play mode, untitled scene). Enters Play mode. | | `return $"{R.IsLiveSession} {R.LiveFrame}";` | `True ` once the recorder runs; `LiveFrame` is the recording frame counter. | -| `unity command editor_stop` | Leaves Play mode; the file finalizes. Poll `polyspatial_recording_metadata` for it. | +| `unity command editor_stop` | Leaves Play mode; the file finalizes. Poll `Load(path).TotalFrameCount` through `eval` for it. | | `return R.StartPlaybackAt("", , true);` | Rebuilds `` on a timeline inside the open scene, parked on ``; `null` on success. Never enters Play mode; the scene's own objects are deactivated until `StopPlayback`. | | `R.SeekTo(, true); return R.CurrentFrame;` | Rebuilds that frame directly, in either direction. | | `R.IsPaused = false;` / `R.IsPaused = true;` | Plays in real time / pauses. | @@ -32,22 +32,44 @@ No `polyspatial_*` command enters Play mode. Call the public `UnityEditor.PolySp `R` stands for the full `UnityEditor.PolySpatial.Utilities.RecordingPlaybackScene`; `eval` has no `using`, so spell it out. -| Command | Flags | Result | -|---|---|---| -| `polyspatial_recording_list` | | One line per `.qrec`: `path`, `name`, `sizeKB`, `lastWriteUtc`. | -| `polyspatial_recording_metadata` | `--recording` | `{ path, name, version, frameCount, recordingType, commandCount }`. | +## Reading a recording (through `eval_file`) -## Recording queries +Recordings are `Library/PolySpatialRecordings/*.qrec`; `ls -t` finds the newest. Load one into a +`PolySpatialSceneState` and shape the answer with `SceneStateQuery`, in a `.cs` script run by +`unity command eval_file --file