PICO MCP Extensions

PICO XR feature construction APIs for Unity MCP agents. Idempotent, non-destructive, ambiguity-aware.

Overview

This Unity package exposes PICO XR building blocks as MCP (Model Context Protocol) tools, enabling AI agents (e.g. Unity AI Assistant) to programmatically configure XR scenes for PICO devices. It is designed to be used together with the PICO Unity Integration SDK 6.1.x and aligns its dependency baseline with that SDK (XR Interaction Toolkit 3.x).

Package name: com.bytedance.pico.mcp-extensions
Version: 0.1.2
Unity: 6000.0+
Author: ByteDance PICO

Requirements

Dependency	Minimum Version
com.unity.xr.core-utils	2.3.0
com.unity.xr.interaction.toolkit	3.4.0
com.unity.inputsystem	1.18.0
Unity AI Assistant (Unity.AI.MCP.Editor)	2.x
com.bytedance.pico.xr	6.1.0

Features

Eight XR building blocks:

Block	Description
VST	Video See-Through (passthrough) - configures camera for transparent background and adds PXR_CameraEffectBlock
Controller	Mounts PICO controller visual models on Left/Right hand anchors
Controller Haptics	Adds one XR-Origin haptics manager with named impulse, AudioClip-buffer, and PHF-buffer effects; gameplay code or UnityEvents choose when to invoke them
Locomotion	Enables XRI locomotion subtree with fine-grained presets (Move, Turn, Teleportation, GrabMove, Climb, Gravity, Jump)
Spatial Mesh	Configures PXR_SpatialMeshManager with auto-detected MeshPrefab (depends on VST)
Plane Detection	Configures PICO SensePack plane detection via a bundled PXR_PlaneDetectionManager driver (depends on VST; PICO-native runtime only)
Hand	Enables PICO hand tracking (virtual hands) plus an XRI hand-interactor rig so a pinch can drive grab
Grab	Object pick-up & drag; ensures the scene XRInteractionManager broker and can upgrade a target object into a grabbable

Additionally, a Package tool manages Unity packages and samples (install / remove / update / import samples / query resolvable version). Mutating package results expose data.changed; removing a package that is not installed reports changed=false and does not claim that a removal occurred.

Runtime support

Most building blocks compile and configure correctly under both PICO XR runtimes:

• PICO-native runtime (ENABLE_PICO_XR_SDK)
• OpenXR runtime (ENABLE_PICO_OPENXR_SDK) — VST enables the PICO PassthroughFeature; Spatial Mesh forces MultiPass rendering and enables the PICOSpatialMesh feature; Hand enables the Unity XR Hands models plus the PICO hand-tracking / hand-interaction OpenXR features.

Plane Detection is PICO-native only. PICO ships no plane-detection OpenXR feature, so under the OpenXR runtime the plane provider is never created and the block is a no-op. Controller Haptics is PICO-native only. Its runtime manager uses the current PXR_Input.SendHapticImpulse, SendHapticBuffer, and buffer-lifecycle APIs; the package still compiles under OpenXR, but haptics attach/upsert actions report unsupported.

Architecture

Editor/
  PXR_MCP_Common.cs        # Shared helpers: XR Origin lifecycle, module visibility
  PXR_MCP_Features.cs      # Building block implementations (VST, Controller, Locomotion, SpatialMesh, Plane, Hand, Grab)
  PXR_MCP_Haptics.cs       # Editor lifecycle for the XR-Origin haptics manager and named effects
  PXR_MCP_PackageOps.cs    # Package Manager operations (add, remove, samples)
  Tools/
    PXR_MCP_Tools.cs       # MCP tool surface ([McpTool] entry points)
    PXR_MCP_Result.cs      # Uniform result envelope for LLM consumption
Runtime/
  PXR_MCP_HapticsManager.cs # Runtime bridge to current PICO impulse and buffer APIs

Layer 1 (Editor): Plain C# static methods for building-block operations. Layer 2 (Tools): [McpTool]-annotated methods that wrap Layer 1 and return PXR_MCP_Result envelopes. The controller-haptics block additionally ships a runtime manager so gameplay code and UnityEvents can invoke named effects in a player build.

MCP Tools

Tool	Actions	Description
pico_xr_vst	Enable, Disable, Status	Manage Video See-Through
pico_xr_controller	Enable, Disable, Status	Manage PICO controller models
pico_xr_play_to_pico	Status, Start, Stop	Select an existing online ADB device, prepare SpatialAdapterServer, then enter Play Mode over wired or wireless transport
pico_xr_haptics	Attach, UpsertEffect, RemoveEffect, Remove, Status	Manage one PICO-native haptics manager and its named effects without choosing gameplay triggers
pico_xr_locomotion	Enable, Disable, Configure, Status	Manage locomotion with preset flags
pico_xr_spatial_mesh	Enable, Disable, Status	Manage spatial mesh (requires VST)
pico_xr_plane	Enable, Disable, Status	Manage PICO plane detection (requires VST; PICO-native runtime only)
pico_xr_hand	Enable, Disable, Status	Manage PICO hand tracking (virtual hands)
pico_xr_grab	Enable, Disable, Status, MakeGrabbable	Manage grab pick-up & drag; make_grabbable upgrades a target object
pico_xr_package	List, Info, Add, Remove, Update, ListSamples, ImportSample, Resolvable	Unity Package Manager operations; resolvable is a read-only query for the latest-compatible version
pico_xr_status	(none)	Aggregate snapshot of all blocks

All pico_xr_* tools are enabled by default on a fresh Unity AI Assistant configuration. An explicit user override in Edit > Project Settings > AI > Unity MCP still takes precedence.

Controller haptics

pico_xr_haptics attaches one PXR_MCP_HapticsManager directly to the existing agent XR Origin. It does not require Controller GameObjects and never creates an XR Origin by itself. The manager stores multiple named effects; every effect has one source type and may target left, right, or both controllers. A both effect uses independent left/right parameters, assets, and runtime source IDs. Use two named effects when the two hands need different source types.

Supported serialized effect types use current PICO APIs:

PXR_Input.SendHapticImpulse(vibrateType, amplitude, durationMs, frequencyHz);
PXR_Input.SendHapticBuffer(vibrateType, audioClip, channelFlip, ref sourceId, cacheType);
PXR_Input.SendHapticBuffer(vibrateType, phfText, channelFlip, amplitudeScale, ref sourceId);

Raw PCM is a code-level entry point because arbitrary float[] data is not a useful Inspector field. The manager exposes UnityEvent-friendly Send, StartEffect, Pause, Resume, UpdateEffect, Stop, StopAndClear, StopAll, and amplitude methods, plus Try* counterparts for code that needs a success value. Serialized callbacks expose requested, sent, source-created, started, paused, resumed, updated, stopped, and failed states. The SDK provides no reliable completion callback, so this component does not claim an OnCompleted event.

The parameterless cached-start and update methods are named StartEffect() and UpdateEffect() so Unity does not invoke them automatically as MonoBehaviour Start / Update lifecycle messages. Named StartEffect(string) and UpdateEffect(string) overloads remain available for UnityEvents and business code.

The manager deliberately does not subscribe to XR Interaction Toolkit or other gameplay events. Application code or a user-selected UnityEvent owns all trigger timing. Deprecated StartVibrateBy*, old Haptic Stream APIs, Unity XR/OpenXR substitutes, and the separate advanced parametric API are not used.

Controller haptics currently require the PICO-native runtime (ENABLE_PICO_XR_SDK). OpenXR projects continue to compile, but attach and configure report unsupported; status and cleanup remain available. Editor execution cannot prove physical vibration; validate the final behavior on a connected PICO device.

Design Principles

• Idempotent: Re-running any operation with the same arguments is a safe no-op.
• Non-destructive: Never destroys or deactivates foreign (non-agent-owned) XR Origins.
• Module isolation: Enabling one block does not implicitly enable unrelated modules. Initial-create hides Controller and Locomotion so VST stays clean.
• No hardcoded versions: XRI paths and types are resolved dynamically via PackageInfo and reflection.
• Undo-safe: All scene modifications go through Unity's Undo system.

Installation

Add this package to your Unity project via the Package Manager:

1. Open Window > Package Manager
2. Click + > Add package from disk... (or add to Packages/manifest.json)
3. Ensure XRI Starter Assets sample is imported (required for XR Origin prefab)

License

Copyright (c) 2015-2022 PICO Technology Co., Ltd. All rights reserved. See LICENSE.md for details.

SpatialAdapter Play-to-PICO

Use the SpatialAdapter branch of pico-unity-buildingblocks. Call status for a read-only gate, then start for an authorized preview. The default connection is wired: PICO physical devices and Emulators use selected-device ADB forwarding to 127.0.0.1:27182, with no LAN check. Explicit wireless uses the PICO device's LAN IPv4 and requires both real subnet prefixes and a direct route without NAT. A network mismatch is an error and never silently falls back to wired.

start first reads adb devices -l and never installs or starts an Emulator. If no device is online, it returns no_adb_device before starting an operation. If multiple devices are online and deviceSerial is absent, it returns stage=device_selection_required with data.devices; ask the user which device to use and retry start with that serial. A single online device is selected automatically. The selected device is then verified as a PICO/SpaceOS device with Android boot completed before any Server installation. Physical devices and Emulators use the same Server, transport, and Play Mode flow; QEMU properties are not a prerequisite. Non-PICO devices return device_not_pico before Server or transport preparation.

start returns pending with data.operationId; query status using that ID. The Host process continues Server and transport preparation independently and the Editor continues to Play Mode after readiness. State and logs are stored in Library/PICO-MCP/PlayToPico, outside project assets. Repeated starts reuse the active operation. stop cancels preparation or exits this session; it removes only this operation's forwarding and restores temporary export assets and SDK Play callbacks. It does not uninstall Server or manage Emulator lifecycle.

Optional parameters are deviceSerial, hostIp (wireless), nodePath, and operationId (status/stop). Node.js 20+ is required for the Host workflow. User-defined paths are passed as process arguments, never shell code.

The extension has no compile-time SpatialAdapter dependency and does not patch SDK source. Runtime compatibility is checked before starting: the installed SDK must provide PlayToDeviceExporter.OnPlayModeChanged, export/restore methods on AssetExportManager, and native SpatialAdapterRuntime.ConnectDevice. Current 6.1.x SDKs expose internal methods; a compatibility bridge temporarily unsubscribes the SDK's automatic Play callback during this operation, exports and connects exactly once, and restores it afterward and across domain reload. Unsupported signatures return sdk_bridge_unsupported before device mutation. The SDK still supplies scene export, native libraries, and resource/frame sync.

stage=running, connectionEstablished=true, and editorPlaying=true confirm native connection and Editor Play Mode. They do not prove visual correctness; validate rendered content and interaction on the selected PICO device separately. Server startup and install errors include a log path. Host transport regression tests run with:

node --test Tools~/play-to-pico/host.test.mjs

Unity Editor tests include tool registration and optional installed-SDK bridge checks. Windows and macOS route behavior also requires platform validation.