Skip to content

mac/immersive: show the game in 3D on a Vision Pro through a RemoteImmersiveSpace - #2737

Open
Pareses967 wants to merge 12 commits into
seedds:mainfrom
Pareses967:remoteimmersivespace-support
Open

Pareses967 wants to merge 12 commits into
seedds:mainfrom
Pareses967:remoteimmersivespace-support

Conversation

@Pareses967

@Pareses967 Pareses967 commented Nov 14, 2026 •

Copy link
Copy Markdown

What and why

Shows the game in 3D on an Apple Vision Pro near the Mac, through a RemoteImmersiveSpace the Mac draws. Off by default: DM_IMMERSIVE=1, or Apple Vision Pro in the Mac settings window. The Mac's window goes on showing the game and pacing it.

What it does

  • The scene (dm/host/immersive_mac.swift, the host's first Swift source): the space is added to the AppKit app through NSHostingSceneRepresentation. Its layer and the headset go to immersive_mac.m, which draws on a thread of its own at the headset's rate, with the head's pose from ARKit, for any of the three layer layouts. The screen is 3.2 m wide, 2.5 m ahead (DM_IMMERSIVE_SCREEN). Hold the Digital Crown to recenter the screen (confirmed on a physical Vision Pro).
  • The stereo tap (dm/gfx/nvn/nvn_mtl.m, mtl_stereo.h): while it is on, each present's picture and the frame's scene depth are copied into host rings in queue order. The scene depth is the visible scene's: among depth/colour pairs with depth-writing draws (a shadow map's pass has no colour), a pair whose viewport has the picture's shape wins, then the one whose colour reaches the presented texture in the fewest draws or copies, then the latest with a fair share of draws; with no known path, the most draws. A scene depth that a later pass clears (a UI pass) is copied before the clear. The draws' viewports give the part of it the scene covers, and their compare gives the far plane's side. No profile key.
  • The pass (dm/gfx/msl/stereo.metal): a depth value is affine in 1 / distance, so the parallax needs no constant of the game. The screen's plane follows the nearest things in the scene, and each eye's source pixel is searched along the row so near surfaces cover far ones. DM_IMMERSIVE_3D scales the depth (0 is flat).
  • The HUD: a HUD is drawn without the depth test, so it would bend with the scene behind it. Before the first overlay into a window texture, the pass is split and the texture copied: the picture before its overlays. On the headset, pixels where the two differ are drawn whole in front of the scene, and the scene's pixels come from the copy. When overlays cover more than 60 % of the picture (a fade, a vignette, a menu), nothing is drawn apart.

DM_IMMERSIVE_SHOT=<present>:<path.png> writes both eyes of a present side by side without a headset. Internals: docs/development.md, "The game on a Vision Pro"; the player's page: docs/settings.md, "Apple Vision Pro".

Checked

  • Tests: dm/gfx/tests/t_stereo (new, 52 checks under Metal validation): the scene depth is chosen over a decoy and a shadow map; both far-plane sides; the parallax of far, middle and near markers; pinned slots are left alone; a 2D frame stays flat; a HUD bar drawn over a window texture through the replay stands in front of the scene with no copy of itself left in the scene. With the HUD handling off, those last checks fail. t_present_alloc, t_present_crop, t_nvn, t_clear_ds and t_present_rescue pass unchanged; dm/tools/tests passes (1,619 tests); all six documentation tests pass. clang-tidy was unavailable.
  • Games run, on the Mac with DM_IMMERSIVE=1 and an Apple Vision Pro:
    • Dredge (01008CD0172D6000, v1.5.3): renders in correct 3D.
    • What the Golf (01002C000E35E000, v20.5.2): renders flat.
    • Untitled Goose Game (010082400BCC6000), Mac captures without a headset: the tap now takes the scene's depth instead of a normals prepass, including when the scene's depth is cleared for the UI before the present. Goose's scene reference correlates 0.995 with the picture inverted versus 0.09 upright, so alignment flips it. Dredge still takes the same scene depth. t_stereo covers the atlas, prepass, tie and clear cases (73 checks). Headset retest pending.
  • Devices and OS: Mac (Apple M4 Pro, macOS 27.2.1, Xcode 27.2) with an Apple Vision Pro. With the setting off, the game runs normally on the Mac.
  • scripts/regress.py check on the two local games with Mac builds: PASS on both (9223/9223 and 3900/3900 shaders, 0 newly failing, 0 changed MSL, no interface change); baseline bee218d97844, candidate ba6c8e3f754c. regress.py impact lists the change as reaching every NVN game's presents (present:*), so it does not pick between games. Neither game has a smoke route, so this checks their shader chains and pipeline interfaces, not a boot. Not rerun in the current workspace, which lacks configured game folders.
screenshot

Open

  • A HUD drawn into the scene's own target before the composite stays with the scene.
  • Vulkan and GL games give no depth and stay flat.
  • The Mac's display link still paces the game, so its window must stay visible.

Front ends

Front-ends: new: DM_IMMERSIVE and DM_IMMERSIVE_3D settings (Mac), in launcher.json

Docs

  • The finding is in its game note or its docs/reference/<subsystem>.md entry, and python3 dm/tools/docs_symptoms.py was run (the symptom index)
  • No game files, keys or data extracted from a game in the change or the attachments

Closes #2755

@github-actions github-actions Bot added host The Horizon host: SVCs, services, threads, memory (dm/host) nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) packing Building an IPA or a Mac app: the NSP script, extraction, the shader split, the packers mac The Mac front end and app (dm/mac, front_mac.m) tools Tools and scripts (dm/tools, scripts) docs Documentation labels Nov 14, 2026
@Pareses967
Pareses967 marked this pull request as ready for review November 14, 2026 22:35
@Pareses967
Pareses967 force-pushed the remoteimmersivespace-support branch 2 times, most recently from 1701d8c to 4cf068a Compare November 15, 2026 03:34
@Pareses967
Pareses967 force-pushed the remoteimmersivespace-support branch from 4cf068a to 174312c Compare November 15, 2026 06:53
@github-actions github-actions Bot added the game: 01006F8002326000 Animal Crossing: New Horizons label Nov 15, 2026
@Pareses967
Pareses967 force-pushed the remoteimmersivespace-support branch 2 times, most recently from 0d72307 to 3622bf6 Compare November 15, 2026 18:45
…mersiveSpace

DM_IMMERSIVE=1 (the Mac settings window's Apple Vision Pro) opens a RemoteImmersiveSpace
on a Vision Pro near the Mac and draws the game there on a screen 2.5 m in front of the
player, with a picture for each eye made from the game's own depth. The Mac's window goes on
showing the game and pacing it.

- The scene (dm/host/immersive_mac.swift, the host's first Swift source): the space added to
  the AppKit app through NSHostingSceneRepresentation; its layer and the headset go to
  immersive_mac.m, which draws on a thread of its own at the headset's rate with the head's
  pose from ARKit, for any of the three layer layouts.
- The stereo tap (dm/gfx/nvn/nvn_mtl.m, mtl_stereo.h): while on, each present's picture is
  drawn into a ring of host textures and the frame's scene depth copied into another, in
  queue order where the game queues the present. The scene depth is the depth attachment
  with the most depth-writing draws in passes that have a colour attachment; the draws'
  viewports give the part it covers and their compare the far plane's side. No profile key.
- The pass (dm/gfx/msl/stereo.metal): a depth value is affine in 1 / distance, so the
  parallax needs no constant of the game; the screen's plane follows the nearest things in
  the scene, and each eye's source pixel is searched along the row so near surfaces cover
  far ones. DM_IMMERSIVE_3D scales it, DM_IMMERSIVE_SCREEN sizes and places the screen,
  DM_IMMERSIVE_SHOT writes both eyes of a present as a PNG without a headset.

Checked: dm/gfx/tests/t_stereo (46 checks under Metal validation: the scene depth chosen
over a decoy and a shadow map, both far-plane sides, parallax of far, middle and near
markers, pinned slots left alone, a 2D frame flat); t_present_alloc, t_present_crop, t_nvn
and t_clear_ds unchanged; dm/tools/tests with no new failures against the parent. Not
checked: a headset, a real game, regress.py (this Mac's Python is 3.10). Open: a HUD takes
the depth of the scene behind it; Vulkan and GL games give no depth and stay flat.

Front-ends: new: DM_IMMERSIVE and DM_IMMERSIVE_3D settings (Mac), in launcher.json
Keep HUD overlays in front of the scene, map the immersive scene files for regression impact analysis, and record physical Vision Pro testing. Remove the redundant Recenter Screen menu and shortcut; holding the Digital Crown recenters the screen.

Validation for the recenter removal: six documentation tests passed, both changed Objective-C files passed compilation syntax checks, and git diff --check passed. clang-tidy was unavailable; gameplay regression checks require configured game folders.
Match a luminance reference from the scene colour attachment against the final picture to recover depth orientation after postprocessing. Compare RGB when identifying HUD overlays so invisible alpha changes do not split geometry. Include stereo coverage and rendering reference entries.
The stereo tap took the depth/colour pair with the most depth-writing draws and
copied it at the present. A square depth atlas with colour, or a normals prepass,
can out-draw the scene: the headset then stretched atlas tiles over the picture
as horizontal bands, or warped the scene with the wrong pass. A scene depth that
a UI pass clears before the present was copied blank.

Rank pairs by the presented picture's shape, then by how few draws or copies
carry their colour into the presented texture (tracked from fragment texture
bindings and texture copies, reset by clears), then the latest pair with a fair
share of draws; the most-draws rule remains the fallback. Copy a candidate's
depth into a reserved slot before a pass clears it, and use that copy when its
epoch is the larger one. t_stereo covers the atlas, prepass, tie and clear cases.
The stereo tap told a HUD from the scene by copying the window texture before
its first overlay. An engine that tone maps into a texture of the window's
size, draws its HUD layouts there, then copies it into the window with one draw
leaves nothing over the window itself: the headset bent the HUD with the scene.

The draw that lays the window's picture down and samples one texture of its
size marks that texture as the composite source (st_feed). The next frame
watches it for overlays and gives its base to the window, drawn through the
composite's sRGB view when it had one so untouched pixels match exactly. The
present pass keeps a pipeline per target format. t_stereo covers both frames.
Main removed the cross-submit validity ledger and with it the list of window
textures (xv_present), which the stereo tap uses to find overlays. The tap now
keeps its own: the textures a window is made with and every presented texture
(mtl_window_tex).
@Pareses967
Pareses967 force-pushed the remoteimmersivespace-support branch from cc11c06 to 64735c0 Compare November 15, 2026 19:37
The composite source a frame's HUD is told from (st_feed) may be a packaged
texture: take the texture an NvPkg holds. Unity's final blit also flips that
texture into the window, so its base was upside down against the picture:
every pixel read as an overlay, and the depth alignment, matching the base
against the scene's reference, took the wrong row order (depth upside down
on the headset).

align_depth now counts the 32x18 cells where the base equals the picture in
each orientation (a HUD differs in both, so it cannot tip the count), takes
the base upside down when that matches clearly more cells, and reports it in
bit 1 of the alignment word; the eye pass, the overlay test and the depth
grid sample the base so. Alignment is encoded before the depth grid, which
now reads the word. t_stereo covers a flipping composite (89 checks).
The depth alignment compared the scene's reference with the base (the
picture before overlays) when there was one. Since the base can come from
the texture composited into the window (st_feed), and Untitled Goose
Game's final blit flips that texture, the match picked the wrong row order
and the headset showed its depth upside down. Compare with the presented
picture instead: a HUD over part of it barely moves the correlation.
Untitled Goose Game composites its scene texture into the window flipped,
draws two edge-glow effects into it after the scene (depth test ALWAYS, no
write), then clears the scene's depth and draws its UI tested against it.
Its base, the picture before overlays, was upside down: the headset
mirrored edges into the scene (a ghost of the title sign's edge) and, with
every pixel differing, never told the UI apart. Turned upright, the base
was still taken before the glow, which then read as HUD and dithered model
edges.

- align_depth counts the grid cells where the base equals the picture in
  each orientation, takes the base upside down when that matches clearly
  more, and reports it in bit 1 of the alignment word; the eye pass, the
  overlay test and the depth grid sample the base so (reverts 53191ee).
- st_overlay: a draw that samples what was rendered from its target's
  scene this frame (st_post, on the st_flow graph) lays the picture down,
  and a depth test counts as the scene's only against a depth holding the
  scene (st_depth_scene: no clear waiting, a depth-writing draw since).

On Goose the upright base now differs from the picture by its UI alone
(0.4 % of pixels) and the UI text stands in front (12.3 px between the
eyes). t_stereo: 98 checks, a flipping composite and the effect-then-UI
sequence included.
tex_get returns the NvPkg of a packaged texture; st_feed sent it -width
and DREDGE stopped on an unrecognized selector at its first composite
with the stereo tap on. Take the texture the package holds, and nothing
that is neither.
…ure's

st_post took a draw for an effect on the picture only when it sampled what
was rendered from its own target's scene pass. With the goose on screen,
Untitled Goose Game draws an edge glow built from a separate full-size
prepass: it was still the first overlay, the base was copied before it, and
the headset dithered model edges between the scene's and the HUD's depth.
Accept what derives from any depth pass of the target's size this frame; a
smaller pass (a minimap's) stays an overlay's source.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs Documentation game: 01006F8002326000 Animal Crossing: New Horizons host The Horizon host: SVCs, services, threads, memory (dm/host) mac The Mac front end and app (dm/mac, front_mac.m) nvn NVN on Metal and its tests (dm/gfx/nvn, dm/gfx/tests) packing Building an IPA or a Mac app: the NSP script, extraction, the shader split, the packers tools Tools and scripts (dm/tools, scripts)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Show games in 3D on Apple Vision Pro from the Mac (RemoteImmersiveSpace)

1 participant