Repository navigation
mac/immersive: show the game in 3D on a Vision Pro through a RemoteImmersiveSpace - #2737
Open
Pareses967 wants to merge 12 commits into
Open
Pareses967 wants to merge 12 commits into
Pareses967 wants to merge 12 commits into
Conversation
Pareses967
marked this pull request as ready for review
November 14, 2026 22:35
Pareses967
force-pushed
the
remoteimmersivespace-support
branch
2 times, most recently
from
November 15, 2026 03:34
1701d8c to
4cf068a
Compare
Closed
Pareses967
force-pushed
the
remoteimmersivespace-support
branch
from
November 15, 2026 06:53
4cf068a to
174312c
Compare
Pareses967
force-pushed
the
remoteimmersivespace-support
branch
2 times, most recently
from
November 15, 2026 18:45
0d72307 to
3622bf6
Compare
…mersiveSpace DM_IMMERSIVE=1 (the Mac settings window's Apple Vision Pro) opens a RemoteImmersiveSpace on a Vision Pro near the Mac and draws the game there on a screen 2.5 m in front of the player, with a picture for each eye made from the game's own depth. The Mac's window goes on showing the game and pacing it. - The scene (dm/host/immersive_mac.swift, the host's first Swift source): the space added to the AppKit app through NSHostingSceneRepresentation; its layer and the headset go to immersive_mac.m, which draws on a thread of its own at the headset's rate with the head's pose from ARKit, for any of the three layer layouts. - The stereo tap (dm/gfx/nvn/nvn_mtl.m, mtl_stereo.h): while on, each present's picture is drawn into a ring of host textures and the frame's scene depth copied into another, in queue order where the game queues the present. The scene depth is the depth attachment with the most depth-writing draws in passes that have a colour attachment; the draws' viewports give the part it covers and their compare the far plane's side. No profile key. - The pass (dm/gfx/msl/stereo.metal): a depth value is affine in 1 / distance, so the parallax needs no constant of the game; the screen's plane follows the nearest things in the scene, and each eye's source pixel is searched along the row so near surfaces cover far ones. DM_IMMERSIVE_3D scales it, DM_IMMERSIVE_SCREEN sizes and places the screen, DM_IMMERSIVE_SHOT writes both eyes of a present as a PNG without a headset. Checked: dm/gfx/tests/t_stereo (46 checks under Metal validation: the scene depth chosen over a decoy and a shadow map, both far-plane sides, parallax of far, middle and near markers, pinned slots left alone, a 2D frame flat); t_present_alloc, t_present_crop, t_nvn and t_clear_ds unchanged; dm/tools/tests with no new failures against the parent. Not checked: a headset, a real game, regress.py (this Mac's Python is 3.10). Open: a HUD takes the depth of the scene behind it; Vulkan and GL games give no depth and stay flat. Front-ends: new: DM_IMMERSIVE and DM_IMMERSIVE_3D settings (Mac), in launcher.json
Keep HUD overlays in front of the scene, map the immersive scene files for regression impact analysis, and record physical Vision Pro testing. Remove the redundant Recenter Screen menu and shortcut; holding the Digital Crown recenters the screen. Validation for the recenter removal: six documentation tests passed, both changed Objective-C files passed compilation syntax checks, and git diff --check passed. clang-tidy was unavailable; gameplay regression checks require configured game folders.
Match a luminance reference from the scene colour attachment against the final picture to recover depth orientation after postprocessing. Compare RGB when identifying HUD overlays so invisible alpha changes do not split geometry. Include stereo coverage and rendering reference entries.
The stereo tap took the depth/colour pair with the most depth-writing draws and copied it at the present. A square depth atlas with colour, or a normals prepass, can out-draw the scene: the headset then stretched atlas tiles over the picture as horizontal bands, or warped the scene with the wrong pass. A scene depth that a UI pass clears before the present was copied blank. Rank pairs by the presented picture's shape, then by how few draws or copies carry their colour into the presented texture (tracked from fragment texture bindings and texture copies, reset by clears), then the latest pair with a fair share of draws; the most-draws rule remains the fallback. Copy a candidate's depth into a reserved slot before a pass clears it, and use that copy when its epoch is the larger one. t_stereo covers the atlas, prepass, tie and clear cases.
The stereo tap told a HUD from the scene by copying the window texture before its first overlay. An engine that tone maps into a texture of the window's size, draws its HUD layouts there, then copies it into the window with one draw leaves nothing over the window itself: the headset bent the HUD with the scene. The draw that lays the window's picture down and samples one texture of its size marks that texture as the composite source (st_feed). The next frame watches it for overlays and gives its base to the window, drawn through the composite's sRGB view when it had one so untouched pixels match exactly. The present pass keeps a pipeline per target format. t_stereo covers both frames.
Main removed the cross-submit validity ledger and with it the list of window textures (xv_present), which the stereo tap uses to find overlays. The tap now keeps its own: the textures a window is made with and every presented texture (mtl_window_tex).
Pareses967
force-pushed
the
remoteimmersivespace-support
branch
from
November 15, 2026 19:37
cc11c06 to
64735c0
Compare
The composite source a frame's HUD is told from (st_feed) may be a packaged texture: take the texture an NvPkg holds. Unity's final blit also flips that texture into the window, so its base was upside down against the picture: every pixel read as an overlay, and the depth alignment, matching the base against the scene's reference, took the wrong row order (depth upside down on the headset). align_depth now counts the 32x18 cells where the base equals the picture in each orientation (a HUD differs in both, so it cannot tip the count), takes the base upside down when that matches clearly more cells, and reports it in bit 1 of the alignment word; the eye pass, the overlay test and the depth grid sample the base so. Alignment is encoded before the depth grid, which now reads the word. t_stereo covers a flipping composite (89 checks).
This reverts commit 7fcda44.
The depth alignment compared the scene's reference with the base (the picture before overlays) when there was one. Since the base can come from the texture composited into the window (st_feed), and Untitled Goose Game's final blit flips that texture, the match picked the wrong row order and the headset showed its depth upside down. Compare with the presented picture instead: a HUD over part of it barely moves the correlation.
Untitled Goose Game composites its scene texture into the window flipped, draws two edge-glow effects into it after the scene (depth test ALWAYS, no write), then clears the scene's depth and draws its UI tested against it. Its base, the picture before overlays, was upside down: the headset mirrored edges into the scene (a ghost of the title sign's edge) and, with every pixel differing, never told the UI apart. Turned upright, the base was still taken before the glow, which then read as HUD and dithered model edges. - align_depth counts the grid cells where the base equals the picture in each orientation, takes the base upside down when that matches clearly more, and reports it in bit 1 of the alignment word; the eye pass, the overlay test and the depth grid sample the base so (reverts 53191ee). - st_overlay: a draw that samples what was rendered from its target's scene this frame (st_post, on the st_flow graph) lays the picture down, and a depth test counts as the scene's only against a depth holding the scene (st_depth_scene: no clear waiting, a depth-writing draw since). On Goose the upright base now differs from the picture by its UI alone (0.4 % of pixels) and the UI text stands in front (12.3 px between the eyes). t_stereo: 98 checks, a flipping composite and the effect-then-UI sequence included.
tex_get returns the NvPkg of a packaged texture; st_feed sent it -width and DREDGE stopped on an unrecognized selector at its first composite with the stereo tap on. Take the texture the package holds, and nothing that is neither.
…ure's st_post took a draw for an effect on the picture only when it sampled what was rendered from its own target's scene pass. With the goose on screen, Untitled Goose Game draws an edge glow built from a separate full-size prepass: it was still the first overlay, the base was copied before it, and the headset dithered model edges between the scene's and the HUD's depth. Accept what derives from any depth pass of the target's size this frame; a smaller pass (a minimap's) stays an overlay's source.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Shows the game in 3D on an Apple Vision Pro near the Mac, through a
RemoteImmersiveSpacethe Mac draws. Off by default:DM_IMMERSIVE=1, or Apple Vision Pro in the Mac settings window. The Mac's window goes on showing the game and pacing it.What it does
dm/host/immersive_mac.swift, the host's first Swift source): the space is added to the AppKit app throughNSHostingSceneRepresentation. Its layer and the headset go toimmersive_mac.m, which draws on a thread of its own at the headset's rate, with the head's pose from ARKit, for any of the three layer layouts. The screen is 3.2 m wide, 2.5 m ahead (DM_IMMERSIVE_SCREEN). Hold the Digital Crown to recenter the screen (confirmed on a physical Vision Pro).dm/gfx/nvn/nvn_mtl.m,mtl_stereo.h): while it is on, each present's picture and the frame's scene depth are copied into host rings in queue order. The scene depth is the visible scene's: among depth/colour pairs with depth-writing draws (a shadow map's pass has no colour), a pair whose viewport has the picture's shape wins, then the one whose colour reaches the presented texture in the fewest draws or copies, then the latest with a fair share of draws; with no known path, the most draws. A scene depth that a later pass clears (a UI pass) is copied before the clear. The draws' viewports give the part of it the scene covers, and their compare gives the far plane's side. No profile key.dm/gfx/msl/stereo.metal): a depth value is affine in 1 / distance, so the parallax needs no constant of the game. The screen's plane follows the nearest things in the scene, and each eye's source pixel is searched along the row so near surfaces cover far ones.DM_IMMERSIVE_3Dscales the depth (0 is flat).DM_IMMERSIVE_SHOT=<present>:<path.png>writes both eyes of a present side by side without a headset. Internals: docs/development.md, "The game on a Vision Pro"; the player's page: docs/settings.md, "Apple Vision Pro".Checked
dm/gfx/tests/t_stereo(new, 52 checks under Metal validation): the scene depth is chosen over a decoy and a shadow map; both far-plane sides; the parallax of far, middle and near markers; pinned slots are left alone; a 2D frame stays flat; a HUD bar drawn over a window texture through the replay stands in front of the scene with no copy of itself left in the scene. With the HUD handling off, those last checks fail.t_present_alloc,t_present_crop,t_nvn,t_clear_dsandt_present_rescuepass unchanged;dm/tools/testspasses (1,619 tests); all six documentation tests pass. clang-tidy was unavailable.DM_IMMERSIVE=1and an Apple Vision Pro:01008CD0172D6000, v1.5.3): renders in correct 3D.01002C000E35E000, v20.5.2): renders flat.010082400BCC6000), Mac captures without a headset: the tap now takes the scene's depth instead of a normals prepass, including when the scene's depth is cleared for the UI before the present. Goose's scene reference correlates 0.995 with the picture inverted versus 0.09 upright, so alignment flips it. Dredge still takes the same scene depth.t_stereocovers the atlas, prepass, tie and clear cases (73 checks). Headset retest pending.scripts/regress.py checkon the two local games with Mac builds: PASS on both (9223/9223 and 3900/3900 shaders, 0 newly failing, 0 changed MSL, no interface change); baselinebee218d97844, candidateba6c8e3f754c.regress.py impactlists the change as reaching every NVN game's presents (present:*), so it does not pick between games. Neither game has a smoke route, so this checks their shader chains and pipeline interfaces, not a boot. Not rerun in the current workspace, which lacks configured game folders.Open
Front ends
Front-ends: new: DM_IMMERSIVE and DM_IMMERSIVE_3D settings (Mac), in launcher.json
Docs
docs/reference/<subsystem>.mdentry, andpython3 dm/tools/docs_symptoms.pywas run (the symptom index)Closes #2755