Repository navigation
feat(demo): a VR demo that lays out the catalog's spatial fields, tested end to end #8781
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
14 commits
Select commit
Hold shift + click to select a range
4c2ba27
feat(demo): add hud + audio code to vr demo
nuts-rice 7beefa6
chore(demo): lock the vr demo's dependencies
nuts-rice 901d1e7
feat(demo): add vr content recenter
nuts-rice 7341acd
refactor(demo): drop the VR demo's catalog flags, which import alread…
kixelated 0dfb627
fix(watch): pass the OpusHead to WebCodecs so multichannel Opus decodes
kixelated c94db0a
test(vr): import a VR180 ambisonic MP4 and check demo/vr lays it out …
kixelated 9ce8bbf
docs(demo): document the VR demo and serve it with just vr
kixelated 6ad3004
quest: follow up with a headset try of the VR demo
kixelated 7a88b4c
fix(watch): keep the Opus pre-skip with Terminal once WebCodecs gets …
kixelated 0edcd21
chore(quest): claim quest/m1/vr-demo.md
kixelated 30e0375
Merge branch 't3/vr-demo-catalog-consumer' into quest/m1/vr-demo
kixelated 9eb9be2
quest: finish the VR demo
kixelated e006212
quest: record the maintainer's pick to host the VR demo once it plays…
kixelated 85d3d88
fix(demo,watch): pose the VR listener on Firefox, and hand WebCodecs …
kixelated File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| set shell := ["bash", "-euo", "pipefail", "-c"] | ||
|
|
||
| # Serve the VR demo against a relay; the page takes ?url= and ?name= to override. | ||
| serve url="http://localhost:4443": install | ||
| VITE_RELAY_URL="{{ url }}" bun --bun vite | ||
|
|
||
| # Install the workspace deps that vite and the demo's imports need. | ||
| install: | ||
| uv run --no-project python ../../sh/js/install.py update |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| { | ||
| "name": "@moq/demo-vr", | ||
| "private": true, | ||
| "type": "module", | ||
| "description": "Media over QUIC - VR demo", | ||
| "license": "(MIT OR Apache-2.0)", | ||
| "scripts": { | ||
| "dev": "vite", | ||
| "build": "vite build", | ||
| "check": "tsc --noEmit" | ||
| }, | ||
| "dependencies": { | ||
| "@moq/watch": "workspace:^", | ||
| "three": "^0.186.1" | ||
| }, | ||
| "devDependencies": { | ||
| "@types/three": "^0.186.0", | ||
| "rolldown": "^1.2.16", | ||
| "typescript": "7.0.2", | ||
| "vite": "^8.3.5" | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,134 @@ | ||
| import * as THREE from "three"; | ||
|
|
||
| export type Placement = { kind: "ambisonic" } | { kind: "point"; position: THREE.Vector3 } | { kind: "head" }; | ||
|
|
||
| // Virtual speakers at the corners of a cube in ambisonic axes (W, Y, Z, X) | ||
| const SPEAKERS = [-1, 1].flatMap((x) => | ||
| [-1, 1].flatMap((y) => [-1, 1].map((z) => [x, y, z].map((v) => v / Math.sqrt(3)))), | ||
| ); | ||
|
|
||
| const MAX_RE = 1 / Math.sqrt(3); | ||
|
|
||
| // Scratch for pose(), which runs every XR frame. | ||
| const POSE = { | ||
| position: new THREE.Vector3(), | ||
| rotation: new THREE.Quaternion(), | ||
| scale: new THREE.Vector3(), | ||
| forward: new THREE.Vector3(), | ||
| up: new THREE.Vector3(), | ||
| }; | ||
|
|
||
| // Renders audio graph around a viewer's head | ||
| export class Spatial { | ||
| #context: BaseAudioContext; | ||
| #nodes: AudioNode[] = []; | ||
| #tracksPosition: boolean; | ||
| // Every placement mixes here, so muting is one param. | ||
| #out: GainNode; | ||
|
|
||
| constructor(root: AudioNode, placement: Placement) { | ||
| this.#context = root.context; | ||
| this.#out = new GainNode(this.#context); | ||
| this.#out.connect(this.#context.destination); | ||
| this.#nodes.push(this.#out); | ||
| const destination = this.#out; | ||
|
|
||
| switch (placement.kind) { | ||
| case "head": | ||
| root.connect(destination); | ||
| this.#tracksPosition = false; | ||
| break; | ||
| case "point": { | ||
| const panner = this.#panner(placement.position, 1); | ||
| root.connect(panner).connect(destination); | ||
| this.#tracksPosition = true; | ||
| break; | ||
| } | ||
| case "ambisonic": | ||
| this.#ambisonic(root, destination); | ||
| // The field is infinitely far, so only the head's rotation moves it. | ||
| this.#tracksPosition = false; | ||
| break; | ||
| } | ||
| this.#nodes.push(root); | ||
| } | ||
|
|
||
| // Decodes out to fixed speakers around the scene. A higher order plays at first order: the | ||
| // splitter's discrete downmix keeps ACN channels 0 to 3 and drops the rest. | ||
| #ambisonic(root: AudioNode, destination: AudioNode) { | ||
| if (root.channelCount < 4) throw new Error(`ambisonics needs at least 4 channels, got ${root.channelCount}`); | ||
|
|
||
| const split = new ChannelSplitterNode(this.#context, { numberOfOutputs: 4 }); | ||
| root.connect(split); | ||
| this.#nodes.push(split); | ||
|
|
||
| for (const [x, y, z] of SPEAKERS) { | ||
| // ACN order is W, Y, Z, X. | ||
| const weights = [1, 3 * MAX_RE * y, 3 * MAX_RE * z, 3 * MAX_RE * x].map((w) => w / SPEAKERS.length); | ||
| // Ambisonic axes to WebAudio | ||
| const panner = this.#panner(new THREE.Vector3(-y, z, -x), 0); | ||
| panner.connect(destination); | ||
|
|
||
| weights.forEach((weight, channel) => { | ||
| const gain = new GainNode(this.#context, { gain: weight }); | ||
| split.connect(gain, channel).connect(panner); | ||
| this.#nodes.push(gain); | ||
| }); | ||
| } | ||
| } | ||
|
|
||
| #panner(position: THREE.Vector3, rolloffFactor: number): PannerNode { | ||
| const panner = new PannerNode(this.#context, { | ||
| panningModel: "HRTF", | ||
| distanceModel: "inverse", | ||
| refDistance: 1, | ||
| rolloffFactor, | ||
| positionX: position.x, | ||
| positionY: position.y, | ||
| positionZ: position.z, | ||
| }); | ||
| this.#nodes.push(panner); | ||
| return panner; | ||
| } | ||
|
|
||
| // The mix every placement feeds, for meters and the harness. | ||
| get output(): AudioNode { | ||
| return this.#out; | ||
| } | ||
|
|
||
| set muted(muted: boolean) { | ||
| this.#out.gain.setTargetAtTime(muted ? 0 : 1, this.#context.currentTime, 0.01); | ||
| } | ||
|
|
||
| // Move the listener to the viewer's head, in the same space as the placement. | ||
| pose(head: THREE.Matrix4) { | ||
| const { position, rotation, scale, forward, up } = POSE; | ||
| head.decompose(position, rotation, scale); | ||
| forward.set(0, 0, -1).applyQuaternion(rotation); | ||
| up.set(0, 1, 0).applyQuaternion(rotation); | ||
|
|
||
| const listener = this.#context.listener; | ||
| if (!this.#tracksPosition) position.set(0, 0, 0); | ||
| // Firefox has no listener AudioParams, only the older setters. | ||
| if (!listener.positionX) { | ||
| listener.setPosition(position.x, position.y, position.z); | ||
| listener.setOrientation(forward.x, forward.y, forward.z, up.x, up.y, up.z); | ||
| return; | ||
| } | ||
| const at = this.#context.currentTime; | ||
| const set = (param: AudioParam, value: number) => param.setTargetAtTime(value, at, 0.01); | ||
| set(listener.positionX, position.x); | ||
| set(listener.positionY, position.y); | ||
| set(listener.positionZ, position.z); | ||
| set(listener.forwardX, forward.x); | ||
| set(listener.forwardY, forward.y); | ||
| set(listener.forwardZ, forward.z); | ||
| set(listener.upX, up.x); | ||
| set(listener.upY, up.y); | ||
| set(listener.upZ, up.z); | ||
| } | ||
|
|
||
| close() { | ||
| for (const node of this.#nodes) node.disconnect(); | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| <!doctype html> | ||
| <html lang="en"> | ||
| <head> | ||
| <meta charset="utf-8" /> | ||
| <meta name="viewport" content="width=device-width, initial-scale=1" /> | ||
| <title>MoQ VR</title> | ||
| <link rel="icon" href="data:," /> | ||
| <style> | ||
| body { | ||
| margin: 0; | ||
| overflow: hidden; | ||
| background: #111; | ||
| color: #ddd; | ||
| font: 14px monospace; | ||
| } | ||
| #help { | ||
| position: fixed; | ||
| left: 8px; | ||
| bottom: 8px; | ||
| opacity: 0.6; | ||
| pointer-events: none; | ||
| } | ||
| </style> | ||
| </head> | ||
| <body> | ||
| <div id="help">drag to look · R recentre · M mute</div> | ||
| <script type="module" src="./index.ts"></script> | ||
| </body> | ||
| </html> |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.