Repository navigation
quest: plan remote desktop's moq primitives, and move teleop to kixel.ai #8702
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
c151630
quest: plan remote desktop's moq primitives, and move teleop to kixel.ai
kixelated 2a69711
Merge origin/main into quest-plan/remote-desktop
kixelated 066d847
quest: answer review of the remote desktop plan
kixelated 30b6d84
quest: make the refresh barrier causal and stamp stills by sampling time
kixelated File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,97 @@ | ||
| # [L] Refresh a track by interval, and on request | ||
|
|
||
| ## Goal | ||
|
|
||
| A publisher can declare in TRACK_INFO a refresh interval: a live join asks | ||
| for a new group only once the newest group arrived at least that long ago. | ||
| A reader can also ask for one with a public `refresh()` on the track | ||
| subscriber, in Rust and `@moq/net`, judged by the same interval. Every hop | ||
| coalesces requests and judges them against its own cache, so the | ||
| publisher's interval caps how often new groups are made, by design rather | ||
| than by a per-hop rate limiter. | ||
|
|
||
| The interval is a duration, encoded and read like `max_age`, so it means the | ||
| same on a timed and a scale-less track. A group's age is measured on the | ||
| judging node's own monotonic clock from when that node first received (or, at | ||
| the publisher, created) the group, as arrival-age expiry already measures it. | ||
| It never compares media timestamps, which carry no wall-clock meaning and, on | ||
| a scale-less track, no unit, and it adds no local clock to the model. | ||
|
|
||
| A track of one-frame groups, such as a [thumbnail](/quest/m4/hang-thumbnail.md), | ||
| then produces a fresh group when a reader arrives or asks, and never more | ||
| often than its interval, however many readers fan out behind the relays. | ||
| A track with no group yet needs no interval: its first reader asks for the | ||
| first group at once, as the first consumer of an empty track does today | ||
| (`wanted()` is `Some(0)`), and the interval applies from that group on. | ||
|
|
||
| ## Plan | ||
|
|
||
| Planned 2026-11-13 (maintainer, /quest-plan of a remote desktop for | ||
| kixel.ai). The consumer is Kixel's desktop host, which serves an agent's | ||
| screenshot from a lossless thumbnail and wants dashboard previews from the | ||
| same track. | ||
|
|
||
| Decided: | ||
|
|
||
| - ✅ moq m2, in lite-07 (rejected: m4 and lite-08). Its consumer is later | ||
| work, which would put it in m4, but it changes lite-07-wip, and missing | ||
| that freeze costs a whole wire version. | ||
| - ✅ An interval in TRACK_INFO plus a public `refresh()` (rejected: changing | ||
| only the byte-ratio check; a per-hop rate limiter). The interval bounds | ||
| the request rate by what the publisher declares. | ||
|
|
||
| Facts found while planning: | ||
|
|
||
| - `Refresh` is a byte ratio today. `over_refresh` | ||
| (`rs/moq-net/src/model/group.rs`) tests `cache - first > first * ratio`, | ||
| so a complete one-frame group never refreshes at any ratio. | ||
| - `Subscriber::join` is `pub(crate)` in `rs/moq-net/src/model/track.rs`. | ||
| The only trigger is a live-edge subscribe, and no reader can ask for a | ||
| group. | ||
| - SUBSCRIBE_UPDATE's `Wanted Group` already carries a relay's coalesced | ||
| demand upstream, and [time addressing](/quest/m2/time-addressing/lite07.md) | ||
| turns it into a start time. Build on that rather than adding a second | ||
| field. The request barrier above rides the update numbers, not that start | ||
| time, so it needs no clock shared between hops. | ||
| - `rs/moq-video/src/encode/producer.rs` forces a keyframe on a join; it | ||
| keeps working on the byte ratio. | ||
|
|
||
| Recommendations for the PR: | ||
|
|
||
| - Whether the interval sits beside the byte ratio or replaces `Refresh` with | ||
| a typed choice. A track may want both (video wants a keyframe when the GOP | ||
| is long, and not more than once a second), so recommend both, either one | ||
| firing within the interval. | ||
| - `refresh()` has a request barrier, and it must be causal at the publisher: | ||
| a group already in flight when the request reached a relay would pass an | ||
| arrival check while showing the state before the request. So a group | ||
| answers a request only if the publisher created it after receiving that | ||
| request. Recommend reusing SUBSCRIBE_UPDATE's update number: each hop | ||
| forwards a request as a new update after it receives it (coalescing | ||
| requests that arrive between two forwards), the publisher tags each group | ||
| it creates with the highest update it had seen, and `refresh()` resolves | ||
| only with a group tagged at or past the update its request rode. The PR | ||
| settles the field's place in the draft and measures its cost; if it costs | ||
| more wire than it earns, weaken the guarantee to "a group that reached the | ||
| node after the request", state that post-action freshness is then best | ||
| effort, and change the thumbnail's promise to match. The interval may | ||
| delay the new group, never substitute an older one. Rust and `@moq/net` | ||
| share this contract. | ||
| - Update the lite draft (TRACK_INFO, SUBSCRIBE_UPDATE, and the changelog), | ||
| `doc/concept/moq-lite.md`, and the matching JS. Run `just test interop | ||
| --all`. | ||
|
|
||
| Tests: a relay chain where many readers join and call `refresh()` on a | ||
| one-frame track produces one group per interval at the publisher, with time | ||
| mocked. | ||
|
|
||
| Public API: additive (`refresh()`, the interval on `Info`). Wire: | ||
| lite-07-wip TRACK_INFO and SUBSCRIBE_UPDATE. | ||
|
|
||
| ## Required | ||
|
|
||
| - [The lite-07 wire addresses groups by start](/quest/m2/time-addressing/lite07.md) - `Wanted Group` becomes the start time this requests by | ||
|
|
||
| ## Related | ||
|
|
||
| - [Hang thumbnails](/quest/m4/hang-thumbnail.md) - the one-frame track this refreshes |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| # [L] The cursor apart from the frame | ||
|
|
||
| ## Goal | ||
|
|
||
| A screen capture in `moq_video::capture` can report the cursor separately | ||
| instead of drawing it into each frame: its position whenever it moves, and | ||
| its image and hotspot whenever the shape changes. A viewer can then draw the | ||
| pointer itself, with no round trip of lag. Where a system cannot report the | ||
| cursor's shape or position, asking for it is an `Unsupported` error naming | ||
| the reason, decided at setup. | ||
|
|
||
| ## Plan | ||
|
|
||
| Planned 2026-11-13 (maintainer, /quest-plan of a remote desktop for | ||
| kixel.ai). The consumer is Kixel's desktop page, which draws the remote | ||
| pointer locally so a human's aim does not trail by the round-trip time, as | ||
| RDP and Parsec do. The track that carries it is Kixel's convention, not | ||
| hang's, until a second consumer needs it. | ||
|
|
||
| Decided: ✅ a separate cursor (rejected: drawn into the frame, with the | ||
| viewer's own pointer shown on top). Milestone: ✅ m4 (rejected: m3), since | ||
| the consumer is Kixel's later (m2) plans. | ||
|
|
||
| Today `capture::Config::cursor` is a bool that draws the cursor in or leaves | ||
| it out: the portal's embedded mode, `SetIsCursorCaptureEnabled` on WGC, | ||
| `setShowsCursor` on macOS, and a CPU blend of the XFixes cursor on X11. | ||
|
|
||
| What each system offers, to confirm in the PR: | ||
|
|
||
| - **Wayland:** the portal's metadata cursor mode puts position and bitmap in | ||
| each PipeWire buffer's metadata. | ||
| - **X11:** XFixes reports shape changes and their image; position by query | ||
| or XInput2 motion events. | ||
| - **Windows:** DXGI Desktop Duplication gives pointer shape and position, but | ||
| capture moved to WGC, which has neither. `GetCursorInfo` plus the cursor's | ||
| icon may do. | ||
| - **macOS:** position from `CGEvent`. No public API gives another app's | ||
| cursor image (`NSCursor.currentSystemCursor` is deprecated). Find out what | ||
| works, and say what does not. | ||
|
|
||
| Recommendations: a stream beside the frames, in the capture's own coordinate | ||
| space and clock, with each shape sent once and named by id after. The report | ||
| is the current state (shape and position) plus change notifications, not an | ||
| event log, so a viewer that joins or reconnects reads the current shape before | ||
| any position that names it. Whatever carries it should start each shape's group | ||
| with that shape, so a late reader at the newest group needs no replay. | ||
| On a Wayland session the caller opened, the metadata cursor mode must have | ||
| been selected before the session started | ||
| ([the portal session](/quest/m4/capture-portal-session.md)); another mode is | ||
| `Unsupported`. | ||
|
|
||
| Public API: additive. Wire: none. | ||
|
|
||
| ## Hosts | ||
|
|
||
| - linux - the portal metadata cursor and XFixes | ||
| - macos - the cursor without a public shape API | ||
| - windows - the cursor under WGC | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| # [M] Capture from the caller's portal session | ||
|
|
||
| ## Goal | ||
|
|
||
| `moq_video::capture` can open a screen capture on a portal session the caller | ||
| already holds: a PipeWire fd and node id from an XDG RemoteDesktop session | ||
| that also started its screencast. One consent dialog then covers both seeing | ||
| and controlling the screen, and the caller's input lands in the same | ||
| coordinate space as the frames. | ||
|
coderabbitai[bot] marked this conversation as resolved.
Resolved
|
||
|
|
||
| ## Plan | ||
|
|
||
| Planned 2026-11-13 (maintainer, /quest-plan of a remote desktop for | ||
| kixel.ai). The consumer is Kixel's Wayland desktop host, which injects | ||
| keyboard and mouse through libei on a RemoteDesktop session | ||
| (`ConnectToEIS`). A ScreenCast session opened separately by moq-video would | ||
| ask the user a second time and could pick another monitor than the one | ||
| being controlled. | ||
|
|
||
| Milestone: ✅ m4 (rejected: m3). The consumer is Kixel's later (m2) plans. | ||
|
|
||
| Today `capture/pipewire.rs` opens its own ScreenCast session through ashpd, | ||
| with only the `screencast` and `camera` features enabled, and keeps the | ||
| restore token itself. | ||
|
|
||
| Recommendations for the PR: | ||
|
|
||
| - Take the stream the portal describes (node id and the stream's properties) | ||
| rather than an ashpd session type, so moq-video does not choose the | ||
| caller's portal library and the input side stays out of moq. Forward the | ||
| stream's `mapping_id` unchanged, since it is what matches the stream to the | ||
| caller's libei region, and take `position` and `size` as optional: the | ||
| portal sends `position` only for monitor streams and `size` is optional, so | ||
| a window stream has no position. Frame pixel size comes from the negotiated | ||
| PipeWire format, never from the portal's compositor-space `size`, and a | ||
| missing value is reported to the caller as absent, not guessed. | ||
| - Take a handle that yields a fresh PipeWire remote fd for each open (the | ||
| caller's `OpenPipeWireRemote` on its session), not one fd. The PipeWire | ||
| connection consumes its fd, and capture reopens after a format change and | ||
| for each on-demand viewer, so a single fd serves only the first open. A | ||
| one-method trait the caller implements fits better than a callback | ||
| parameter; settle its shape in the PR. | ||
| - The caller selects the metadata cursor mode in `SelectSources` before it | ||
| starts the session, because the portal fixes the cursor mode there. The | ||
| stream description carries the mode the caller chose, and a | ||
| [cursor](/quest/m4/capture-cursor.md) request on a session with another | ||
| mode is `Unsupported` at setup, naming the mode. | ||
| - The caller owns the session's lifetime and restore token. A capture on a | ||
| session that ends reports the end like any other stopped capture. | ||
| - Test against a headless compositor in CI if one runs the RemoteDesktop | ||
| portal; otherwise a by-hand run on GNOME and KDE, recorded in the PR. | ||
|
|
||
| Public API: additive. Wire: none. | ||
|
|
||
| ## Hosts | ||
|
|
||
| - linux - the XDG portal and PipeWire | ||
|
|
||
| ## Related | ||
|
|
||
| - [The cursor apart from the frame](/quest/m4/capture-cursor.md) - needs the metadata cursor mode this session carries | ||
| - [Linux capture parity](/quest/m3/capture-linux.md) - the rest of Wayland capture | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,70 @@ | ||
| # [L] Hang thumbnails | ||
|
|
||
| ## Goal | ||
|
|
||
| A hang broadcast can carry still images in a new `thumbnail` catalog | ||
| section: renditions keyed by name, each with a mime (`image/png` for | ||
| lossless, and lossy formats such as `image/jpeg`), a width, and a height, | ||
| one image per group. Rust (`hang`) and `@moq/hang` both read and write it. | ||
| A moq-video producer scales and encodes the newest captured or decoded | ||
| frame into each rendition whenever its track | ||
| [refreshes](/quest/m2/refresh-interval.md), so a reader that arrives or asks | ||
|
coderabbitai[bot] marked this conversation as resolved.
Resolved
|
||
| gets a current still, and the publisher encodes no more than one per | ||
|
kixelated marked this conversation as resolved.
Resolved
|
||
| interval. Before any reader has asked there is no still: the first reader's | ||
| join asks for the first one at once, and the interval applies from then on. | ||
|
|
||
| ## Plan | ||
|
|
||
| Planned 2026-11-13 (maintainer, /quest-plan of a remote desktop for | ||
| kixel.ai). The consumer is Kixel's desktop host: an agent's screenshot is a | ||
| lossless rendition, refreshed after each action settles, and the dashboard's | ||
| previews are a small lossy one. | ||
|
|
||
| Decided: | ||
|
|
||
| - ✅ A `thumbnail` section in the hang schema (rejected: the existing | ||
| `binary` section with `mime: image/png`). Stills get typed fields (size, | ||
| format), and several renditions, as video has. | ||
| - ✅ The schema and the moq-video producer in one quest (rejected: schema | ||
| only). Rooms and the studio would otherwise each write the encoder again. | ||
| - ✅ m4 (rejected: m3). Additive, and the consumer is Kixel's later (m2) | ||
| plans. | ||
|
|
||
| Facts found while planning: | ||
|
|
||
| - Both catalog parsers tolerate unknown sections (`z.looseObject` in | ||
| `js/hang/src/catalog/root.ts`, and serde without `deny_unknown_fields`), so | ||
| the section is additive on the wire too. | ||
| - `rs/hang/src/catalog/root.rs` already has a test with | ||
| `"thumbnail": {"mode":"snapshot","mime":"image/jpeg"}` under `binary`. | ||
| Replace it with the new section rather than keeping two ways to say the | ||
| same thing. | ||
|
|
||
| Recommendations for the PR: | ||
|
|
||
| - A group's identity is its start timestamp, and the same identity must never | ||
| carry different content. So each still is stamped with the time it was | ||
| sampled, on the root's `media.clock`, strictly after the previous still's | ||
| (one tick later when two samplings collide), and never with the source | ||
| frame's own timestamp: a static screen would give two refreshes the same | ||
| frame and so the same start. The still shows the newest frame at that | ||
| time, so it lines up with the video at its timestamp. | ||
| - Each still is encoded after the request that asks for it, from the newest | ||
| frame the capture has delivered by then, and never reuses an earlier | ||
| encode, so it meets the `refresh()` request barrier. A capture that | ||
| delivers frames only on change (PipeWire does) has its last frame as the | ||
| current screen, so the producer must not wait for a newer one. A caller | ||
| that needs the result of an action calls `refresh()` once the action has | ||
| settled. | ||
| - The producer refuses, at setup, a mime it cannot encode, rather than | ||
| publishing a rendition it never fills. PNG and JPEG first. Take a | ||
| dependency for encoding only at its newest stable version, and only if it | ||
| earns its build time. | ||
| - Readers, such as a watch poster image or a room's grid, are follow-ups | ||
| once someone needs them. | ||
|
|
||
| Public API: additive. Wire: the hang catalog gains an optional section. | ||
|
|
||
| ## Required | ||
|
|
||
| - [Refresh a track by interval, and on request](/quest/m2/refresh-interval.md) - what makes a one-image group refresh on demand | ||
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.