From c15163016848d89b09f739c82bf8906cf1e2ee80 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 13 Nov 2026 18:28:44 -0800 Subject: [PATCH 1/3] quest: plan remote desktop's moq primitives, and move teleop to kixel.ai Co-Authored-By: Claude Opus 5.5 --- quest/m2/refresh-interval.md | 74 +++++++++++++++ quest/m3/mesh/README.md | 1 - quest/m4/capture-cursor.md | 50 ++++++++++ quest/m4/capture-portal-session.md | 44 +++++++++ quest/m4/hang-thumbnail.md | 57 ++++++++++++ quest/m5/teleop-arbitration.md | 45 --------- quest/m5/teleop-mavlink.md | 93 ------------------- quest/m5/teleop-proof.md | 40 -------- quest/m5/teleop-robot.md | 144 ----------------------------- 9 files changed, 225 insertions(+), 323 deletions(-) create mode 100644 quest/m2/refresh-interval.md create mode 100644 quest/m4/capture-cursor.md create mode 100644 quest/m4/capture-portal-session.md create mode 100644 quest/m4/hang-thumbnail.md delete mode 100644 quest/m5/teleop-arbitration.md delete mode 100644 quest/m5/teleop-mavlink.md delete mode 100644 quest/m5/teleop-proof.md delete mode 100644 quest/m5/teleop-robot.md diff --git a/quest/m2/refresh-interval.md b/quest/m2/refresh-interval.md new file mode 100644 index 00000000000..fdd735a80e1 --- /dev/null +++ b/quest/m2/refresh-interval.md @@ -0,0 +1,74 @@ +# [L] Refresh a track by interval, and on request + +## Goal + +A publisher can declare in TRACK_INFO a refresh interval: a live join asks +for a new group only once the newest group started at least that long ago. +A reader can also ask for one with a public `refresh()` on the track +subscriber, in Rust and `@moq/net`, judged by the same interval. Every hop +coalesces requests and judges them against its own cache, so the +publisher's interval caps how often new groups are made, by design rather +than by a per-hop rate limiter. + +A track of one-frame groups, such as a [thumbnail](/quest/m4/hang-thumbnail.md), +then produces a fresh group when a reader arrives or asks, and never more +often than its interval, however many readers fan out behind the relays. + +## Plan + +Planned 2026-11-13 (maintainer, /quest-plan of a remote desktop for +kixel.ai). The consumer is Kixel's desktop host, which serves an agent's +screenshot from a lossless thumbnail and wants dashboard previews from the +same track. + +Decided: + +- ✅ moq m2, in lite-07 (rejected: m4 and lite-08). Its consumer is later + work, which would put it in m4, but it changes lite-07-wip, and missing + that freeze costs a whole wire version. +- ✅ An interval in TRACK_INFO plus a public `refresh()` (rejected: changing + only the byte-ratio check; a per-hop rate limiter). The interval bounds + the request rate by what the publisher declares. + +Facts found while planning: + +- `Refresh` is a byte ratio today. `over_refresh` + (`rs/moq-net/src/model/group.rs`) tests `cache - first > first * ratio`, + so a complete one-frame group never refreshes at any ratio. +- `Subscriber::join` is `pub(crate)` in `rs/moq-net/src/model/track.rs`. + The only trigger is a live-edge subscribe, and no reader can ask for a + group. +- SUBSCRIBE_UPDATE's `Wanted Group` already carries a relay's coalesced + demand upstream, and [time addressing](/quest/m2/time-addressing/lite07.md) + turns it into a start time. Build on that rather than adding a second + field. +- `rs/moq-video/src/encode/producer.rs` forces a keyframe on a join; it + keeps working on the byte ratio. + +Recommendations for the PR: + +- Whether the interval sits beside the byte ratio or replaces `Refresh` with + a typed choice. A track may want both (video wants a keyframe when the GOP + is long, and not more than once a second), so recommend both, either one + firing within the interval. +- `refresh()` asks for a group starting after the newest one the reader + holds, and resolves when one arrives, which the interval may delay. It + never resolves with a group no newer than the one the reader held. +- Update the lite draft (TRACK_INFO, SUBSCRIBE_UPDATE, and the changelog), + `doc/concept/moq-lite.md`, and the matching JS. Run `just test interop + --all`. + +Tests: a relay chain where many readers join and call `refresh()` on a +one-frame track produces one group per interval at the publisher, with time +mocked. + +Public API: additive (`refresh()`, the interval on `Info`). Wire: +lite-07-wip TRACK_INFO and SUBSCRIBE_UPDATE. + +## Required + +- [The lite-07 wire addresses groups by start](/quest/m2/time-addressing/lite07.md) - `Wanted Group` becomes the start time this requests by + +## Related + +- [Hang thumbnails](/quest/m4/hang-thumbnail.md) - the one-frame track this refreshes diff --git a/quest/m3/mesh/README.md b/quest/m3/mesh/README.md index 5023c3e27d2..c881d2959e7 100644 --- a/quest/m3/mesh/README.md +++ b/quest/m3/mesh/README.md @@ -111,4 +111,3 @@ the third, and survives a relay restart. - [P2P](/quest/m4/p2p/README.md) - internet peer sessions, which the m3 mesh does not use - [Mesh over ICE](/quest/m5/mesh-ice.md) - deferred direct paths between mesh nodes on different networks - [End-to-end encryption](/quest/m5/e2ee/README.md) - the AEAD the encryption level will reuse -- [Teleoperation](/quest/m5/teleop-robot.md) - robot track conventions that could ride the mesh diff --git a/quest/m4/capture-cursor.md b/quest/m4/capture-cursor.md new file mode 100644 index 00000000000..7d60a00ffca --- /dev/null +++ b/quest/m4/capture-cursor.md @@ -0,0 +1,50 @@ +# [L] The cursor apart from the frame + +## Goal + +A screen capture in `moq_video::capture` can report the cursor separately +instead of drawing it into each frame: its position whenever it moves, and +its image and hotspot whenever the shape changes. A viewer can then draw the +pointer itself, with no round trip of lag. Where a system cannot report the +cursor's shape or position, asking for it is an `Unsupported` error naming +the reason, decided at setup. + +## Plan + +Planned 2026-11-13 (maintainer, /quest-plan of a remote desktop for +kixel.ai). The consumer is Kixel's desktop page, which draws the remote +pointer locally so a human's aim does not trail by the round-trip time, as +RDP and Parsec do. The track that carries it is Kixel's convention, not +hang's, until a second consumer needs it. + +Decided: ✅ a separate cursor (rejected: drawn into the frame, with the +viewer's own pointer shown on top). Milestone: ✅ m4 (rejected: m3), since +the consumer is Kixel's later (m2) plans. + +Today `capture::Config::cursor` is a bool that draws the cursor in or leaves +it out: the portal's embedded mode, `SetIsCursorCaptureEnabled` on WGC, +`setShowsCursor` on macOS, and a CPU blend of the XFixes cursor on X11. + +What each system offers, to confirm in the PR: + +- **Wayland:** the portal's metadata cursor mode puts position and bitmap in + each PipeWire buffer's metadata. +- **X11:** XFixes reports shape changes and their image; position by query + or XInput2 motion events. +- **Windows:** DXGI Desktop Duplication gives pointer shape and position, but + capture moved to WGC, which has neither. `GetCursorInfo` plus the cursor's + icon may do. +- **macOS:** position from `CGEvent`. No public API gives another app's + cursor image (`NSCursor.currentSystemCursor` is deprecated). Find out what + works, and say what does not. + +Recommendations: a stream beside the frames, in the capture's own coordinate +space and clock, with each shape sent once and named by id after. + +Public API: additive. Wire: none. + +## Hosts + +- linux - the portal metadata cursor and XFixes +- macos - the cursor without a public shape API +- windows - the cursor under WGC diff --git a/quest/m4/capture-portal-session.md b/quest/m4/capture-portal-session.md new file mode 100644 index 00000000000..cc8a938a145 --- /dev/null +++ b/quest/m4/capture-portal-session.md @@ -0,0 +1,44 @@ +# [M] Capture from the caller's portal session + +## Goal + +`moq_video::capture` can open a screen capture on a portal session the caller +already holds: a PipeWire fd and node id from an XDG RemoteDesktop session +that also started its screencast. One consent dialog then covers both seeing +and controlling the screen, and the caller's input lands in the same +coordinate space as the frames. + +## Plan + +Planned 2026-11-13 (maintainer, /quest-plan of a remote desktop for +kixel.ai). The consumer is Kixel's Wayland desktop host, which injects +keyboard and mouse through libei on a RemoteDesktop session +(`ConnectToEIS`). A ScreenCast session opened separately by moq-video would +ask the user a second time and could pick another monitor than the one +being controlled. + +Milestone: ✅ m4 (rejected: m3). The consumer is Kixel's later (m2) plans. + +Today `capture/pipewire.rs` opens its own ScreenCast session through ashpd, +with only the `screencast` and `camera` features enabled, and keeps the +restore token itself. + +Recommendations for the PR: + +- Take the fd and node (and the stream's size and position from the portal) + rather than an ashpd session type, so moq-video does not choose the + caller's portal library and the input side stays out of moq. +- The caller owns the session's lifetime and restore token. A capture on a + session that ends reports the end like any other stopped capture. +- Test against a headless compositor in CI if one runs the RemoteDesktop + portal; otherwise a by-hand run on GNOME and KDE, recorded in the PR. + +Public API: additive. Wire: none. + +## Hosts + +- linux - the XDG portal and PipeWire + +## Related + +- [Linux capture parity](/quest/m3/capture-linux.md) - the rest of Wayland capture diff --git a/quest/m4/hang-thumbnail.md b/quest/m4/hang-thumbnail.md new file mode 100644 index 00000000000..6418ea496e0 --- /dev/null +++ b/quest/m4/hang-thumbnail.md @@ -0,0 +1,57 @@ +# [L] Hang thumbnails + +## Goal + +A hang broadcast can carry still images in a new `thumbnail` catalog +section: renditions keyed by name, each with a mime (`image/png` for +lossless, and lossy formats such as `image/jpeg`), a width, and a height, +one image per group. Rust (`hang`) and `@moq/hang` both read and write it. +A moq-video producer scales and encodes the newest captured or decoded +frame into each rendition whenever its track +[refreshes](/quest/m2/refresh-interval.md), so a reader that arrives or asks +gets a current still, and the publisher encodes no more than one per +interval. + +## Plan + +Planned 2026-11-13 (maintainer, /quest-plan of a remote desktop for +kixel.ai). The consumer is Kixel's desktop host: an agent's screenshot is a +lossless rendition, refreshed after each action settles, and the dashboard's +previews are a small lossy one. + +Decided: + +- ✅ A `thumbnail` section in the hang schema (rejected: the existing + `binary` section with `mime: image/png`). Stills get typed fields (size, + format), and several renditions, as video has. +- ✅ The schema and the moq-video producer in one quest (rejected: schema + only). Rooms and the studio would otherwise each write the encoder again. +- ✅ m4 (rejected: m3). Additive, and the consumer is Kixel's later (m2) + plans. + +Facts found while planning: + +- Both catalog parsers tolerate unknown sections (`z.looseObject` in + `js/hang/src/catalog/root.ts`, and serde without `deny_unknown_fields`), so + the section is additive on the wire too. +- `rs/hang/src/catalog/root.rs` already has a test with + `"thumbnail": {"mode":"snapshot","mime":"image/jpeg"}` under `binary`. + Replace it with the new section rather than keeping two ways to say the + same thing. + +Recommendations for the PR: + +- Timestamps follow the root's `media.clock`, so a still lines up with the + video frame it came from. +- The producer refuses, at setup, a mime it cannot encode, rather than + publishing a rendition it never fills. PNG and JPEG first. Take a + dependency for encoding only at its newest stable version, and only if it + earns its build time. +- Readers, such as a watch poster image or a room's grid, are follow-ups + once someone needs them. + +Public API: additive. Wire: the hang catalog gains an optional section. + +## Required + +- [Refresh a track by interval, and on request](/quest/m2/refresh-interval.md) - what makes a one-image group refresh on demand diff --git a/quest/m5/teleop-arbitration.md b/quest/m5/teleop-arbitration.md deleted file mode 100644 index 4fd076cd8b1..00000000000 --- a/quest/m5/teleop-arbitration.md +++ /dev/null @@ -1,45 +0,0 @@ -# [M] Operator arbitration - -## Goal - -Exactly one controller commands a vehicle at a time, with explicit handoff, and -the boundary between arbitration and authorization is stated rather than -assumed. - -## Plan - -Decided 2026-10-08: parked in m5 with [robot](/quest/m5/teleop-robot.md), -which it builds on. - -`moq-boy` merges every viewer's input, which is right for crowd control and -wrong for a vehicle. Arbitration is the part of a teleoperation stack that -cannot be lifted from it: one controller is active, the others are read-only, -and handoff is explicit rather than last-writer-wins. - -### Arbitration is not authorization - -The robot subscribes an announce prefix that is a sibling of what it publishes, -so on its own nothing stops anyone able to announce under `control//` from -joining the pool. `moq-boy`'s fan-in is safe because a hostile viewer can only -press B; a vehicle is not that. - -Admission belongs to the relay as a path-scoped grant, not to this crate. -Scoped signing keys landed in moq#2416 (`6b86e612b`), so the enforcing -mechanism already exists: `Scope` in `rs/moq-auth/src/claims.rs` and -`Key::with_scope` in `key.rs` bind a key to publish and subscribe path -prefixes. Write the concrete claims down (a robot publishes `robot/` and -subscribes `control//`; an operator publishes `control//` and -subscribes `robot/`) and prove with a relay test that a key scoped -elsewhere cannot announce under `control//`. Documenting the requirement is -not an acceptable outcome now that enforcement ships. Revoking a superseded or -compromised operator's grant mid-session is relay auth work, not something -this crate implements; scoped keys only gate admission. - -Link loss is the other half: a class whose contract is "drop stale samples" -needs a stated behavior when samples stop arriving, and the vehicle's own -failsafe is what actually holds. Say what the crate reports and what it does -not attempt. - -## Required - -- [Robot teleoperation conventions](/quest/m5/teleop-robot.md) diff --git a/quest/m5/teleop-mavlink.md b/quest/m5/teleop-mavlink.md deleted file mode 100644 index 6f653a91bb3..00000000000 --- a/quest/m5/teleop-mavlink.md +++ /dev/null @@ -1,93 +0,0 @@ -# [L] MAVLink bridge - -## Goal - -A `moq-mavlink` gateway replaces the VPN plus two unmanaged UDP flows that -every cellular ArduPilot vehicle runs today. QGroundControl, Mission Planner -and MAVProxy connect unchanged. - -## Plan - -Parked in m5 until a real ArduPilot user or partner asks (2026-09-30 audit). - -Decided 2026-10-08: the lossy class is a snapshot track per key with a -bounded key set, since one snapshot track carries only one latest value. - -### Shape - -A library crate reached through `moq-cli`, the way `moq-srt` and `moq-rtmp` -already are (neither is a binary; both are crates behind subcommands). - -Each side is bidirectional on one session. The vehicle side publishes what the -flight controller emits and subscribes the operator's commands to write back -into it; the operator side publishes commands and subscribes telemetry to serve -UDP 14550, because QGC speaks UDP and TCP only and its forwarding is one-way. -Neither side is a one-way media import or export, so do not let the subcommand -naming imply one leg. - -### Route by msgid, never parse payloads - -Read the header only: the 24-bit msgid in v2, 8-bit in v1. Payloads stay opaque -bytes and route to a delivery class through a config table, with an unknown -msgid falling to a default. Every track declares `Timescale::MILLI` -explicitly, since an undeclared Rust timescale means untimed. - -This is what makes frame-aware routing free of the dialect tax. CRC_EXTRA (the -per-message schema hash) is only needed to validate a frame, never to read its -header, so custom dialects, new messages and version skew all just work. The -alternative, compiling in `ardupilotmega` the way `mavlink-camera-manager` and -`mavlink-server` do, buys per-field logic we do not need and couples us to a -schema that changes without us. - -### One lossy track for every msgid would starve the slow ones - -Latest-wins is per track. A single lossy track carrying every routed msgid as -its own group lets 50 Hz `ATTITUDE` supersede a 1 Hz `HEARTBEAT` or a GPS fix, -because the newest group wins regardless of which message it holds. Putting -them all in one long-lived group has the opposite failure: nothing can be -skipped and head-of-line blocking is back. - -So the lossy class is a plain track carrying the latest opaque frame, with -one self-contained group per update, so a newer value -never waits behind an older one. One detail decides whether it actually -delivers latest-value: - -- **Key on `(sysid, compid, msgid)`, not msgid alone.** A vehicle is several - components, and an autopilot, a gimbal and a camera all emit `HEARTBEAT` - under msgid 0; keying on msgid alone lets them overwrite each other. Decide - explicitly what to do about instances distinguished only inside a payload, - which the msgid-only rule cannot see. -- **One latest-value track per key.** Per-key latest-value means a track per - `(sysid, compid, msgid)`, named from the key and advertised in the catalog - as it appears. That state grows with whatever the flight-controller link - emits, so bound it: cap the number of keys (or derive the set from the - config table) and fail loud past the cap rather than creating tracks - without limit. - -Not `moq_json::snapshot`: a MAVLink frame is not JSON, and its default batches -up to `MAX_DELTA_FRAMES` merge patches into one ordered group, so an earlier -50 Hz attitude delta would head-of-line block a later heartbeat. - -The reliable class is one bounded ordered group on a plain track: commands, -ACKs, mission, parameter and file transfer are stop-and-wait exchanges carried -in one ordered group. Note the scope in -[robot](/quest/m5/teleop-robot.md): that is gap-free for a live reader, not -recoverable after a lag or a reconnect, so these MAVLink services keep relying -on their own retransmission across a link drop. - -The [compression migration](https://github.com/moq-dev/moq/pull/7046) deleted the -moq-flate snapshot and stream helpers. Keep these framing rules on plain -tracks, setting `compression` when useful, rather than depending on those helpers. - -### Deliberately out of scope - -Advertising the stream in-band. No `VIDEO_STREAM_TYPE` value can describe a -MoQ endpoint, so a GCS discovers the link out of band. Check the current -enumeration when this is picked up: it grew a WHEP entry after the original -four, which is the precedent that adding one upstream in `common.xml` is a -small, real contribution and its own future quest. - -## Required - -- [Robot teleoperation conventions](/quest/m5/teleop-robot.md) -- [Operator arbitration](/quest/m5/teleop-arbitration.md) diff --git a/quest/m5/teleop-proof.md b/quest/m5/teleop-proof.md deleted file mode 100644 index 4bdb3d6e483..00000000000 --- a/quest/m5/teleop-proof.md +++ /dev/null @@ -1,40 +0,0 @@ -# [M] SITL proof and browser ground station - -## Goal - -ArduPilot SITL plus a synthetic camera, flown from a browser ground station -through a real relay, with no hardware. Someone with a laptop reproduces it in -five minutes. - -## Plan - -Parked in m5 until a real ArduPilot user or partner asks (2026-09-30 audit). - -The browser client subscribes video and telemetry and publishes control, the -shape Blue Robotics' Cockpit proves is viable. It is the demo and the -end-to-end proof of the primitive, not a bid to out-feature QGroundControl: -real users reach the same link through the `moq-mavlink` gateway. - -SITL is the right proof surface precisely because it removes the camera, which -otherwise dominates the latency budget and would make the demo a measurement of -somebody's webcam. - -The ground station's catalog and delivery-class shapes come from a zod -schema mirroring the documented robot conventions, written once and extending the root -schema through the same seam `js/hang/src/catalog/root.ts` uses (there is no -`@moq/mux`). Decided 2026-10-08: no separate `@moq/robot` package until a -second browser consumer needs it; the schema folds in here. A browser -observer cannot degrade the operator's classes (the publisher's -`Info::max_age` bounds the window), so the schema is for reuse, not safety. - -Standing SITL up in CI is separate work with its own build dependencies. -Reproducibility by hand is the bar here; automate it later if it proves worth -the maintenance. - -## Required - -- [MAVLink bridge](/quest/m5/teleop-mavlink.md) - the gateway the vehicle side runs - -## Related - -- [Robot teleoperation conventions](/quest/m5/teleop-robot.md) - the conventions the zod schema mirrors diff --git a/quest/m5/teleop-robot.md b/quest/m5/teleop-robot.md deleted file mode 100644 index 2d10477c0ea..00000000000 --- a/quest/m5/teleop-robot.md +++ /dev/null @@ -1,144 +0,0 @@ -# [M] Robot teleoperation conventions - -## Goal - -The track shapes and discovery every teleoperated machine needs are -documented conventions in `doc/`, built on moq-net and moq-json, so an -integrator writes an adapter rather than a stack. There is no `moq-robot` -crate, and this quest ships documentation only; [the MAVLink -gateway](/quest/m5/teleop-mavlink.md) implements the conventions in its own -quest. It is the gate for the rest of the questline. - -## Plan - -Decided 2026-11-07 (maintainer, /quest-plan from the API sweep of open PRs and remaining quests): ✅ no `moq-robot` crate (rejected: a generic robot -crate, too niche for a published library with no second consumer). The -shapes below become documented conventions, and `moq-mavlink` ships as a -moq-cli gateway like moq-srt. If a second consumer appears, generalize -an announce-prefix fan-in helper into moq-net instead of adding a robot -crate. Where the text below says "the crate", read "the conventions". - -Decided 2026-10-08: parked in m5. No teleoperation consumer needs the crate -yet; telemetry beside video needs [data sync](/quest/m3/watch-data-sync.md), -not this primitive. - -### Direction - -The operator publishes and the robot subscribes. The robot serves its own -broadcast and subscribes an announce prefix for controllers; observers get the -control stream for free, and nothing needs relay support that does not already -exist. - -| Broadcast | Tracks | -|---|---| -| `robot/` | `video`, `telemetry` (lossy), `rpc` (reliable) | -| `control//` | `command` (lossy), `rpc` (reliable) | - -### Two delivery classes, and both already exist - -`moq-json` implements exactly this split, and the crate reuses it rather than -re-deriving it from track knobs: `snapshot` is lossy (one value updated over -time, intermediate updates collapsed, older groups dropped) and `stream` is -lossless (an ordered append-log where nothing is superseded). - -The use-cases draft says the same thing in terms of framing -(`drafts/draft-lcurley-moq-use-cases.md`, "Interaction"): a GROUP per input for -latency, a single GROUP with a FRAME per input for reliability. A group is one -QUIC stream (`open_uni`, `rs/moq-net/src/lite/publisher.rs`), so frames inside -it are ordered and delivered exactly once. - -That guarantee is scoped, and the crate must say so rather than promise -losslessness. A group holds at most `MAX_CACHE_BYTES` and `MAX_GROUP_FRAMES` -(`rs/moq-net/src/model/group.rs`). A raw group write past either aborts it -with `Error::GroupTooLarge`, but the JSON stream helper refuses an append -that might exceed the budget before encoding, leaving the log intact and -writable (`rs/moq-json/src/stream/mod.rs`). It never rolls its one group; -when its budget is spent the publisher opens a new track. A link drop or -reconnect loses what was in flight. -Those caps hold until [Unbounded groups](/quest/m2/unbounded-groups/README.md) -lands; after it, a lagging reader boards a flight if the publisher serves -replay, and otherwise refreshes or sees `Evicted`. -So the contract is ordered, gap-free delivery for a live reader within one -bounded log, and recovery after a reconnect belongs to the application -protocol. That is an acceptable division for MAVLink, whose mission, parameter -and file-transfer services already carry their own stop-and-wait -retransmission, but it must be stated, not assumed. - -The framing is where the guarantee lives, not the subscription flags: - -- `track::Subscriber::ordered()` returns an `Ordered` handle that reads the - groups it receives in sequence order. It is a local cursor, not a delivery - guarantee: groups that aged out or were skipped never arrive. A class built - on it would not be reliable, which is why the reliable class is a single - group instead. -- What makes the lossy class lossy on the wire is the publisher's - `Info::max_age`: `evict_expired_scan` aborts an aged-out group with `Error::Old` - (`rs/moq-net/src/model/track.rs`), and an abort resets the QUIC stream, so - stale bytes stop being retransmitted. -- A subscriber cannot weaken either class. `clamp_combined` - (`rs/moq-net/src/model/track.rs`) clamps the aggregate window down to - `Info::max_age`, and `Subscription::max_delay` already defaults to - `Duration::ZERO`, so a raw observer subscribing through `moq-net` neither - widens the window nor has to be prevented from trying. - -### Contents - -- The catalog entries. The hang catalog already advertises data tracks in its - `json` and `binary` sections (`rs/hang/src/catalog/{json,binary}.rs`), - written by the `moq-mux` data producers, and each entry carries `extra` - fields. Decide whether the robot's fields ride there or in a namespaced - root section through `moq-mux`'s `CatalogExt` and `RenditionConfig`. - No hang schema change. -- Announce-prefix fan-in, documented against moq-net's origin announce - stream (`origin::Consumer`'s announcements under a prefix). moq-boy no - longer reads viewer broadcasts (its viewers dial a socket endpoint), so - there is no fan-in code to move; a helper lands in moq-net only when a - second consumer needs one. -- The two delivery classes with the group structure and `Info::max_age` each - one needs: `moq-json`'s snapshot and stream modes for JSON; plain tracks - for opaque bytes, one group per latest value or one bounded ordered group - for the append-log. The [compression migration](https://github.com/moq-dev/moq/pull/7046) - deleted moq-flate's snapshot/stream helpers, so use the track's `compression` - flag when compression is useful, not those helpers. -- Per-stage timestamp instrumentation, generalised from moq-boy's `status` - track. Check it against the publisher-reported stats track - ([media stats](/doc/concept/stats.md#media-reports), moq#2734) before adding a - second stats surface. Capability only: publishing a competitive benchmark - is out of scope, because Transitive's breakdown puts camera plus USB at - roughly 100 ms of a 130 ms glass-to-glass total, so we would mostly be - measuring somebody's webcam. - -### Cross-track correlation - -Decided in the 2026-09-30 audit: correlation folds in here rather than being -its own quest. A command, the telemetry sample it produced, and the video frame -showing the result share one timebase, so a recording is usable as training -data and an operator sees what the machine actually saw. - -The bridge is each broadcast's fixed catalog `media.clock: { wall, timescale }`, -documented in `doc/concept/hang.md` (#4461). Media and command tracks keep -their own timescales; convert PTS explicitly into the broadcast clock before -joining samples. The robot's video and telemetry share a clock; the operator's -command broadcast supplies its own mapping. Command and telemetry tracks -declare `Timescale::MILLI` explicitly: an undeclared Rust timescale means -untimed, so a track that leaves it out carries no timestamps to join. - -The clock assumption: the two hosts' wall clocks are synchronized by the -deployment, not by the library. State this beside the API, since a join across -unsynchronized hosts looks valid while being wrong. Report whether a mapping is -present, but never infer synchronization from its presence. No per-record -anchors or clock-sync mechanism; see -[#2278](https://github.com/moq-dev/moq/issues/2278). - -### Check against moq-boy - -With no crate there is nothing to port. Instead, check the conventions -against `moq-boy`, the only existing consumer, as the no-arbitration case: it -merges viewers' JSON button states, where the gateway routes MAVLink frames, -so if the documented shapes cannot describe its crowd control they are the -wrong shapes. moq-boy changes only where it contradicts a convention. - -## Related - -- [Unbounded groups](/quest/m2/unbounded-groups/README.md) - removes the group caps that bound the reliable class's one log -- [arbitration](/quest/m5/teleop-arbitration.md) - which controller is obeyed From 066d847b9697fee8f26c2f1b3f001d2639d6ba14 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 14 Nov 2026 09:29:03 -0800 Subject: [PATCH 2/3] quest: answer review of the remote desktop plan Define the refresh interval's clock and unit, give refresh() a request barrier, settle the first image of an empty track, and pin the portal session and cursor contracts the review found open. Co-Authored-By: Claude Sonnet 5.5 --- quest/m2/refresh-interval.md | 25 ++++++++++++++++++++----- quest/m4/capture-cursor.md | 10 +++++++++- quest/m4/capture-portal-session.md | 22 ++++++++++++++++++++-- quest/m4/hang-thumbnail.md | 10 +++++++++- 4 files changed, 58 insertions(+), 9 deletions(-) diff --git a/quest/m2/refresh-interval.md b/quest/m2/refresh-interval.md index fdd735a80e1..7bee164b023 100644 --- a/quest/m2/refresh-interval.md +++ b/quest/m2/refresh-interval.md @@ -3,16 +3,26 @@ ## Goal A publisher can declare in TRACK_INFO a refresh interval: a live join asks -for a new group only once the newest group started at least that long ago. +for a new group only once the newest group arrived at least that long ago. A reader can also ask for one with a public `refresh()` on the track subscriber, in Rust and `@moq/net`, judged by the same interval. Every hop coalesces requests and judges them against its own cache, so the publisher's interval caps how often new groups are made, by design rather than by a per-hop rate limiter. +The interval is a duration, encoded and read like `max_age`, so it means the +same on a timed and a scale-less track. A group's age is measured on the +judging node's own monotonic clock from when that node first received (or, at +the publisher, created) the group, as arrival-age expiry already measures it. +It never compares media timestamps, which carry no wall-clock meaning and, on +a scale-less track, no unit, and it adds no local clock to the model. + A track of one-frame groups, such as a [thumbnail](/quest/m4/hang-thumbnail.md), then produces a fresh group when a reader arrives or asks, and never more often than its interval, however many readers fan out behind the relays. +A track with no group yet needs no interval: its first reader asks for the +first group at once, as the first consumer of an empty track does today +(`wanted()` is `Some(0)`), and the interval applies from that group on. ## Plan @@ -41,7 +51,8 @@ Facts found while planning: - SUBSCRIBE_UPDATE's `Wanted Group` already carries a relay's coalesced demand upstream, and [time addressing](/quest/m2/time-addressing/lite07.md) turns it into a start time. Build on that rather than adding a second - field. + field. The request barrier above rides the request's arrival at each hop, + not that start time, so it needs no clock shared between hops. - `rs/moq-video/src/encode/producer.rs` forces a keyframe on a join; it keeps working on the byte ratio. @@ -51,9 +62,13 @@ Recommendations for the PR: a typed choice. A track may want both (video wants a keyframe when the GOP is long, and not more than once a second), so recommend both, either one firing within the interval. -- `refresh()` asks for a group starting after the newest one the reader - holds, and resolves when one arrives, which the interval may delay. It - never resolves with a group no newer than the one the reader held. +- `refresh()` has a request barrier: it resolves only with a group that + reached the answering node after that node received the request, so a + group made before an action settled never answers a request made after it. + A relay answers from its cache only with such a group, else it passes the + request upstream, and the publisher makes one. The interval may delay the + new group, never substitute an older one. Rust and `@moq/net` share this + contract. - Update the lite draft (TRACK_INFO, SUBSCRIBE_UPDATE, and the changelog), `doc/concept/moq-lite.md`, and the matching JS. Run `just test interop --all`. diff --git a/quest/m4/capture-cursor.md b/quest/m4/capture-cursor.md index 7d60a00ffca..ff3888289a0 100644 --- a/quest/m4/capture-cursor.md +++ b/quest/m4/capture-cursor.md @@ -39,7 +39,15 @@ What each system offers, to confirm in the PR: works, and say what does not. Recommendations: a stream beside the frames, in the capture's own coordinate -space and clock, with each shape sent once and named by id after. +space and clock, with each shape sent once and named by id after. The report +is the current state (shape and position) plus change notifications, not an +event log, so a viewer that joins or reconnects reads the current shape before +any position that names it. Whatever carries it should start each shape's group +with that shape, so a late reader at the newest group needs no replay. +On a Wayland session the caller opened, the metadata cursor mode must have +been selected before the session started +([the portal session](/quest/m4/capture-portal-session.md)); another mode is +`Unsupported`. Public API: additive. Wire: none. diff --git a/quest/m4/capture-portal-session.md b/quest/m4/capture-portal-session.md index cc8a938a145..5aa30c561f9 100644 --- a/quest/m4/capture-portal-session.md +++ b/quest/m4/capture-portal-session.md @@ -25,9 +25,26 @@ restore token itself. Recommendations for the PR: -- Take the fd and node (and the stream's size and position from the portal) +- Take the stream the portal describes (node id and the stream's properties) rather than an ashpd session type, so moq-video does not choose the - caller's portal library and the input side stays out of moq. + caller's portal library and the input side stays out of moq. Forward the + stream's `mapping_id` unchanged, since it is what matches the stream to the + caller's libei region, and take `position` and `size` as optional: the + portal sends `position` only for monitor streams and `size` is optional, so + a window stream has no position. Frame pixel size comes from the negotiated + PipeWire format, never from the portal's compositor-space `size`, and a + missing value is reported to the caller as absent, not guessed. +- Take a handle that yields a fresh PipeWire remote fd for each open (the + caller's `OpenPipeWireRemote` on its session), not one fd. The PipeWire + connection consumes its fd, and capture reopens after a format change and + for each on-demand viewer, so a single fd serves only the first open. A + one-method trait the caller implements fits better than a callback + parameter; settle its shape in the PR. +- The caller selects the metadata cursor mode in `SelectSources` before it + starts the session, because the portal fixes the cursor mode there. The + stream description carries the mode the caller chose, and a + [cursor](/quest/m4/capture-cursor.md) request on a session with another + mode is `Unsupported` at setup, naming the mode. - The caller owns the session's lifetime and restore token. A capture on a session that ends reports the end like any other stopped capture. - Test against a headless compositor in CI if one runs the RemoteDesktop @@ -41,4 +58,5 @@ Public API: additive. Wire: none. ## Related +- [The cursor apart from the frame](/quest/m4/capture-cursor.md) - needs the metadata cursor mode this session carries - [Linux capture parity](/quest/m3/capture-linux.md) - the rest of Wayland capture diff --git a/quest/m4/hang-thumbnail.md b/quest/m4/hang-thumbnail.md index 6418ea496e0..b033f063077 100644 --- a/quest/m4/hang-thumbnail.md +++ b/quest/m4/hang-thumbnail.md @@ -10,7 +10,8 @@ A moq-video producer scales and encodes the newest captured or decoded frame into each rendition whenever its track [refreshes](/quest/m2/refresh-interval.md), so a reader that arrives or asks gets a current still, and the publisher encodes no more than one per -interval. +interval. Before any reader has asked there is no still: the first reader's +join asks for the first one at once, and the interval applies from then on. ## Plan @@ -43,6 +44,13 @@ Recommendations for the PR: - Timestamps follow the root's `media.clock`, so a still lines up with the video frame it came from. +- Each still is encoded after the request that asks for it, from the newest + frame the capture has delivered by then, and never reuses an earlier + encode, so it meets the `refresh()` request barrier. A capture that + delivers frames only on change (PipeWire does) has its last frame as the + current screen, so the producer must not wait for a newer one. A caller + that needs the result of an action calls `refresh()` once the action has + settled. - The producer refuses, at setup, a mime it cannot encode, rather than publishing a rendition it never fills. PNG and JPEG first. Take a dependency for encoding only at its newest stable version, and only if it From 30b6d84ab86bbfd1a4e3410fa3f3950ca718402d Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 14 Nov 2026 09:39:31 -0800 Subject: [PATCH 3/3] quest: make the refresh barrier causal and stamp stills by sampling time Co-Authored-By: Claude Sonnet 5.5 --- quest/m2/refresh-interval.md | 26 +++++++++++++++++--------- quest/m4/hang-thumbnail.md | 9 +++++++-- 2 files changed, 24 insertions(+), 11 deletions(-) diff --git a/quest/m2/refresh-interval.md b/quest/m2/refresh-interval.md index 7bee164b023..da89cbe3bd5 100644 --- a/quest/m2/refresh-interval.md +++ b/quest/m2/refresh-interval.md @@ -51,8 +51,8 @@ Facts found while planning: - SUBSCRIBE_UPDATE's `Wanted Group` already carries a relay's coalesced demand upstream, and [time addressing](/quest/m2/time-addressing/lite07.md) turns it into a start time. Build on that rather than adding a second - field. The request barrier above rides the request's arrival at each hop, - not that start time, so it needs no clock shared between hops. + field. The request barrier above rides the update numbers, not that start + time, so it needs no clock shared between hops. - `rs/moq-video/src/encode/producer.rs` forces a keyframe on a join; it keeps working on the byte ratio. @@ -62,13 +62,21 @@ Recommendations for the PR: a typed choice. A track may want both (video wants a keyframe when the GOP is long, and not more than once a second), so recommend both, either one firing within the interval. -- `refresh()` has a request barrier: it resolves only with a group that - reached the answering node after that node received the request, so a - group made before an action settled never answers a request made after it. - A relay answers from its cache only with such a group, else it passes the - request upstream, and the publisher makes one. The interval may delay the - new group, never substitute an older one. Rust and `@moq/net` share this - contract. +- `refresh()` has a request barrier, and it must be causal at the publisher: + a group already in flight when the request reached a relay would pass an + arrival check while showing the state before the request. So a group + answers a request only if the publisher created it after receiving that + request. Recommend reusing SUBSCRIBE_UPDATE's update number: each hop + forwards a request as a new update after it receives it (coalescing + requests that arrive between two forwards), the publisher tags each group + it creates with the highest update it had seen, and `refresh()` resolves + only with a group tagged at or past the update its request rode. The PR + settles the field's place in the draft and measures its cost; if it costs + more wire than it earns, weaken the guarantee to "a group that reached the + node after the request", state that post-action freshness is then best + effort, and change the thumbnail's promise to match. The interval may + delay the new group, never substitute an older one. Rust and `@moq/net` + share this contract. - Update the lite draft (TRACK_INFO, SUBSCRIBE_UPDATE, and the changelog), `doc/concept/moq-lite.md`, and the matching JS. Run `just test interop --all`. diff --git a/quest/m4/hang-thumbnail.md b/quest/m4/hang-thumbnail.md index b033f063077..3f798a709d5 100644 --- a/quest/m4/hang-thumbnail.md +++ b/quest/m4/hang-thumbnail.md @@ -42,8 +42,13 @@ Facts found while planning: Recommendations for the PR: -- Timestamps follow the root's `media.clock`, so a still lines up with the - video frame it came from. +- A group's identity is its start timestamp, and the same identity must never + carry different content. So each still is stamped with the time it was + sampled, on the root's `media.clock`, strictly after the previous still's + (one tick later when two samplings collide), and never with the source + frame's own timestamp: a static screen would give two refreshes the same + frame and so the same start. The still shows the newest frame at that + time, so it lines up with the video at its timestamp. - Each still is encoded after the request that asks for it, from the newest frame the capture has delivered by then, and never reuses an earlier encode, so it meets the `refresh()` request barrier. A capture that