diff --git a/quest/m2/refresh-interval.md b/quest/m2/refresh-interval.md new file mode 100644 index 00000000000..da89cbe3bd5 --- /dev/null +++ b/quest/m2/refresh-interval.md @@ -0,0 +1,97 @@ +# [L] Refresh a track by interval, and on request + +## Goal + +A publisher can declare in TRACK_INFO a refresh interval: a live join asks +for a new group only once the newest group arrived at least that long ago. +A reader can also ask for one with a public `refresh()` on the track +subscriber, in Rust and `@moq/net`, judged by the same interval. Every hop +coalesces requests and judges them against its own cache, so the +publisher's interval caps how often new groups are made, by design rather +than by a per-hop rate limiter. + +The interval is a duration, encoded and read like `max_age`, so it means the +same on a timed and a scale-less track. A group's age is measured on the +judging node's own monotonic clock from when that node first received (or, at +the publisher, created) the group, as arrival-age expiry already measures it. +It never compares media timestamps, which carry no wall-clock meaning and, on +a scale-less track, no unit, and it adds no local clock to the model. + +A track of one-frame groups, such as a [thumbnail](/quest/m4/hang-thumbnail.md), +then produces a fresh group when a reader arrives or asks, and never more +often than its interval, however many readers fan out behind the relays. +A track with no group yet needs no interval: its first reader asks for the +first group at once, as the first consumer of an empty track does today +(`wanted()` is `Some(0)`), and the interval applies from that group on. + +## Plan + +Planned 2026-11-13 (maintainer, /quest-plan of a remote desktop for +kixel.ai). The consumer is Kixel's desktop host, which serves an agent's +screenshot from a lossless thumbnail and wants dashboard previews from the +same track. + +Decided: + +- ✅ moq m2, in lite-07 (rejected: m4 and lite-08). Its consumer is later + work, which would put it in m4, but it changes lite-07-wip, and missing + that freeze costs a whole wire version. +- ✅ An interval in TRACK_INFO plus a public `refresh()` (rejected: changing + only the byte-ratio check; a per-hop rate limiter). The interval bounds + the request rate by what the publisher declares. + +Facts found while planning: + +- `Refresh` is a byte ratio today. `over_refresh` + (`rs/moq-net/src/model/group.rs`) tests `cache - first > first * ratio`, + so a complete one-frame group never refreshes at any ratio. +- `Subscriber::join` is `pub(crate)` in `rs/moq-net/src/model/track.rs`. + The only trigger is a live-edge subscribe, and no reader can ask for a + group. +- SUBSCRIBE_UPDATE's `Wanted Group` already carries a relay's coalesced + demand upstream, and [time addressing](/quest/m2/time-addressing/lite07.md) + turns it into a start time. Build on that rather than adding a second + field. The request barrier above rides the update numbers, not that start + time, so it needs no clock shared between hops. +- `rs/moq-video/src/encode/producer.rs` forces a keyframe on a join; it + keeps working on the byte ratio. + +Recommendations for the PR: + +- Whether the interval sits beside the byte ratio or replaces `Refresh` with + a typed choice. A track may want both (video wants a keyframe when the GOP + is long, and not more than once a second), so recommend both, either one + firing within the interval. +- `refresh()` has a request barrier, and it must be causal at the publisher: + a group already in flight when the request reached a relay would pass an + arrival check while showing the state before the request. So a group + answers a request only if the publisher created it after receiving that + request. Recommend reusing SUBSCRIBE_UPDATE's update number: each hop + forwards a request as a new update after it receives it (coalescing + requests that arrive between two forwards), the publisher tags each group + it creates with the highest update it had seen, and `refresh()` resolves + only with a group tagged at or past the update its request rode. The PR + settles the field's place in the draft and measures its cost; if it costs + more wire than it earns, weaken the guarantee to "a group that reached the + node after the request", state that post-action freshness is then best + effort, and change the thumbnail's promise to match. The interval may + delay the new group, never substitute an older one. Rust and `@moq/net` + share this contract. +- Update the lite draft (TRACK_INFO, SUBSCRIBE_UPDATE, and the changelog), + `doc/concept/moq-lite.md`, and the matching JS. Run `just test interop + --all`. + +Tests: a relay chain where many readers join and call `refresh()` on a +one-frame track produces one group per interval at the publisher, with time +mocked. + +Public API: additive (`refresh()`, the interval on `Info`). Wire: +lite-07-wip TRACK_INFO and SUBSCRIBE_UPDATE. + +## Required + +- [The lite-07 wire addresses groups by start](/quest/m2/time-addressing/lite07.md) - `Wanted Group` becomes the start time this requests by + +## Related + +- [Hang thumbnails](/quest/m4/hang-thumbnail.md) - the one-frame track this refreshes diff --git a/quest/m3/mesh/README.md b/quest/m3/mesh/README.md index 268b1d25a54..59b2b64d17a 100644 --- a/quest/m3/mesh/README.md +++ b/quest/m3/mesh/README.md @@ -135,4 +135,3 @@ Decided 2026-11-12 (maintainer, /quest-audit before the cut): - [P2P](/quest/m4/p2p/README.md) - internet peer sessions, which the m3 mesh does not use - [Mesh over ICE](/quest/m5/mesh-ice.md) - deferred direct paths between mesh nodes on different networks - [End-to-end encryption](/quest/m5/e2ee/README.md) - the AEAD the encryption level will reuse -- [Teleoperation](/quest/m5/teleop-robot.md) - robot track conventions that could ride the mesh diff --git a/quest/m4/capture-cursor.md b/quest/m4/capture-cursor.md new file mode 100644 index 00000000000..ff3888289a0 --- /dev/null +++ b/quest/m4/capture-cursor.md @@ -0,0 +1,58 @@ +# [L] The cursor apart from the frame + +## Goal + +A screen capture in `moq_video::capture` can report the cursor separately +instead of drawing it into each frame: its position whenever it moves, and +its image and hotspot whenever the shape changes. A viewer can then draw the +pointer itself, with no round trip of lag. Where a system cannot report the +cursor's shape or position, asking for it is an `Unsupported` error naming +the reason, decided at setup. + +## Plan + +Planned 2026-11-13 (maintainer, /quest-plan of a remote desktop for +kixel.ai). The consumer is Kixel's desktop page, which draws the remote +pointer locally so a human's aim does not trail by the round-trip time, as +RDP and Parsec do. The track that carries it is Kixel's convention, not +hang's, until a second consumer needs it. + +Decided: ✅ a separate cursor (rejected: drawn into the frame, with the +viewer's own pointer shown on top). Milestone: ✅ m4 (rejected: m3), since +the consumer is Kixel's later (m2) plans. + +Today `capture::Config::cursor` is a bool that draws the cursor in or leaves +it out: the portal's embedded mode, `SetIsCursorCaptureEnabled` on WGC, +`setShowsCursor` on macOS, and a CPU blend of the XFixes cursor on X11. + +What each system offers, to confirm in the PR: + +- **Wayland:** the portal's metadata cursor mode puts position and bitmap in + each PipeWire buffer's metadata. +- **X11:** XFixes reports shape changes and their image; position by query + or XInput2 motion events. +- **Windows:** DXGI Desktop Duplication gives pointer shape and position, but + capture moved to WGC, which has neither. `GetCursorInfo` plus the cursor's + icon may do. +- **macOS:** position from `CGEvent`. No public API gives another app's + cursor image (`NSCursor.currentSystemCursor` is deprecated). Find out what + works, and say what does not. + +Recommendations: a stream beside the frames, in the capture's own coordinate +space and clock, with each shape sent once and named by id after. The report +is the current state (shape and position) plus change notifications, not an +event log, so a viewer that joins or reconnects reads the current shape before +any position that names it. Whatever carries it should start each shape's group +with that shape, so a late reader at the newest group needs no replay. +On a Wayland session the caller opened, the metadata cursor mode must have +been selected before the session started +([the portal session](/quest/m4/capture-portal-session.md)); another mode is +`Unsupported`. + +Public API: additive. Wire: none. + +## Hosts + +- linux - the portal metadata cursor and XFixes +- macos - the cursor without a public shape API +- windows - the cursor under WGC diff --git a/quest/m4/capture-portal-session.md b/quest/m4/capture-portal-session.md new file mode 100644 index 00000000000..5aa30c561f9 --- /dev/null +++ b/quest/m4/capture-portal-session.md @@ -0,0 +1,62 @@ +# [M] Capture from the caller's portal session + +## Goal + +`moq_video::capture` can open a screen capture on a portal session the caller +already holds: a PipeWire fd and node id from an XDG RemoteDesktop session +that also started its screencast. One consent dialog then covers both seeing +and controlling the screen, and the caller's input lands in the same +coordinate space as the frames. + +## Plan + +Planned 2026-11-13 (maintainer, /quest-plan of a remote desktop for +kixel.ai). The consumer is Kixel's Wayland desktop host, which injects +keyboard and mouse through libei on a RemoteDesktop session +(`ConnectToEIS`). A ScreenCast session opened separately by moq-video would +ask the user a second time and could pick another monitor than the one +being controlled. + +Milestone: ✅ m4 (rejected: m3). The consumer is Kixel's later (m2) plans. + +Today `capture/pipewire.rs` opens its own ScreenCast session through ashpd, +with only the `screencast` and `camera` features enabled, and keeps the +restore token itself. + +Recommendations for the PR: + +- Take the stream the portal describes (node id and the stream's properties) + rather than an ashpd session type, so moq-video does not choose the + caller's portal library and the input side stays out of moq. Forward the + stream's `mapping_id` unchanged, since it is what matches the stream to the + caller's libei region, and take `position` and `size` as optional: the + portal sends `position` only for monitor streams and `size` is optional, so + a window stream has no position. Frame pixel size comes from the negotiated + PipeWire format, never from the portal's compositor-space `size`, and a + missing value is reported to the caller as absent, not guessed. +- Take a handle that yields a fresh PipeWire remote fd for each open (the + caller's `OpenPipeWireRemote` on its session), not one fd. The PipeWire + connection consumes its fd, and capture reopens after a format change and + for each on-demand viewer, so a single fd serves only the first open. A + one-method trait the caller implements fits better than a callback + parameter; settle its shape in the PR. +- The caller selects the metadata cursor mode in `SelectSources` before it + starts the session, because the portal fixes the cursor mode there. The + stream description carries the mode the caller chose, and a + [cursor](/quest/m4/capture-cursor.md) request on a session with another + mode is `Unsupported` at setup, naming the mode. +- The caller owns the session's lifetime and restore token. A capture on a + session that ends reports the end like any other stopped capture. +- Test against a headless compositor in CI if one runs the RemoteDesktop + portal; otherwise a by-hand run on GNOME and KDE, recorded in the PR. + +Public API: additive. Wire: none. + +## Hosts + +- linux - the XDG portal and PipeWire + +## Related + +- [The cursor apart from the frame](/quest/m4/capture-cursor.md) - needs the metadata cursor mode this session carries +- [Linux capture parity](/quest/m3/capture-linux.md) - the rest of Wayland capture diff --git a/quest/m4/hang-thumbnail.md b/quest/m4/hang-thumbnail.md new file mode 100644 index 00000000000..3f798a709d5 --- /dev/null +++ b/quest/m4/hang-thumbnail.md @@ -0,0 +1,70 @@ +# [L] Hang thumbnails + +## Goal + +A hang broadcast can carry still images in a new `thumbnail` catalog +section: renditions keyed by name, each with a mime (`image/png` for +lossless, and lossy formats such as `image/jpeg`), a width, and a height, +one image per group. Rust (`hang`) and `@moq/hang` both read and write it. +A moq-video producer scales and encodes the newest captured or decoded +frame into each rendition whenever its track +[refreshes](/quest/m2/refresh-interval.md), so a reader that arrives or asks +gets a current still, and the publisher encodes no more than one per +interval. Before any reader has asked there is no still: the first reader's +join asks for the first one at once, and the interval applies from then on. + +## Plan + +Planned 2026-11-13 (maintainer, /quest-plan of a remote desktop for +kixel.ai). The consumer is Kixel's desktop host: an agent's screenshot is a +lossless rendition, refreshed after each action settles, and the dashboard's +previews are a small lossy one. + +Decided: + +- ✅ A `thumbnail` section in the hang schema (rejected: the existing + `binary` section with `mime: image/png`). Stills get typed fields (size, + format), and several renditions, as video has. +- ✅ The schema and the moq-video producer in one quest (rejected: schema + only). Rooms and the studio would otherwise each write the encoder again. +- ✅ m4 (rejected: m3). Additive, and the consumer is Kixel's later (m2) + plans. + +Facts found while planning: + +- Both catalog parsers tolerate unknown sections (`z.looseObject` in + `js/hang/src/catalog/root.ts`, and serde without `deny_unknown_fields`), so + the section is additive on the wire too. +- `rs/hang/src/catalog/root.rs` already has a test with + `"thumbnail": {"mode":"snapshot","mime":"image/jpeg"}` under `binary`. + Replace it with the new section rather than keeping two ways to say the + same thing. + +Recommendations for the PR: + +- A group's identity is its start timestamp, and the same identity must never + carry different content. So each still is stamped with the time it was + sampled, on the root's `media.clock`, strictly after the previous still's + (one tick later when two samplings collide), and never with the source + frame's own timestamp: a static screen would give two refreshes the same + frame and so the same start. The still shows the newest frame at that + time, so it lines up with the video at its timestamp. +- Each still is encoded after the request that asks for it, from the newest + frame the capture has delivered by then, and never reuses an earlier + encode, so it meets the `refresh()` request barrier. A capture that + delivers frames only on change (PipeWire does) has its last frame as the + current screen, so the producer must not wait for a newer one. A caller + that needs the result of an action calls `refresh()` once the action has + settled. +- The producer refuses, at setup, a mime it cannot encode, rather than + publishing a rendition it never fills. PNG and JPEG first. Take a + dependency for encoding only at its newest stable version, and only if it + earns its build time. +- Readers, such as a watch poster image or a room's grid, are follow-ups + once someone needs them. + +Public API: additive. Wire: the hang catalog gains an optional section. + +## Required + +- [Refresh a track by interval, and on request](/quest/m2/refresh-interval.md) - what makes a one-image group refresh on demand diff --git a/quest/m5/teleop-arbitration.md b/quest/m5/teleop-arbitration.md deleted file mode 100644 index 4fd076cd8b1..00000000000 --- a/quest/m5/teleop-arbitration.md +++ /dev/null @@ -1,45 +0,0 @@ -# [M] Operator arbitration - -## Goal - -Exactly one controller commands a vehicle at a time, with explicit handoff, and -the boundary between arbitration and authorization is stated rather than -assumed. - -## Plan - -Decided 2026-10-08: parked in m5 with [robot](/quest/m5/teleop-robot.md), -which it builds on. - -`moq-boy` merges every viewer's input, which is right for crowd control and -wrong for a vehicle. Arbitration is the part of a teleoperation stack that -cannot be lifted from it: one controller is active, the others are read-only, -and handoff is explicit rather than last-writer-wins. - -### Arbitration is not authorization - -The robot subscribes an announce prefix that is a sibling of what it publishes, -so on its own nothing stops anyone able to announce under `control//` from -joining the pool. `moq-boy`'s fan-in is safe because a hostile viewer can only -press B; a vehicle is not that. - -Admission belongs to the relay as a path-scoped grant, not to this crate. -Scoped signing keys landed in moq#2416 (`6b86e612b`), so the enforcing -mechanism already exists: `Scope` in `rs/moq-auth/src/claims.rs` and -`Key::with_scope` in `key.rs` bind a key to publish and subscribe path -prefixes. Write the concrete claims down (a robot publishes `robot/` and -subscribes `control//`; an operator publishes `control//` and -subscribes `robot/`) and prove with a relay test that a key scoped -elsewhere cannot announce under `control//`. Documenting the requirement is -not an acceptable outcome now that enforcement ships. Revoking a superseded or -compromised operator's grant mid-session is relay auth work, not something -this crate implements; scoped keys only gate admission. - -Link loss is the other half: a class whose contract is "drop stale samples" -needs a stated behavior when samples stop arriving, and the vehicle's own -failsafe is what actually holds. Say what the crate reports and what it does -not attempt. - -## Required - -- [Robot teleoperation conventions](/quest/m5/teleop-robot.md) diff --git a/quest/m5/teleop-mavlink.md b/quest/m5/teleop-mavlink.md deleted file mode 100644 index 6f653a91bb3..00000000000 --- a/quest/m5/teleop-mavlink.md +++ /dev/null @@ -1,93 +0,0 @@ -# [L] MAVLink bridge - -## Goal - -A `moq-mavlink` gateway replaces the VPN plus two unmanaged UDP flows that -every cellular ArduPilot vehicle runs today. QGroundControl, Mission Planner -and MAVProxy connect unchanged. - -## Plan - -Parked in m5 until a real ArduPilot user or partner asks (2026-09-30 audit). - -Decided 2026-10-08: the lossy class is a snapshot track per key with a -bounded key set, since one snapshot track carries only one latest value. - -### Shape - -A library crate reached through `moq-cli`, the way `moq-srt` and `moq-rtmp` -already are (neither is a binary; both are crates behind subcommands). - -Each side is bidirectional on one session. The vehicle side publishes what the -flight controller emits and subscribes the operator's commands to write back -into it; the operator side publishes commands and subscribes telemetry to serve -UDP 14550, because QGC speaks UDP and TCP only and its forwarding is one-way. -Neither side is a one-way media import or export, so do not let the subcommand -naming imply one leg. - -### Route by msgid, never parse payloads - -Read the header only: the 24-bit msgid in v2, 8-bit in v1. Payloads stay opaque -bytes and route to a delivery class through a config table, with an unknown -msgid falling to a default. Every track declares `Timescale::MILLI` -explicitly, since an undeclared Rust timescale means untimed. - -This is what makes frame-aware routing free of the dialect tax. CRC_EXTRA (the -per-message schema hash) is only needed to validate a frame, never to read its -header, so custom dialects, new messages and version skew all just work. The -alternative, compiling in `ardupilotmega` the way `mavlink-camera-manager` and -`mavlink-server` do, buys per-field logic we do not need and couples us to a -schema that changes without us. - -### One lossy track for every msgid would starve the slow ones - -Latest-wins is per track. A single lossy track carrying every routed msgid as -its own group lets 50 Hz `ATTITUDE` supersede a 1 Hz `HEARTBEAT` or a GPS fix, -because the newest group wins regardless of which message it holds. Putting -them all in one long-lived group has the opposite failure: nothing can be -skipped and head-of-line blocking is back. - -So the lossy class is a plain track carrying the latest opaque frame, with -one self-contained group per update, so a newer value -never waits behind an older one. One detail decides whether it actually -delivers latest-value: - -- **Key on `(sysid, compid, msgid)`, not msgid alone.** A vehicle is several - components, and an autopilot, a gimbal and a camera all emit `HEARTBEAT` - under msgid 0; keying on msgid alone lets them overwrite each other. Decide - explicitly what to do about instances distinguished only inside a payload, - which the msgid-only rule cannot see. -- **One latest-value track per key.** Per-key latest-value means a track per - `(sysid, compid, msgid)`, named from the key and advertised in the catalog - as it appears. That state grows with whatever the flight-controller link - emits, so bound it: cap the number of keys (or derive the set from the - config table) and fail loud past the cap rather than creating tracks - without limit. - -Not `moq_json::snapshot`: a MAVLink frame is not JSON, and its default batches -up to `MAX_DELTA_FRAMES` merge patches into one ordered group, so an earlier -50 Hz attitude delta would head-of-line block a later heartbeat. - -The reliable class is one bounded ordered group on a plain track: commands, -ACKs, mission, parameter and file transfer are stop-and-wait exchanges carried -in one ordered group. Note the scope in -[robot](/quest/m5/teleop-robot.md): that is gap-free for a live reader, not -recoverable after a lag or a reconnect, so these MAVLink services keep relying -on their own retransmission across a link drop. - -The [compression migration](https://github.com/moq-dev/moq/pull/7046) deleted the -moq-flate snapshot and stream helpers. Keep these framing rules on plain -tracks, setting `compression` when useful, rather than depending on those helpers. - -### Deliberately out of scope - -Advertising the stream in-band. No `VIDEO_STREAM_TYPE` value can describe a -MoQ endpoint, so a GCS discovers the link out of band. Check the current -enumeration when this is picked up: it grew a WHEP entry after the original -four, which is the precedent that adding one upstream in `common.xml` is a -small, real contribution and its own future quest. - -## Required - -- [Robot teleoperation conventions](/quest/m5/teleop-robot.md) -- [Operator arbitration](/quest/m5/teleop-arbitration.md) diff --git a/quest/m5/teleop-proof.md b/quest/m5/teleop-proof.md deleted file mode 100644 index 4bdb3d6e483..00000000000 --- a/quest/m5/teleop-proof.md +++ /dev/null @@ -1,40 +0,0 @@ -# [M] SITL proof and browser ground station - -## Goal - -ArduPilot SITL plus a synthetic camera, flown from a browser ground station -through a real relay, with no hardware. Someone with a laptop reproduces it in -five minutes. - -## Plan - -Parked in m5 until a real ArduPilot user or partner asks (2026-09-30 audit). - -The browser client subscribes video and telemetry and publishes control, the -shape Blue Robotics' Cockpit proves is viable. It is the demo and the -end-to-end proof of the primitive, not a bid to out-feature QGroundControl: -real users reach the same link through the `moq-mavlink` gateway. - -SITL is the right proof surface precisely because it removes the camera, which -otherwise dominates the latency budget and would make the demo a measurement of -somebody's webcam. - -The ground station's catalog and delivery-class shapes come from a zod -schema mirroring the documented robot conventions, written once and extending the root -schema through the same seam `js/hang/src/catalog/root.ts` uses (there is no -`@moq/mux`). Decided 2026-10-08: no separate `@moq/robot` package until a -second browser consumer needs it; the schema folds in here. A browser -observer cannot degrade the operator's classes (the publisher's -`Info::max_age` bounds the window), so the schema is for reuse, not safety. - -Standing SITL up in CI is separate work with its own build dependencies. -Reproducibility by hand is the bar here; automate it later if it proves worth -the maintenance. - -## Required - -- [MAVLink bridge](/quest/m5/teleop-mavlink.md) - the gateway the vehicle side runs - -## Related - -- [Robot teleoperation conventions](/quest/m5/teleop-robot.md) - the conventions the zod schema mirrors diff --git a/quest/m5/teleop-robot.md b/quest/m5/teleop-robot.md deleted file mode 100644 index e813c769696..00000000000 --- a/quest/m5/teleop-robot.md +++ /dev/null @@ -1,104 +0,0 @@ -# [S] Robot teleoperation conventions - -## Goal - -The track layout, discovery, and timing every teleoperated machine needs are -documented conventions in `doc/`, beside the delivery shapes the -[teleoperation guide](/doc/concept/use-case/teleoperation.md) already -documents, so an integrator writes an adapter rather than a stack. There is no -`moq-robot` crate, and this quest ships documentation only; -[the MAVLink gateway](/quest/m5/teleop-mavlink.md) implements the conventions -in its own quest. It is the gate for the rest of the questline. - -## Plan - -Decided 2026-11-07 (maintainer, /quest-plan from the API sweep of open PRs and remaining quests): ✅ no `moq-robot` crate (rejected: a generic robot -crate, too niche for a published library with no second consumer). The -shapes below become documented conventions, and `moq-mavlink` ships as a -moq-cli gateway like moq-srt. If a second consumer appears, generalize -an announce-prefix fan-in helper into moq-net instead of adding a robot -crate. - -Decided 2026-10-08: parked in m5. No teleoperation consumer needs it yet; -telemetry beside video needs [data sync](/quest/m5/watch-data-sync.md), -not this primitive. - -Narrowed 2026-11-12 (/quest-audit): the teleoperation guide -(`doc/concept/use-case/teleoperation.md`, with `rs/moq-json/examples/teleop.rs`) -already documents the two delivery shapes (moq-json snapshot for replaceable -state, an ordered moq-json stream for commands), the group cache bound and -`Evicted`, a new command track per control session, command IDs and -acknowledgments, and that a connected subscriber is not an authorized -operator. What remains extends that guide rather than restating it. - -### Direction - -The operator publishes and the robot subscribes. The robot serves its own -broadcast and subscribes an announce prefix for controllers; observers get the -control stream for free, and nothing needs relay support that does not already -exist. - -| Broadcast | Tracks | -|---|---| -| `robot/` | `video`, `telemetry` (lossy), `rpc` (reliable) | -| `control//` | `command` (lossy), `rpc` (reliable) | - -### Contents - -- The broadcast and track layout above, and how a robot learns its operators - by announce prefix, documented against moq-net's origin announce stream - (`origin::Consumer`'s announcements under a prefix). moq-boy no longer reads - viewer broadcasts (its viewers dial a socket endpoint), so there is no - fan-in code to move; a helper lands in moq-net only when a second consumer - needs one. -- The catalog entries. The hang catalog already advertises data tracks in its - `json` and `binary` sections (`rs/hang/src/catalog/{json,binary}.rs`), and - each entry carries `extra` fields. Decide whether the robot's fields ride - there or in a namespaced root section through `moq-mux`'s catalog - extension. No hang schema change. -- Opaque bytes: plain tracks with one group per latest value for the lossy - class, or one bounded ordered group for the append-log, setting the track's - `compression` flag when useful. The lossy class is lossy on the wire because - the publisher's `Info::max_age` evicts an aged-out group with `Error::Old` - (`rs/moq-net/src/model/track.rs`), resetting its stream. -- Per-stage timestamp instrumentation, generalised from moq-boy's `status` - track (`rs/moq-boy/src/status.rs`). Check it against the publisher-reported - stats track ([media stats](/doc/concept/stats.md#media-reports)) before - adding a second stats surface. Capability only: publishing a competitive - benchmark is out of scope, since camera and USB dominate glass-to-glass - latency. - -### Cross-track correlation - -Decided in the 2026-09-30 audit: correlation folds in here rather than being -its own quest. A command, the telemetry sample it produced, and the video frame -showing the result share one timebase, so a recording is usable as training -data and an operator sees what the machine actually saw. - -The bridge is each broadcast's fixed catalog `media.clock`, documented in -`doc/concept/hang.md`. Media and command tracks keep their own timescales; -convert PTS explicitly into the broadcast clock before joining samples. The -robot's video and telemetry share a clock; the operator's command broadcast -supplies its own mapping. Command and telemetry tracks declare -`Timescale::MILLI` explicitly: an undeclared Rust timescale means untimed, so a -track that leaves it out carries no timestamps to join. - -The clock assumption: the two hosts' wall clocks are synchronized by the -deployment, not by the library. State this in the docs, since a join across -unsynchronized hosts looks valid while being wrong. Report whether a mapping is -present, but never infer synchronization from its presence. No per-record -anchors or clock-sync mechanism; see -[#2278](https://github.com/moq-dev/moq/issues/2278). - -### Check against moq-boy - -Check the conventions against `moq-boy`, the only existing consumer, as the -no-arbitration case: it merges viewers' JSON button states, where the gateway -routes MAVLink frames, so if the documented shapes cannot describe its crowd -control they are the wrong shapes. moq-boy changes only where it contradicts a -convention. - -## Related - -- [Unbounded groups](/quest/m3/unbounded-groups/README.md) - changes the group cache bound the reliable class's one log relies on -- [arbitration](/quest/m5/teleop-arbitration.md) - which controller is obeyed