Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
97 changes: 97 additions & 0 deletions quest/m2/refresh-interval.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# [L] Refresh a track by interval, and on request

## Goal

A publisher can declare in TRACK_INFO a refresh interval: a live join asks
for a new group only once the newest group arrived at least that long ago.
A reader can also ask for one with a public `refresh()` on the track
subscriber, in Rust and `@moq/net`, judged by the same interval. Every hop
coalesces requests and judges them against its own cache, so the
publisher's interval caps how often new groups are made, by design rather
than by a per-hop rate limiter.

The interval is a duration, encoded and read like `max_age`, so it means the
same on a timed and a scale-less track. A group's age is measured on the
judging node's own monotonic clock from when that node first received (or, at
the publisher, created) the group, as arrival-age expiry already measures it.
It never compares media timestamps, which carry no wall-clock meaning and, on
a scale-less track, no unit, and it adds no local clock to the model.

A track of one-frame groups, such as a [thumbnail](/quest/m4/hang-thumbnail.md),
then produces a fresh group when a reader arrives or asks, and never more
often than its interval, however many readers fan out behind the relays.
A track with no group yet needs no interval: its first reader asks for the
first group at once, as the first consumer of an empty track does today
(`wanted()` is `Some(0)`), and the interval applies from that group on.

## Plan

Planned 2026-11-13 (maintainer, /quest-plan of a remote desktop for
kixel.ai). The consumer is Kixel's desktop host, which serves an agent's
screenshot from a lossless thumbnail and wants dashboard previews from the
same track.

Decided:

- ✅ moq m2, in lite-07 (rejected: m4 and lite-08). Its consumer is later
work, which would put it in m4, but it changes lite-07-wip, and missing
that freeze costs a whole wire version.
- ✅ An interval in TRACK_INFO plus a public `refresh()` (rejected: changing
only the byte-ratio check; a per-hop rate limiter). The interval bounds
the request rate by what the publisher declares.

Facts found while planning:

- `Refresh` is a byte ratio today. `over_refresh`
(`rs/moq-net/src/model/group.rs`) tests `cache - first > first * ratio`,
so a complete one-frame group never refreshes at any ratio.
- `Subscriber::join` is `pub(crate)` in `rs/moq-net/src/model/track.rs`.
The only trigger is a live-edge subscribe, and no reader can ask for a
group.
- SUBSCRIBE_UPDATE's `Wanted Group` already carries a relay's coalesced
demand upstream, and [time addressing](/quest/m2/time-addressing/lite07.md)
turns it into a start time. Build on that rather than adding a second
field. The request barrier above rides the update numbers, not that start
time, so it needs no clock shared between hops.
- `rs/moq-video/src/encode/producer.rs` forces a keyframe on a join; it
keeps working on the byte ratio.

Recommendations for the PR:

- Whether the interval sits beside the byte ratio or replaces `Refresh` with
a typed choice. A track may want both (video wants a keyframe when the GOP
is long, and not more than once a second), so recommend both, either one
firing within the interval.
- `refresh()` has a request barrier, and it must be causal at the publisher:
a group already in flight when the request reached a relay would pass an
arrival check while showing the state before the request. So a group
answers a request only if the publisher created it after receiving that
request. Recommend reusing SUBSCRIBE_UPDATE's update number: each hop
forwards a request as a new update after it receives it (coalescing
requests that arrive between two forwards), the publisher tags each group
it creates with the highest update it had seen, and `refresh()` resolves
only with a group tagged at or past the update its request rode. The PR
settles the field's place in the draft and measures its cost; if it costs
more wire than it earns, weaken the guarantee to "a group that reached the
node after the request", state that post-action freshness is then best
effort, and change the thumbnail's promise to match. The interval may
delay the new group, never substitute an older one. Rust and `@moq/net`
share this contract.
- Update the lite draft (TRACK_INFO, SUBSCRIBE_UPDATE, and the changelog),
`doc/concept/moq-lite.md`, and the matching JS. Run `just test interop
--all`.

Tests: a relay chain where many readers join and call `refresh()` on a
one-frame track produces one group per interval at the publisher, with time
mocked.

Public API: additive (`refresh()`, the interval on `Info`). Wire:
lite-07-wip TRACK_INFO and SUBSCRIBE_UPDATE.

## Required

- [The lite-07 wire addresses groups by start](/quest/m2/time-addressing/lite07.md) - `Wanted Group` becomes the start time this requests by

## Related

- [Hang thumbnails](/quest/m4/hang-thumbnail.md) - the one-frame track this refreshes
1 change: 0 additions & 1 deletion quest/m3/mesh/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -135,4 +135,3 @@ Decided 2026-11-12 (maintainer, /quest-audit before the cut):
- [P2P](/quest/m4/p2p/README.md) - internet peer sessions, which the m3 mesh does not use
- [Mesh over ICE](/quest/m5/mesh-ice.md) - deferred direct paths between mesh nodes on different networks
- [End-to-end encryption](/quest/m5/e2ee/README.md) - the AEAD the encryption level will reuse
- [Teleoperation](/quest/m5/teleop-robot.md) - robot track conventions that could ride the mesh
58 changes: 58 additions & 0 deletions quest/m4/capture-cursor.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# [L] The cursor apart from the frame

## Goal

A screen capture in `moq_video::capture` can report the cursor separately
instead of drawing it into each frame: its position whenever it moves, and
its image and hotspot whenever the shape changes. A viewer can then draw the
pointer itself, with no round trip of lag. Where a system cannot report the
cursor's shape or position, asking for it is an `Unsupported` error naming
the reason, decided at setup.

## Plan

Planned 2026-11-13 (maintainer, /quest-plan of a remote desktop for
kixel.ai). The consumer is Kixel's desktop page, which draws the remote
pointer locally so a human's aim does not trail by the round-trip time, as
RDP and Parsec do. The track that carries it is Kixel's convention, not
hang's, until a second consumer needs it.

Decided: ✅ a separate cursor (rejected: drawn into the frame, with the
viewer's own pointer shown on top). Milestone: ✅ m4 (rejected: m3), since
the consumer is Kixel's later (m2) plans.

Today `capture::Config::cursor` is a bool that draws the cursor in or leaves
it out: the portal's embedded mode, `SetIsCursorCaptureEnabled` on WGC,
`setShowsCursor` on macOS, and a CPU blend of the XFixes cursor on X11.

What each system offers, to confirm in the PR:

- **Wayland:** the portal's metadata cursor mode puts position and bitmap in
each PipeWire buffer's metadata.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
- **X11:** XFixes reports shape changes and their image; position by query
or XInput2 motion events.
- **Windows:** DXGI Desktop Duplication gives pointer shape and position, but
capture moved to WGC, which has neither. `GetCursorInfo` plus the cursor's
icon may do.
- **macOS:** position from `CGEvent`. No public API gives another app's
cursor image (`NSCursor.currentSystemCursor` is deprecated). Find out what
works, and say what does not.

Recommendations: a stream beside the frames, in the capture's own coordinate
space and clock, with each shape sent once and named by id after. The report
is the current state (shape and position) plus change notifications, not an
event log, so a viewer that joins or reconnects reads the current shape before
any position that names it. Whatever carries it should start each shape's group
with that shape, so a late reader at the newest group needs no replay.
On a Wayland session the caller opened, the metadata cursor mode must have
been selected before the session started
([the portal session](/quest/m4/capture-portal-session.md)); another mode is
`Unsupported`.

Public API: additive. Wire: none.

## Hosts

- linux - the portal metadata cursor and XFixes
- macos - the cursor without a public shape API
- windows - the cursor under WGC
62 changes: 62 additions & 0 deletions quest/m4/capture-portal-session.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# [M] Capture from the caller's portal session

## Goal

`moq_video::capture` can open a screen capture on a portal session the caller
already holds: a PipeWire fd and node id from an XDG RemoteDesktop session
that also started its screencast. One consent dialog then covers both seeing
and controlling the screen, and the caller's input lands in the same
coordinate space as the frames.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved

## Plan

Planned 2026-11-13 (maintainer, /quest-plan of a remote desktop for
kixel.ai). The consumer is Kixel's Wayland desktop host, which injects
keyboard and mouse through libei on a RemoteDesktop session
(`ConnectToEIS`). A ScreenCast session opened separately by moq-video would
ask the user a second time and could pick another monitor than the one
being controlled.

Milestone: ✅ m4 (rejected: m3). The consumer is Kixel's later (m2) plans.

Today `capture/pipewire.rs` opens its own ScreenCast session through ashpd,
with only the `screencast` and `camera` features enabled, and keeps the
restore token itself.

Recommendations for the PR:

- Take the stream the portal describes (node id and the stream's properties)
rather than an ashpd session type, so moq-video does not choose the
caller's portal library and the input side stays out of moq. Forward the
stream's `mapping_id` unchanged, since it is what matches the stream to the
caller's libei region, and take `position` and `size` as optional: the
portal sends `position` only for monitor streams and `size` is optional, so
a window stream has no position. Frame pixel size comes from the negotiated
PipeWire format, never from the portal's compositor-space `size`, and a
missing value is reported to the caller as absent, not guessed.
- Take a handle that yields a fresh PipeWire remote fd for each open (the
caller's `OpenPipeWireRemote` on its session), not one fd. The PipeWire
connection consumes its fd, and capture reopens after a format change and
for each on-demand viewer, so a single fd serves only the first open. A
one-method trait the caller implements fits better than a callback
parameter; settle its shape in the PR.
- The caller selects the metadata cursor mode in `SelectSources` before it
starts the session, because the portal fixes the cursor mode there. The
stream description carries the mode the caller chose, and a
[cursor](/quest/m4/capture-cursor.md) request on a session with another
mode is `Unsupported` at setup, naming the mode.
- The caller owns the session's lifetime and restore token. A capture on a
session that ends reports the end like any other stopped capture.
- Test against a headless compositor in CI if one runs the RemoteDesktop
portal; otherwise a by-hand run on GNOME and KDE, recorded in the PR.

Public API: additive. Wire: none.

## Hosts

- linux - the XDG portal and PipeWire

## Related

- [The cursor apart from the frame](/quest/m4/capture-cursor.md) - needs the metadata cursor mode this session carries
- [Linux capture parity](/quest/m3/capture-linux.md) - the rest of Wayland capture
70 changes: 70 additions & 0 deletions quest/m4/hang-thumbnail.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# [L] Hang thumbnails

## Goal

A hang broadcast can carry still images in a new `thumbnail` catalog
section: renditions keyed by name, each with a mime (`image/png` for
lossless, and lossy formats such as `image/jpeg`), a width, and a height,
one image per group. Rust (`hang`) and `@moq/hang` both read and write it.
A moq-video producer scales and encodes the newest captured or decoded
frame into each rendition whenever its track
[refreshes](/quest/m2/refresh-interval.md), so a reader that arrives or asks
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Resolved
gets a current still, and the publisher encodes no more than one per
Comment thread
kixelated marked this conversation as resolved.
Resolved
interval. Before any reader has asked there is no still: the first reader's
join asks for the first one at once, and the interval applies from then on.

## Plan

Planned 2026-11-13 (maintainer, /quest-plan of a remote desktop for
kixel.ai). The consumer is Kixel's desktop host: an agent's screenshot is a
lossless rendition, refreshed after each action settles, and the dashboard's
previews are a small lossy one.

Decided:

- ✅ A `thumbnail` section in the hang schema (rejected: the existing
`binary` section with `mime: image/png`). Stills get typed fields (size,
format), and several renditions, as video has.
- ✅ The schema and the moq-video producer in one quest (rejected: schema
only). Rooms and the studio would otherwise each write the encoder again.
- ✅ m4 (rejected: m3). Additive, and the consumer is Kixel's later (m2)
plans.

Facts found while planning:

- Both catalog parsers tolerate unknown sections (`z.looseObject` in
`js/hang/src/catalog/root.ts`, and serde without `deny_unknown_fields`), so
the section is additive on the wire too.
- `rs/hang/src/catalog/root.rs` already has a test with
`"thumbnail": {"mode":"snapshot","mime":"image/jpeg"}` under `binary`.
Replace it with the new section rather than keeping two ways to say the
same thing.

Recommendations for the PR:

- A group's identity is its start timestamp, and the same identity must never
carry different content. So each still is stamped with the time it was
sampled, on the root's `media.clock`, strictly after the previous still's
(one tick later when two samplings collide), and never with the source
frame's own timestamp: a static screen would give two refreshes the same
frame and so the same start. The still shows the newest frame at that
time, so it lines up with the video at its timestamp.
- Each still is encoded after the request that asks for it, from the newest
frame the capture has delivered by then, and never reuses an earlier
encode, so it meets the `refresh()` request barrier. A capture that
delivers frames only on change (PipeWire does) has its last frame as the
current screen, so the producer must not wait for a newer one. A caller
that needs the result of an action calls `refresh()` once the action has
settled.
- The producer refuses, at setup, a mime it cannot encode, rather than
publishing a rendition it never fills. PNG and JPEG first. Take a
dependency for encoding only at its newest stable version, and only if it
earns its build time.
- Readers, such as a watch poster image or a room's grid, are follow-ups
once someone needs them.

Public API: additive. Wire: the hang catalog gains an optional section.

## Required

- [Refresh a track by interval, and on request](/quest/m2/refresh-interval.md) - what makes a one-image group refresh on demand
45 changes: 0 additions & 45 deletions quest/m5/teleop-arbitration.md

This file was deleted.

Loading