Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions quest/m3/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,7 @@ federation) land.
- [A published capture exposes its bandwidth grant](/quest/m3/video-publish-grant.md) - a moq-video publish shows each track's granted bitrate, not only its target
- [The test relay can enable stats](/quest/m3/relay-test-stats.md) - `moq_relay::test_relay` starts with stats on when asked
- [macOS capture listing futures are Send](/quest/m3/capture-shareable-send.md) - ScreenCaptureKit listing and start no longer hold a non-Send block across an await
- [A capture listing a picker can draw](/quest/m3/capture-list.md) - windows grouped by app with icons, camera kinds, low-rate pictures, and a wait for changes on macOS, Windows, and X11, for moq-dev/studio's Add page
- [Denied macOS system audio names its settings pane](/quest/m3/audio-system-denied-hint.md) - a permission error pointing at Screen & System Audio Recording
- [One encoder follows several connections' estimates](/quest/m3/bandwidth-multi-connection.md) - a track published to two relays tracks both estimates
- [Bandwidth reservations ignore in-process readers](/quest/m3/reservation-local-readers.md) - a local recorder no longer counts as uplink demand
Expand Down
1 change: 1 addition & 0 deletions quest/m3/capture-linux.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,3 +45,4 @@ unimplemented, because xdg-desktop-portal owns source selection:

- [Windows capture parity](/quest/m3/capture-windows.md) - the same gaps,
through Windows.Graphics.Capture and WASAPI
- [A capture listing a picker can draw](/quest/m3/capture-list.md) - lists apps on X11 with the identity an app capture would name, and tells a caller that the picker stands in for a list on Wayland
150 changes: 150 additions & 0 deletions quest/m3/capture-list.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,150 @@
# [XL] A capture listing a picker can draw

## Goal

`moq_video::capture` lists what can be captured with enough for a picker to
show pictures instead of names, on macOS, Windows, and X11:

- **Apps:** every listed window names its app by an identity that is the same
after the app restarts, and apps are listed with that identity, a name, and
their icon as pixels.
- **Cameras:** each says what kind of device it is (built in, webcam, capture
card, phone, virtual) where the system says, and says nothing where it does
not.
- **Pictures:** a caller gets a current, small picture of any listed window,
display, or camera at a low rate without holding a full-size capture of it,
and the docs state what that costs on each desktop.
- **Changes:** a caller waits for the list to change (a camera or monitor
plugged in or out, a window opened, closed, or retitled) instead of
re-listing on a timer, and an id names the same item before and after.

Each desktop's path for each of these is decided at setup. What a desktop
cannot do is an `Unsupported` error naming the reason, never an empty list, a
blank picture, or a watch that stays silent. On a Wayland session the caller
learns up front that the system's picker owns windows and displays, so every
window and display listing call is `Unsupported`, even with XWayland running.

Not part of this: capturing an app off macOS, which
[Windows](/quest/m3/capture-windows.md) and
[Linux](/quest/m3/capture-linux.md) capture parity decide, microphones, and
the picker itself.

## Plan

Planned 2026-11-09 (maintainer, studio /quest-plan): ✅ m3, since a named
consumer waits on it (the studio's m0). Consumer: the studio's
[Add page](https://github.com/moq-dev/studio/blob/main/quest/m0/ux/add.md),
through
[its pin](https://github.com/moq-dev/studio/blob/main/quest/m0/pin-capture-list.md).
It draws every window as a live thumbnail under its app's icon, grouped by
app, and each display and camera as a live picture with a device icon, and
refreshes when a device is plugged in or a window opens.

Today (`rs/moq-video/src/capture/mod.rs`) the listing is `Camera { id, name }`,
`Display { id, name, width, height }`, `Window { id, title, app, width,
height }`, and `App { id, name }`, each from a one-shot `cameras()`,
`displays()`, `windows()`, or `apps()`, and `apps()` is `Unsupported` off
macOS. `moq devices` and moq-cli's completions are the other callers and move
with the API.

Settled in the interview:

- Identity and icons cover every desktop, not macOS alone.
- A supported path per desktop is decided at setup, with an error otherwise
and never a silent fallback.
- Where the system owns the picker, listing is unsupported and says why.
xdg-desktop-portal's ScreenCast shows its own picker and hands back a
stream: no call lists windows or displays or names one by an id that
outlives the session. [Linux capture parity](/quest/m3/capture-linux.md)
owns the picked source and its restore token.

What today's code gets wrong for this, found while planning:

- **Identity.** Windows names an app by its process's file stem
(`app_name` in `capture/wgc/native.rs`), so every packaged app hosted by
`ApplicationFrameHost` is one app. macOS dedupes `apps()` on the bundle
identifier, which is empty for an app with no bundle, so those collapse
into one. X11 has only the `WM_CLASS` class. The identity must survive a
restart, since the studio keeps lists of apps per user, so a process id is
not one.
- **Ids across a change.** A display's id is its enumeration index on macOS,
X11 (`x11:<index>`), and Windows, so plugging in a monitor renumbers the
rest. An id has to mean one item for as long as it is listed.
- **Listing an app is not capturing it.** `App::source()` builds
`Source::App`, which is `Unsupported` off macOS, and `moq devices` prints
app ids for `--app`. An app listed where it cannot be captured must not
hand out a source that fails at open or an id the CLI refuses.
- **Wayland with XWayland.** `windows()` and `displays()` succeed there with
XWayland's items only, which a picker would take for the whole desktop.

Recommendations, for the PR to confirm against each system rather than take
as given:

- **Shape.** One handle the caller awaits, yielding the whole current list
each time it changes, rather than events to apply to a list fetched
separately. Icons are fetched per app identity at a requested size, apart
from the list, so a retitled window re-reads no icon.
- **Icons.** The system's own icon for the app: the bundle's on macOS, the
executable's or package's on Windows, and the window's `_NET_WM_ICON` on
X11, which is already pixels. Resolving a desktop entry through the icon
theme would add PNG and SVG decoders; take them only if too many apps set
no window icon. An app with no icon has none, never a made-up one.
- **Camera kind.** From what the device reports, never from its name:
AVFoundation's device type, the V4L2 driver or PipeWire node properties,
and the Media Foundation device attributes. No system tells a capture card
from a webcam by class, since both are UVC, so expect that kind only where
a property says so.
- **Pictures.** `capture::open` already takes a size and a rate as hints.
Reuse it if the hints can be made to hold and the frames reach the caller
as small CPU pixels; add a call only where they cannot. A camera's picture
opens the camera, so its light comes on and a device another capture holds
may refuse; listing alone opens no device. Measure what one picture stream
costs per item at its default rate on each desktop, and state it on the
API.
- **Changes.** X11 has `PropertyNotify` on the root's client list and each
window's title, and RandR for monitors, which `capture/x11.rs` already
selects. Cameras have udev and PipeWire's registry on Linux, device
notifications on Windows, and AVFoundation's connect and disconnect
notifications on macOS. Windows has `SetWinEventHook`. macOS reports
another app's windows only through the Accessibility API, which is the
decision under Waiting.

Tests, sized to what each runner has:

- **Linux under Xvfb**, in the lane the X11 capture workflow runs in
(`.github/workflows/x11.yml`, Nightly and the release gate): two windows of
one class with an icon are listed under one app identity with that icon's
pixels; a window painted one color yields a picture of that color at the
bounded size; opening, retitling, and closing a window each wake the watch
with the new list, with no timer in the test.
- **Merge gate, every host:** the identity, id, and list-change logic that
needs no desktop, and the Wayland refusal.
- **Nightly, Linux with `vivid`** (the lane `just rs v4l2` runs in): loading
and unloading the module wakes the watch, the camera's kind is virtual, and
its picture arrives.
- **By hand on a Mac and a Windows desktop**, as ignored tests with each
result in the PR, since no hosted runner has a desktop session or can
grant Screen Recording: grouping and icons for a bundled, an unbundled, and
a packaged app, a picture of each kind of source, and every change in the
Goal.

Public API: breaking for `capture::{Camera, Display, Window, App}` and the
listing calls. Wire: none.

## Waiting

- Maintainer: how macOS notices another app's windows opening, closing, and
being retitled, which only the Accessibility API reports as events. An
agent cannot choose it because one option adds a permission prompt to every
embedder. Options: (1, recommended) moq-video re-lists inside the watch at
a stated rate while a caller holds one, on macOS windows only, so no caller
polls and no permission is added; (2) ask for the Accessibility permission
and observe each app; (3) notify on app launch, quit, and activation only,
missing a window that opens inside a running app. Done when this item names
the pick.

## Related

- [macOS capture listing futures are Send](/quest/m3/capture-shareable-send.md) - the ScreenCaptureKit listing this rewrites; whatever replaces it stays `Send`
- [Windows capture parity](/quest/m3/capture-windows.md) - decides whether a listed app can be captured on Windows
- [Linux capture parity](/quest/m3/capture-linux.md) - the portal-picked source that stands in for a list on Wayland, and app capture on Linux
1 change: 1 addition & 0 deletions quest/m3/capture-windows.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,5 +32,6 @@ window-only decision still stands after that backend lands.
## Related

- [Windows.Graphics.Capture](/quest/m4/capture-wgc.md) - display and window capture with the cursor, which this quest no longer owns
- [A capture listing a picker can draw](/quest/m3/capture-list.md) - lists apps on Windows and gives them the identity an app capture would name
- [Linux capture parity](/quest/m3/capture-linux.md) - the same gaps, through
the portal and PipeWire
2 changes: 2 additions & 0 deletions quest/m4/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,8 @@ deleted.
- [io_uring quiescence on a quiet host](/quest/m4/uring-bench-followups.md) - quiet-host acceptance numbers and a nightly `worker_turns` QUIC request-reply series
- [#2819](/quest/m4/2819-moq-video-carry-pipewire-dma-bufs-safely-into-the-vulkan.md) - moq-video: validate PipeWire DMA-BUFs into the Vulkan renderer on hardware
- [Windows.Graphics.Capture](/quest/m4/capture-wgc.md) - the WGC display and window backend verified on real Windows hardware
- [The foreground app](/quest/m4/capture-foreground.md) - which app and window are in front, with a wait for changes, on macOS, Windows, and X11, for moq-dev/studio's source that follows the game
- [Leaving apps out of a display capture](/quest/m4/capture-exclude.md) - a display capture that never shows chosen apps where the system can filter, for moq-dev/studio's privacy shield
- [The moq CLI can publish a changing catalog](/quest/m4/cli-catalog-updates.md) - catalog updates mid-stream, so recovery replays need no hand-built clients
- [The relay over QUIC on the sim](/quest/m4/quic-sim.md) - the relay and moq-tokio's `Client` and `Server` run over `moq-quic` on the seeded sim
- [moq-shaper's own socket never passes for path loss](/quest/m4/shaper-socket.md) - relay-sized buffers and its own socket drops reported beside path drops
Expand Down
101 changes: 101 additions & 0 deletions quest/m4/capture-exclude.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
# [M] Leaving apps out of a display capture

## Goal

A display capture in `moq_video::capture` can name apps to leave out, by the
identity [the capture listing](/quest/m3/capture-list.md) gives them. No
frame it delivers shows a window of an excluded app, including a window that
opens later and an app that starts after the capture did. The set can change
while the capture runs, and an app added to it is gone from the next frame.

Whether a desktop can do this is decided when the capture is set up. Where
the system cannot leave another app's windows out, asking for an exclusion is
an `Unsupported` error naming the reason, never a capture that shows the app
anyway. A caller can ask whether exclusion is supported without opening a
capture.

Not part of this: hiding the caller's own windows. On Windows,
`SetWindowDisplayAffinity` does that for the caller's own windows. On macOS,
ScreenCaptureKit ignores `NSWindow.sharingType`, so the caller names its own
app in the exclusion like any other; elsewhere its windows are not protected.
Also not part of this: drawing anything where an excluded window was,
since the capture shows whatever the system composites behind it; window and
app captures, which already show only what was chosen; and sound, since
`moq_audio`'s system capture still carries an excluded app's audio.

## Plan

Planned 2026-11-09 (maintainer, studio /quest-plan). Consumer: the studio's
[privacy shield](https://github.com/moq-dev/studio/blob/main/quest/m1/privacy-shield.md),
through
[its pin](https://github.com/moq-dev/studio/blob/main/quest/m1/pin-capture-exclude.md).
It keeps apps the streamer marks private, such as a password manager, out of
every display capture, and turns the setting off with a note on a desktop
that cannot protect them.

Milestone: ✅ m4 (rejected: m3). The maintainer's rule in that interview is
that work a named consumer waits on joins m3, and
[m3](/quest/m3/README.md) sends work only a downstream's later plans need to
m4. The consumer is in the studio's m1, "after the studio ships".

What each desktop can do, found while planning. Re-check each before relying
on it, and record what the PR finds here:

- **macOS: supported.** `open_display` in `capture/screencapture.rs` already
builds its `SCContentFilter` with an empty list of excluded windows, and
ScreenCaptureKit also filters by app and updates a running stream's
filter.
- **Windows: unsupported.** Windows.Graphics.Capture captures a monitor
whole, with no filter. `SetWindowDisplayAffinity` leaves a window out of
capture only when the caller's own process owns it, which is the caller's
half above. The Magnification API has an exclusion list, but the callback
that hands back its pixels is deprecated since Windows 7, runs only with
the Desktop Window Manager off, and has no replacement.
- **X11: unsupported.** A display capture reads the root window, which
already holds every window. Neither the core protocol nor Composite leaves
one out short of moq-video compositing the desktop itself.
- **Wayland: unsupported.** The portal's picker chooses what is shared and
takes no exclusion.

Rejected: painting over an excluded window's rectangle after capture on the
desktops that cannot filter. The rectangle is read apart from the frame, so
a moving or newly opened window shows for a frame or more, which claims a
protection it does not give.

Recommendations for the PR:

- **Apps, not single windows.** The consumer marks apps, and nothing asks to
leave out one window of an app that otherwise shows.
- **A late app.** A filter that excludes running apps cannot name one that
has not started, so a private app could show between its launch and the
filter's update. Either prove that gap cannot reach a frame or build the
filter from the apps to include, so an app nobody has allowed is absent
until it is added. An inclusion filter must admit each app that is not
excluded as it launches, so excluding one app never hides a newly started
game or browser. The Goal holds either way.
- **Shape.** The exclusion belongs to the display selection, not to a
`Config` field every other source ignores, as `cursor` is by cameras. It
is a breaking change to `Source::Display` if it lands there.

Tests:

- **Merge gate, every host:** each backend that cannot filter (X11, WGC, the
portal) refuses a display capture with an exclusion at setup, and the
support query says the same without opening anything.
- **By hand on a Mac**, as ignored tests with each result in the PR, since
no hosted runner can grant Screen Recording: a window of a known color is
on screen and no captured frame holds that color once its app is excluded;
the same for a second window the app opens later, for an app started after
the capture, and for an app added to the set mid-capture.

Public API: breaking if the exclusion joins `Source::Display`, additive
otherwise. Wire: none.

## Required

- [A capture listing a picker can draw](/quest/m3/capture-list.md) - the app identity an exclusion names

## Related

- [The foreground app](/quest/m4/capture-foreground.md) - names apps by the same identity
- [Per-app audio on macOS](/quest/m3/capture-app-audio-macos.md) - the same content filter, limiting audio to one app
Loading