Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 42 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -314,7 +314,7 @@ is named for it, with these sections:
| Shape | Pick its outline, square, rounded, circle, or arch, give it a border inside that outline and the border's color, and give it a soft shadow, which follows the outline and is the layer's first drop shadow filter. |
| Look | Remove its source's background, which every layer showing the source loses (make it independent to cut out here alone); grade its color through a LUT source; add, order, set, and remove its filters, copy them, or paste copied filters after its own; and set its blend mode. |
| Shown in | See how many places show its source, **Linked in 3 scenes**, and **Make independent**, which points this layer at a copy of the source of its own, keeping its look, so editing that look no longer edits the others. A copy says what it copies. |
| Source | See what the source is and whether it and its own track are live. A camera or broadcast without a new picture, or a microphone or system audio without new sound, for a second reads as stalled; a broadcast says how far behind it plays, and a web page whether it is copied on the GPU or, as a fallback, uploaded from memory, saying why. A media file reads as ended once it plays to its end without looping: **Play** plays it from its start, **Stop** stops it at once, blank and silent, until played again, and **Reload** reads the file again; **Reload** restarts a web page, and opens a picked display or window again, as after the picker was cancelled or sharing stopped. Publish it independently (see [Published independently](#published-independently)), and set its own crop and filters, before the layer's, on every layer showing it. **Warn when it goes dark**, on by default for a camera, display, or window, watches it for [going dark](#a-capture-gone-dark). For a role, pick the source filling it. |
| Source | See what the source is and whether it and its own track are live. A camera or broadcast without a new picture, or a microphone or system audio without new sound, for a second reads as stalled; a broadcast says how far behind it plays, and a web page whether it is copied on the GPU or, as a fallback, uploaded from memory, saying why. A media file reads as ended once it plays to its end without looping: **Play** plays it from its start, **Stop** stops it at once, blank and silent, until played again, and **Reload** reads the file again; **Reload** restarts a web page, and opens a picked display or window again, as after the picker was cancelled or sharing stopped. Publish it independently, then whether viewers may switch it off and its priority for them (see [Published independently](#published-independently)), and set its own crop and filters, before the layer's, on every layer showing it. **Warn when it goes dark**, on by default for a camera, display, or window, watches it for [going dark](#a-capture-gone-dark). For a role, pick the source filling it. |

Its **Animate** tab holds the layer's keyframes (see [Keyframes](#keyframes)).
With no layer selected, the inspector shows the edited scene: rename it,
Expand Down Expand Up @@ -1422,8 +1422,15 @@ program and holding nothing. `[audio.<source>]` sets a source's linear `gain`,
`muted` (which the strip's mute and the panic key set), `pan` (`-1.0` hard left to `1.0` hard right, centered by
default), and `delay`, in every audio scene holding it.

`published = ["camera", "mic"]`, a top-level key, names the sources
[published independently](#published-independently).
`[published.<source>]` publishes a source
[published independently](#published-independently), even empty: its
`priority` (0 by default) ranks its tracks against the program's for
viewers, and `toggle = true` lets each viewer switch it off.
Scene files from before named them in a `published = [...]` list, which no
longer loads: write a `[published.<name>]` table for each. A history saved
before then that holds a publish edit fails to load the same way, naming
`show.history.toml` beside the scene file: delete it to start a new history,
so undo no longer reaches the edits before. A history without one loads.

`portrait = true`, a top-level key, sends a portrait picture beside the
landscape one (see [Both formats at once](#both-formats-at-once)). Meanwhile,
Expand Down Expand Up @@ -1640,9 +1647,9 @@ and software otherwise.

A broadcast source, such as a guest's camera, or a source with a track of its
own can be **Published independently**, a switch in the inspector's Source
section, off by default and kept in the scene file's `published` list.
Viewers then fetch it as a broadcast of its own and place it over the program
themselves, as the [composition](#composition) says:
section, off by default and kept in the scene file's `[published.<name>]`
table. Viewers then fetch it as a broadcast of its own and place it over the
program themselves, as the [composition](#composition) says:

- A local source's track goes to the relay as a broadcast of its own at
`<broadcast>/source/<name>`, with a hang catalog on the program's clock,
Expand Down Expand Up @@ -1670,6 +1677,21 @@ carries one rendition, the plain capture before any filter or gain, and
encodes for as long as it is published, holding an encode session as one on
program does.

Once it is published, two more controls say how viewers may treat it, which
the composition carries. They sit under the switch, and in Settings, Advanced
under each published source, which is where a sound alone, with no layer to
inspect, sets them:

- **Viewers may switch it off**, off by default, lets each viewer turn it
off in their player, which then stops fetching it, such as a facecam or a
screen share they can do without. The studio encodes it either way.
- **Priority**, 0 by default, ranks its tracks against the program's when a
viewer's connection cannot carry everything: its picture and sound are
fetched at hang's priority for each, 60 and 80, plus this, so below 0 it
degrades before the program and above 0 after. Within 19 either way, its
sound still goes ahead of the program's picture, and its picture behind
the program's sound.

Every player must reproduce how it is placed, so a layer showing it takes no
filter but opacity, no blend mode, and no shape, nor do its source's filters or
its keyframes, it does not keep out of the way, and no scene that another scene
Expand Down Expand Up @@ -1709,7 +1731,7 @@ a composition:
"start": 10000000,
"program": { "broadcast": ".", "instance": "5c1f0e3a9b7d4e21a8f6c0d3b2e19f47" },
"sources": {
"camera": { "broadcast": "source/camera" },
"camera": { "broadcast": "source/camera", "priority": -10, "toggle": true },
"mic": { "broadcast": "source/mic", "sound": { "gain": 1.0, "delay": "80ms" } },
"alice": {
"broadcast": "../guests/alice/camera.hang",
Expand Down Expand Up @@ -1755,8 +1777,12 @@ a composition:
its offset is 0.
- A source's `sound`, when the audio scene on program holds it, is the
`gain`, `muted`, `pan`, and `delay` it plays at.
- Each source may carry a subscription `priority` (0 if absent) and whether a
viewer may `toggle` it off (`false` if absent).
- A source's `priority` (0 if absent) ranks its tracks against the
program's: a player subscribes to each of them at hang's priority for its
kind (60 for video, 80 for audio) plus it, saturating, so under congestion
one below 0 degrades first. `toggle` (`false` if absent) lets a viewer
switch the source off, which unsubscribes it and drops its pictures and
sound.
- `pictures` lists each layer of the scene on program, then of the
always-on-top ones, showing a published source, back to front over the
program, shown or hidden. Each is a layer as the scene file writes it: its
Expand Down Expand Up @@ -1801,7 +1827,13 @@ goes once `moq play` plays compositions.
untimed catalog, or one it cannot place, such as one listing a broadcast
above the relay URL's path, plays its program alone; the window says why
for the latter.
- `priority` and `toggle` are not applied.
- Each source the composition lets a viewer `toggle` has a switch in the
window's top-right corner, on until switched off. Off, it is not fetched,
and neither its pictures nor its sound play; the switch holds across
compositions, and a source the composition no longer lets a viewer toggle
plays, whatever its switch said.
- `priority` is not applied yet: every source is fetched at hang's
priorities, as the program is.
- Sound needs a build with `capture`; without it the window says so in the
log and plays the pictures alone.

Expand Down
40 changes: 27 additions & 13 deletions quest/m2/compose/controls.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,10 @@
# [M] Viewer controls
# [S] Viewer priorities

## Goal

A viewer watching a studio broadcast can switch off any source the
composition marks toggleable (a facecam, a screen share), and stops fetching
its broadcast. The composition's per-source priorities steer delivery, so
under congestion the least important sources degrade first. The creator sets both per source in
the studio.
The `moq-studio --watch` prototype fetches each source of a composition at the
priority the composition gives it, so under congestion the least important
sources degrade first, as the creator ranked them in the studio.

## Plan

Expand All @@ -22,10 +20,26 @@ moq's native player,
player's toggles would have no home). moq adds the toggles to that quest in
its own tree.

Map layer priority onto the subscription priorities moq-lite already has
(`moq/doc/concept/moq-lite.md`), rather than a second mechanism. A toggled-off
layer is unsubscribed, so it costs the viewer and the relay nothing. The
studio keeps encoding it, so toggling it back on can reach its recent past.
The composition carries `priority` and `toggle` on each source's entry
(`rs/moq-scene/src/composition.rs`), not on each picture, so a source shown by
several pictures is one subscription, on or off as a whole.
[moq-dev/studio#218](https://github.com/moq-dev/studio/pull/218) landed the
rest of this quest: the creator sets each published source's priority and
whether viewers may switch it off (`[published.<name>]` in the scene file, and
the inspector), the composition carries both on each source's entry, and
`--watch` leaves out a source the viewer switched off, unsubscribing it.

Map priority onto the subscription priorities moq-lite already has
(`moq/doc/concept/moq-lite.md`), rather than a second mechanism. A source's
`priority` is relative to the program's tracks: each of its tracks is fetched
at hang's priority for its kind plus it, saturating
(`Child::priority` in `rs/moq-scene/src/composition.rs`). Once the wait
clears, advance the gitlink with `just upgrade-moq` in this PR unless a bump
already has, carry the priority on `Source::Layer`, and pass it to the
layer's video and audio decode consumers (`follow_video` and `follow_audio` in
`rs/moq-scene/src/source.rs`). Prove it with a layer test that reads the
priority off the publisher's aggregate subscription.

## Waiting

- moq-dev/moq: `moq_video::decode::Options` and `moq_audio::decode::Options`
take the subscription priority, which each consumer fixes today at hang's
`PRIORITY.video` and `PRIORITY.audio`. Check: both options carry a priority
on moq's `main`.
2 changes: 1 addition & 1 deletion quest/m2/compose/hang-layers.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,4 +39,4 @@ layer layout track and are replanned in moq's tree.

## Required

- [Viewer controls](/quest/m2/compose/controls.md) - toggles and priorities the spec carries
- [Viewer priorities](/quest/m2/compose/controls.md) - the priorities the spec carries, proven in the prototype
81 changes: 70 additions & 11 deletions rs/moq-scene/src/collection.rs
Original file line number Diff line number Diff line change
Expand Up @@ -58,8 +58,9 @@ pub struct Collection {
/// its own can be, and not one with its background removed, since its
/// track would show the background. Wherever it is placed, it takes only what every
/// player can reproduce, and no scene that another scene draws holds it.
#[serde(default, skip_serializing_if = "BTreeSet::is_empty")]
pub published: BTreeSet<String>,
/// Each says how viewers may treat it.
#[serde(default, skip_serializing_if = "BTreeMap::is_empty")]
pub published: BTreeMap<String, Published>,
/// The scenes, in the order the scene list shows them.
#[serde(default)]
pub scenes: Vec<Scene>,
Expand Down Expand Up @@ -92,7 +93,7 @@ impl Collection {
sources: BTreeMap::new(),
audio: BTreeMap::new(),
video: BTreeMap::new(),
published: BTreeSet::new(),
published: BTreeMap::new(),
scenes: vec![Scene::new("Scene")],
audio_scenes: AudioScene::defaults(),
top: Vec::new(),
Expand Down Expand Up @@ -684,7 +685,7 @@ impl Collection {
self.validate_chain(&video.filters, &on)?;
}

for name in &self.published {
for name in self.published.keys() {
let Some(source) = self.sources.get(name) else {
return Err(Error::Invalid(format!(
"`{name}` is published, but there is no such source"
Expand Down Expand Up @@ -842,7 +843,7 @@ impl Collection {
for (layer, items) in self.layers() {
for item in items {
let source = self.capture(&item.source);
if !self.published.contains(source) {
if !self.published.contains_key(source) {
continue;
}
let on = format!(
Expand Down Expand Up @@ -2655,6 +2656,24 @@ fn is_center(pan: &f32) -> bool {
*pan == 0.0
}

/// How viewers may treat a source published independently.
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct Published {
/// How far ahead of the program's tracks viewers fetch its own under
/// congestion, as a [`Child::priority`](crate::composition::Child::priority)
/// does: 0 ranks with the program, below it degrades first.
#[serde(default, skip_serializing_if = "is_level")]
pub priority: i8,
/// Whether a viewer may switch it off, so their player stops fetching it.
#[serde(default, skip_serializing_if = "is_false")]
pub toggle: bool,
}

fn is_level(priority: &i8) -> bool {
*priority == 0
}

/// An sRGB color with straight alpha, written `#rrggbb` or `#rrggbbaa`.
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, SerializeDisplay, DeserializeFromStr)]
pub struct Rgba(pub [u8; 4]);
Expand Down Expand Up @@ -3267,15 +3286,15 @@ path = "sting.webm"
collection
.sources
.insert("cut".into(), Source::Clone { source: "cam".into() });
collection.published.insert("cam".into());
collection.published.insert("cam".into(), Default::default());
collection.validate().unwrap();

let mut missing = collection.clone();
missing.published.insert("gone".into());
missing.published.insert("gone".into(), Default::default());
assert!(missing.validate().is_err());

let mut clone = collection.clone();
clone.published.insert("cut".into());
clone.published.insert("cut".into(), Default::default());
assert!(clone.validate().is_err(), "a clone has no track of its own");

let mut removed = collection.clone();
Expand All @@ -3289,6 +3308,46 @@ path = "sting.webm"
assert!(removed.validate().is_err(), "its track would show the background");
}

/// Each published source's table says how viewers may treat it, and an
/// empty one ranks it with the program and keeps it on for them.
#[test]
fn reads_how_viewers_may_treat_a_published_source() {
let text = r#"
version = 1

[canvas]
width = 1280
height = 720
framerate = 30

[sources.cam]
kind = "camera"

[sources.mic]
kind = "microphone"

[published.cam]
priority = -10
toggle = true

[published.mic]
"#;
let collection = Collection::from_toml(text).unwrap();
let cam = Published {
priority: -10,
toggle: true,
};
assert_eq!(collection.published["cam"], cam);
assert_eq!(collection.published["mic"], Published::default());
assert_eq!(
Collection::from_toml(&collection.to_toml().unwrap()).unwrap(),
collection
);

let unknown = text.replace("toggle = true", "hidden = true");
assert!(matches!(Collection::from_toml(&unknown), Err(Error::Format(_))));
}

/// A background is removed only from a picture of the source's own that
/// may show a person.
#[test]
Expand Down Expand Up @@ -3495,7 +3554,7 @@ background = "remove"

let mut published = Collection::new();
published.sources.insert("cam".into(), Source::Camera { id: None });
published.published.insert("cam".into());
published.published.insert("cam".into(), Default::default());
let shaped = Item {
shape: collection.scenes[0].items[0].shape,
..Item::new("cam", Rect::fill(&published.canvas))
Expand Down Expand Up @@ -3722,7 +3781,7 @@ items = [
// file does, so it cannot move.
let mut published = Collection::new();
published.sources.insert("cam".into(), Source::Camera { id: None });
published.published.insert("cam".into());
published.published.insert("cam".into(), Default::default());
published.scenes = vec![Scene {
items: vec![Item {
keep_clear: Some(KeepClear::default()),
Expand Down Expand Up @@ -4279,7 +4338,7 @@ rect = { x = 1700, y = 40, width = 160, height = 90 }
refused(
&|c| {
c.sources.insert("Cam".into(), Source::Camera { id: None });
c.published.insert("Cam".into());
c.published.insert("Cam".into(), Default::default());
c.scenes[0].items[1].source = "Cam".into();
},
"cannot be in a slot",
Expand Down
Loading