diff --git a/README.md b/README.md index 158438f8..fac1713b 100644 --- a/README.md +++ b/README.md @@ -314,7 +314,7 @@ is named for it, with these sections: | Shape | Pick its outline, square, rounded, circle, or arch, give it a border inside that outline and the border's color, and give it a soft shadow, which follows the outline and is the layer's first drop shadow filter. | | Look | Remove its source's background, which every layer showing the source loses (make it independent to cut out here alone); grade its color through a LUT source; add, order, set, and remove its filters, copy them, or paste copied filters after its own; and set its blend mode. | | Shown in | See how many places show its source, **Linked in 3 scenes**, and **Make independent**, which points this layer at a copy of the source of its own, keeping its look, so editing that look no longer edits the others. A copy says what it copies. | -| Source | See what the source is and whether it and its own track are live. A camera or broadcast without a new picture, or a microphone or system audio without new sound, for a second reads as stalled; a broadcast says how far behind it plays, and a web page whether it is copied on the GPU or, as a fallback, uploaded from memory, saying why. A media file reads as ended once it plays to its end without looping: **Play** plays it from its start, **Stop** stops it at once, blank and silent, until played again, and **Reload** reads the file again; **Reload** restarts a web page, and opens a picked display or window again, as after the picker was cancelled or sharing stopped. Publish it independently (see [Published independently](#published-independently)), and set its own crop and filters, before the layer's, on every layer showing it. **Warn when it goes dark**, on by default for a camera, display, or window, watches it for [going dark](#a-capture-gone-dark). For a role, pick the source filling it. | +| Source | See what the source is and whether it and its own track are live. A camera or broadcast without a new picture, or a microphone or system audio without new sound, for a second reads as stalled; a broadcast says how far behind it plays, and a web page whether it is copied on the GPU or, as a fallback, uploaded from memory, saying why. A media file reads as ended once it plays to its end without looping: **Play** plays it from its start, **Stop** stops it at once, blank and silent, until played again, and **Reload** reads the file again; **Reload** restarts a web page, and opens a picked display or window again, as after the picker was cancelled or sharing stopped. Publish it independently, then whether viewers may switch it off and its priority for them (see [Published independently](#published-independently)), and set its own crop and filters, before the layer's, on every layer showing it. **Warn when it goes dark**, on by default for a camera, display, or window, watches it for [going dark](#a-capture-gone-dark). For a role, pick the source filling it. | Its **Animate** tab holds the layer's keyframes (see [Keyframes](#keyframes)). With no layer selected, the inspector shows the edited scene: rename it, @@ -1422,8 +1422,15 @@ program and holding nothing. `[audio.]` sets a source's linear `gain`, `muted` (which the strip's mute and the panic key set), `pan` (`-1.0` hard left to `1.0` hard right, centered by default), and `delay`, in every audio scene holding it. -`published = ["camera", "mic"]`, a top-level key, names the sources -[published independently](#published-independently). +`[published.]` publishes a source +[published independently](#published-independently), even empty: its +`priority` (0 by default) ranks its tracks against the program's for +viewers, and `toggle = true` lets each viewer switch it off. +Scene files from before named them in a `published = [...]` list, which no +longer loads: write a `[published.]` table for each. A history saved +before then that holds a publish edit fails to load the same way, naming +`show.history.toml` beside the scene file: delete it to start a new history, +so undo no longer reaches the edits before. A history without one loads. `portrait = true`, a top-level key, sends a portrait picture beside the landscape one (see [Both formats at once](#both-formats-at-once)). Meanwhile, @@ -1640,9 +1647,9 @@ and software otherwise. A broadcast source, such as a guest's camera, or a source with a track of its own can be **Published independently**, a switch in the inspector's Source -section, off by default and kept in the scene file's `published` list. -Viewers then fetch it as a broadcast of its own and place it over the program -themselves, as the [composition](#composition) says: +section, off by default and kept in the scene file's `[published.]` +table. Viewers then fetch it as a broadcast of its own and place it over the +program themselves, as the [composition](#composition) says: - A local source's track goes to the relay as a broadcast of its own at `/source/`, with a hang catalog on the program's clock, @@ -1670,6 +1677,21 @@ carries one rendition, the plain capture before any filter or gain, and encodes for as long as it is published, holding an encode session as one on program does. +Once it is published, two more controls say how viewers may treat it, which +the composition carries. They sit under the switch, and in Settings, Advanced +under each published source, which is where a sound alone, with no layer to +inspect, sets them: + +- **Viewers may switch it off**, off by default, lets each viewer turn it + off in their player, which then stops fetching it, such as a facecam or a + screen share they can do without. The studio encodes it either way. +- **Priority**, 0 by default, ranks its tracks against the program's when a + viewer's connection cannot carry everything: its picture and sound are + fetched at hang's priority for each, 60 and 80, plus this, so below 0 it + degrades before the program and above 0 after. Within 19 either way, its + sound still goes ahead of the program's picture, and its picture behind + the program's sound. + Every player must reproduce how it is placed, so a layer showing it takes no filter but opacity, no blend mode, and no shape, nor do its source's filters or its keyframes, it does not keep out of the way, and no scene that another scene @@ -1709,7 +1731,7 @@ a composition: "start": 10000000, "program": { "broadcast": ".", "instance": "5c1f0e3a9b7d4e21a8f6c0d3b2e19f47" }, "sources": { - "camera": { "broadcast": "source/camera" }, + "camera": { "broadcast": "source/camera", "priority": -10, "toggle": true }, "mic": { "broadcast": "source/mic", "sound": { "gain": 1.0, "delay": "80ms" } }, "alice": { "broadcast": "../guests/alice/camera.hang", @@ -1755,8 +1777,12 @@ a composition: its offset is 0. - A source's `sound`, when the audio scene on program holds it, is the `gain`, `muted`, `pan`, and `delay` it plays at. -- Each source may carry a subscription `priority` (0 if absent) and whether a - viewer may `toggle` it off (`false` if absent). +- A source's `priority` (0 if absent) ranks its tracks against the + program's: a player subscribes to each of them at hang's priority for its + kind (60 for video, 80 for audio) plus it, saturating, so under congestion + one below 0 degrades first. `toggle` (`false` if absent) lets a viewer + switch the source off, which unsubscribes it and drops its pictures and + sound. - `pictures` lists each layer of the scene on program, then of the always-on-top ones, showing a published source, back to front over the program, shown or hidden. Each is a layer as the scene file writes it: its @@ -1801,7 +1827,13 @@ goes once `moq play` plays compositions. untimed catalog, or one it cannot place, such as one listing a broadcast above the relay URL's path, plays its program alone; the window says why for the latter. -- `priority` and `toggle` are not applied. +- Each source the composition lets a viewer `toggle` has a switch in the + window's top-right corner, on until switched off. Off, it is not fetched, + and neither its pictures nor its sound play; the switch holds across + compositions, and a source the composition no longer lets a viewer toggle + plays, whatever its switch said. +- `priority` is not applied yet: every source is fetched at hang's + priorities, as the program is. - Sound needs a build with `capture`; without it the window says so in the log and plays the pictures alone. diff --git a/quest/m2/compose/controls.md b/quest/m2/compose/controls.md index c355b6f4..5fc0499f 100644 --- a/quest/m2/compose/controls.md +++ b/quest/m2/compose/controls.md @@ -1,12 +1,10 @@ -# [M] Viewer controls +# [S] Viewer priorities ## Goal -A viewer watching a studio broadcast can switch off any source the -composition marks toggleable (a facecam, a screen share), and stops fetching -its broadcast. The composition's per-source priorities steer delivery, so -under congestion the least important sources degrade first. The creator sets both per source in -the studio. +The `moq-studio --watch` prototype fetches each source of a composition at the +priority the composition gives it, so under congestion the least important +sources degrade first, as the creator ranked them in the studio. ## Plan @@ -22,10 +20,26 @@ moq's native player, player's toggles would have no home). moq adds the toggles to that quest in its own tree. -Map layer priority onto the subscription priorities moq-lite already has -(`moq/doc/concept/moq-lite.md`), rather than a second mechanism. A toggled-off -layer is unsubscribed, so it costs the viewer and the relay nothing. The -studio keeps encoding it, so toggling it back on can reach its recent past. -The composition carries `priority` and `toggle` on each source's entry -(`rs/moq-scene/src/composition.rs`), not on each picture, so a source shown by -several pictures is one subscription, on or off as a whole. +[moq-dev/studio#218](https://github.com/moq-dev/studio/pull/218) landed the +rest of this quest: the creator sets each published source's priority and +whether viewers may switch it off (`[published.]` in the scene file, and +the inspector), the composition carries both on each source's entry, and +`--watch` leaves out a source the viewer switched off, unsubscribing it. + +Map priority onto the subscription priorities moq-lite already has +(`moq/doc/concept/moq-lite.md`), rather than a second mechanism. A source's +`priority` is relative to the program's tracks: each of its tracks is fetched +at hang's priority for its kind plus it, saturating +(`Child::priority` in `rs/moq-scene/src/composition.rs`). Once the wait +clears, advance the gitlink with `just upgrade-moq` in this PR unless a bump +already has, carry the priority on `Source::Layer`, and pass it to the +layer's video and audio decode consumers (`follow_video` and `follow_audio` in +`rs/moq-scene/src/source.rs`). Prove it with a layer test that reads the +priority off the publisher's aggregate subscription. + +## Waiting + +- moq-dev/moq: `moq_video::decode::Options` and `moq_audio::decode::Options` + take the subscription priority, which each consumer fixes today at hang's + `PRIORITY.video` and `PRIORITY.audio`. Check: both options carry a priority + on moq's `main`. diff --git a/quest/m2/compose/hang-layers.md b/quest/m2/compose/hang-layers.md index bf326256..4aadbf03 100644 --- a/quest/m2/compose/hang-layers.md +++ b/quest/m2/compose/hang-layers.md @@ -39,4 +39,4 @@ layer layout track and are replanned in moq's tree. ## Required -- [Viewer controls](/quest/m2/compose/controls.md) - toggles and priorities the spec carries +- [Viewer priorities](/quest/m2/compose/controls.md) - the priorities the spec carries, proven in the prototype diff --git a/rs/moq-scene/src/collection.rs b/rs/moq-scene/src/collection.rs index e87d0243..99ec0a24 100644 --- a/rs/moq-scene/src/collection.rs +++ b/rs/moq-scene/src/collection.rs @@ -58,8 +58,9 @@ pub struct Collection { /// its own can be, and not one with its background removed, since its /// track would show the background. Wherever it is placed, it takes only what every /// player can reproduce, and no scene that another scene draws holds it. - #[serde(default, skip_serializing_if = "BTreeSet::is_empty")] - pub published: BTreeSet, + /// Each says how viewers may treat it. + #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] + pub published: BTreeMap, /// The scenes, in the order the scene list shows them. #[serde(default)] pub scenes: Vec, @@ -92,7 +93,7 @@ impl Collection { sources: BTreeMap::new(), audio: BTreeMap::new(), video: BTreeMap::new(), - published: BTreeSet::new(), + published: BTreeMap::new(), scenes: vec![Scene::new("Scene")], audio_scenes: AudioScene::defaults(), top: Vec::new(), @@ -684,7 +685,7 @@ impl Collection { self.validate_chain(&video.filters, &on)?; } - for name in &self.published { + for name in self.published.keys() { let Some(source) = self.sources.get(name) else { return Err(Error::Invalid(format!( "`{name}` is published, but there is no such source" @@ -842,7 +843,7 @@ impl Collection { for (layer, items) in self.layers() { for item in items { let source = self.capture(&item.source); - if !self.published.contains(source) { + if !self.published.contains_key(source) { continue; } let on = format!( @@ -2655,6 +2656,24 @@ fn is_center(pan: &f32) -> bool { *pan == 0.0 } +/// How viewers may treat a source published independently. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct Published { + /// How far ahead of the program's tracks viewers fetch its own under + /// congestion, as a [`Child::priority`](crate::composition::Child::priority) + /// does: 0 ranks with the program, below it degrades first. + #[serde(default, skip_serializing_if = "is_level")] + pub priority: i8, + /// Whether a viewer may switch it off, so their player stops fetching it. + #[serde(default, skip_serializing_if = "is_false")] + pub toggle: bool, +} + +fn is_level(priority: &i8) -> bool { + *priority == 0 +} + /// An sRGB color with straight alpha, written `#rrggbb` or `#rrggbbaa`. #[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, SerializeDisplay, DeserializeFromStr)] pub struct Rgba(pub [u8; 4]); @@ -3267,15 +3286,15 @@ path = "sting.webm" collection .sources .insert("cut".into(), Source::Clone { source: "cam".into() }); - collection.published.insert("cam".into()); + collection.published.insert("cam".into(), Default::default()); collection.validate().unwrap(); let mut missing = collection.clone(); - missing.published.insert("gone".into()); + missing.published.insert("gone".into(), Default::default()); assert!(missing.validate().is_err()); let mut clone = collection.clone(); - clone.published.insert("cut".into()); + clone.published.insert("cut".into(), Default::default()); assert!(clone.validate().is_err(), "a clone has no track of its own"); let mut removed = collection.clone(); @@ -3289,6 +3308,46 @@ path = "sting.webm" assert!(removed.validate().is_err(), "its track would show the background"); } + /// Each published source's table says how viewers may treat it, and an + /// empty one ranks it with the program and keeps it on for them. + #[test] + fn reads_how_viewers_may_treat_a_published_source() { + let text = r#" +version = 1 + +[canvas] +width = 1280 +height = 720 +framerate = 30 + +[sources.cam] +kind = "camera" + +[sources.mic] +kind = "microphone" + +[published.cam] +priority = -10 +toggle = true + +[published.mic] +"#; + let collection = Collection::from_toml(text).unwrap(); + let cam = Published { + priority: -10, + toggle: true, + }; + assert_eq!(collection.published["cam"], cam); + assert_eq!(collection.published["mic"], Published::default()); + assert_eq!( + Collection::from_toml(&collection.to_toml().unwrap()).unwrap(), + collection + ); + + let unknown = text.replace("toggle = true", "hidden = true"); + assert!(matches!(Collection::from_toml(&unknown), Err(Error::Format(_)))); + } + /// A background is removed only from a picture of the source's own that /// may show a person. #[test] @@ -3495,7 +3554,7 @@ background = "remove" let mut published = Collection::new(); published.sources.insert("cam".into(), Source::Camera { id: None }); - published.published.insert("cam".into()); + published.published.insert("cam".into(), Default::default()); let shaped = Item { shape: collection.scenes[0].items[0].shape, ..Item::new("cam", Rect::fill(&published.canvas)) @@ -3722,7 +3781,7 @@ items = [ // file does, so it cannot move. let mut published = Collection::new(); published.sources.insert("cam".into(), Source::Camera { id: None }); - published.published.insert("cam".into()); + published.published.insert("cam".into(), Default::default()); published.scenes = vec![Scene { items: vec![Item { keep_clear: Some(KeepClear::default()), @@ -4279,7 +4338,7 @@ rect = { x = 1700, y = 40, width = 160, height = 90 } refused( &|c| { c.sources.insert("Cam".into(), Source::Camera { id: None }); - c.published.insert("Cam".into()); + c.published.insert("Cam".into(), Default::default()); c.scenes[0].items[1].source = "Cam".into(); }, "cannot be in a slot", diff --git a/rs/moq-scene/src/composition.rs b/rs/moq-scene/src/composition.rs index a6267425..c8554099 100644 --- a/rs/moq-scene/src/composition.rs +++ b/rs/moq-scene/src/composition.rs @@ -86,9 +86,9 @@ impl Composition { /// The composition of `collection` with the scene called `scene` on /// program since `start` on the media clock, or no scene for `None`, /// drawn over `program`. Each source published independently is listed - /// as `children` locates it, its sound filled in; one `children` leaves - /// out, such as a guest offline, is left out with its pictures, as the - /// studio shows nothing of it either. + /// as `children` locates it, its sound, priority, and toggle filled in; + /// one `children` leaves out, such as a guest offline, is left out with + /// its pictures, as the studio shows nothing of it either. pub fn new( collection: &Collection, scene: Option<&str>, @@ -116,6 +116,11 @@ impl Composition { } } children.retain(|name, child| child.sound.is_some() || pictures.iter().any(|item| item.source == *name)); + for (name, child) in &mut children { + let viewers = collection.published.get(name).copied().unwrap_or_default(); + child.priority = viewers.priority; + child.toggle = viewers.toggle; + } Self { version: Version, canvas: Size { @@ -141,7 +146,7 @@ pub(crate) fn pictures(collection: &Collection, scene: Option<&str>, top: bool) for place in places { for item in collection.items(&place).unwrap_or_default() { let source = collection.capture(&item.source); - if !collection.published.contains(source) { + if !collection.published.contains_key(source) { continue; } pictures.push(Item { @@ -223,11 +228,13 @@ pub struct Child { /// the program's audio does not play it. #[serde(default, skip_serializing_if = "Option::is_none")] pub sound: Option