diff --git a/README.md b/README.md index 2a6f3fa0..441d0374 100644 --- a/README.md +++ b/README.md @@ -413,12 +413,15 @@ window, so there the desktop picks where it opens. - **Where it goes:** the show's relay and its failover relay, each with a switch and its state, such as *Connected, 18 ms away*. A relay switched off - is left out of this stream. + is left out of this stream. A show going out in both formats says each + relay carries both broadcasts. - **Checks**, filled in as they finish: whether each camera has a picture, whether a microphone's level moves, whether the encoder keeps up with the canvas (timed as [setup](#first-launch) times it), and whether the connection to the first relay switched on carries the program (measured as - setup measures it; published scenes and sources share what is left). A failed check is one sentence and its fix, + setup measures it; published scenes and sources share what is left). A show + going out in both formats needs about twice of each: two frames' encoding in + a frame's time, and two videos' bitrate. A failed check is one sentence and its fix, such as **Send 720p instead**, which changes the canvas, or **Check again**. - **GO LIVE TO** the number of destinations switched on. A failed check never stops you; only having none switched on does. @@ -466,7 +469,7 @@ setting costs. Escape or a click outside closes it. | Pane | Use it to | | --- | --- | -| Show | Pick the format, landscape or portrait, which swaps the canvas's sides; the canvas size as 720p, 1080p, 1440p, or 4K, with the upload each wants, or **Custom** for exact numbers; the frame rate as 30, 60, or **Custom**; and, under **Scene sizes**, each scene's own size (see [Published scenes](#published-scenes)). Saved in the scene file, and undone like any edit. | +| Show | Pick the format, landscape or portrait, which swaps the canvas's sides; **Both at once**, which sends a portrait picture beside a landscape one (see [Both formats at once](#both-formats-at-once)); the canvas size as 720p, 1080p, 1440p, or 4K, with the upload each wants, for both pictures while Both at once is on, or **Custom** for exact numbers; the frame rate as 30, 60, or **Custom**; and, under **Scene sizes**, each scene's own size (see [Published scenes](#published-scenes)). Saved in the scene file, and undone like any edit. | | Stream | Set the relay, its token, the failover relay, the broadcast's name, and on demand (see [Stream settings](#stream-settings)). | | Recording | Change or open the recordings folder, and pick the picture and the audio scene a recording holds, and whether each source records a file of its own too (see [Recording](#recording)). | | Hotkeys | Bind keys, by what they are for: going live, ending the stream, the panic key, and recording; Studio mode, TAKE, and AUTO; each scene; each sound's mute; and replay and clips (see [Hotkeys](#hotkeys)). | @@ -784,7 +787,66 @@ is drawn, and undoing that puts it back keeping clear where it was. Only a scene's layer set free of its [slots](#slots-and-roles) keeps out of the way, and not one showing a source [published independently](#published-independently), which viewers place -themselves; putting one in a slot switches it off. +themselves; putting one in a slot switches it off. While the show goes out in +[both formats at once](#both-formats-at-once), it moves only in the landscape +picture: in the portrait one it stays at the place set there, or fitted from +where its rect rests. + +### Both formats at once + +A landscape show can go out in portrait too, for phones, beside the +landscape picture for desks and TVs: switch on **Both at once** in Settings, +Show. Each scene of the show's size then has two arrangements of the same +layers, and the studio draws, encodes, and sends both pictures. A scene of its +own size, such as a published portrait scene, stays one picture, in its own +shape, the same in both down to the scenes it shows, so it cannot go on +program meanwhile. A scene given the show's size as +its own follows the show's size instead when you turn it on. + +Turning it on is one edit you can undo, and starts each portrait arrangement +from the landscape one. The largest picture, covering a quarter of the canvas +or more, is the main one: it is cropped to the top 60% of the portrait canvas, +with the largest camera stacked under it, cropped to fill the rest, or fitted +there keeping its outline if it has a round or rounded shape. Without a +camera, the main picture covers the whole portrait canvas. A backdrop covering +the landscape canvas covers the portrait one, cropped, but a web page or +overlay covering it, mostly see-through, is fitted whole and is never the main +picture. What sits on the main picture, such as a lower third or a logo, keeps +its place on it. A layer set anywhere else keeps its place on the turned +canvas, scaled to fit. A scene laid out by slots arranges them in a row in +portrait, or in a grid or inset if that is how it arranges them in landscape, +and picks its own portrait arrangement in BUILD's scene inspector. A layer +showing another scene of the show's size, or the program, shows that scene's +portrait arrangement in the portrait picture, tall as the portrait canvas is, +so its place there is shaped for it: one covering the landscape canvas, such +as a scene of overlays, covers the portrait canvas whole rather than being the +main picture, and one set anywhere else keeps its place, as wide and as much +taller as its picture turned. Giving that scene a size of its own in Settings, +or the show's back, changes the shape of its picture there, so each layer +showing it is fitted inside the place it had in the portrait picture, in the +same edit. + +LIVE shows both pictures side by side, each in its tally frame, with its +format, size, and what its video sends under it. BUILD edits one picture at a +time: **Landscape** or **Portrait** under the picture picks which, and the other +shows small beside it, where a click edits it instead. A drag, Fit, Fill, the +inspector's X, Y, W, H, and crop all place the layer in the picture edited, +and leave its place in the other alone, even one fitted there from its +landscape place. A layer in a slot has one crop in both. +Each scene's pad shows the pair, and Studio mode's NEXT card shows both +pictures of the scene cued. + +A layer is shown in both pictures, or only in one: **Format** in its +inspector, and the chip under the pictures naming the selected layer, picks +**Both**, **Landscape**, or **Portrait**. A layer shown only in landscape keeps +no portrait place. Switching **Both at once** off removes every layer shown +only in portrait and forgets every portrait place and arrangement, as one +edit. + +Only a landscape canvas turns: picking Portrait in Settings, Show, switches +Both off. The portrait picture goes out as its own broadcast, at +`/portrait` (see [Output](#output)), so a published scene cannot +be named `portrait` meanwhile. ### Audio scenes @@ -1225,6 +1287,15 @@ items = [ { source = "logo", rect = { x = 1600, y = 40, width = 280, height = 120 } }, ] +[[scenes]] +name = "Game" +portrait = { arrangement = "row" } # its slots in portrait, while the show goes out in both +items = [ + # Its place in the portrait picture, while the show goes out in both. + { source = "game", rect = { x = 0, y = 0, width = 1920, height = 1080 }, portrait = { rect = { x = 0, y = 0, width = 1080, height = 1152 }, crop = { left = 0.2, right = 0.2 } } }, + { source = "logo", rect = { x = 1600, y = 40, width = 280, height = 120 }, format = "landscape" }, # shown only in landscape +] + [[scenes]] name = "Portrait" size = { width = 1080, height = 1920 } # its own size; the program's without one @@ -1292,6 +1363,16 @@ default), and `delay`, in every audio scene holding it. `published = ["camera", "mic"]`, a top-level key, names the sources [published independently](#published-independently). +`portrait = true`, a top-level key, sends a portrait picture beside the +landscape one (see [Both formats at once](#both-formats-at-once)). Meanwhile, +in a scene of the show's size or `[[top]]`, a layer's `format`, `landscape` +or `portrait`, shows it only in that picture, both without one, and its +`portrait`, a `rect` and `crop`, places it in the portrait picture, where it +sits on the landscape canvas, fitted to the turned one, without one. A layer +in a slot, or shown only in landscape, takes no `portrait` place, and a +scene's `portrait = { arrangement = "..." }` arranges its slots there, `row` +without one. + Each entry of a scene's `items` is one of its layers, back to front, and each `[[top]]` an always-on-top layer: the `source` it shows and its `rect` on the canvas, in pixels, or, in a scene, its `slot`, `main` (one per scene) or @@ -1391,7 +1472,12 @@ under the broadcast's path, while on air. A canvas under 360 lines has no half size. Each published scene publishes the same way at `/`, at its own size and half of it, its tracks sharing the connection's bandwidth -with the program's, and its own encode of the audio scene it carries. +with the program's, and its own encode of the audio scene it carries. A show +going out in both formats publishes its portrait picture the same way at +`/portrait`, at the canvas turned and half of it, with its own +encode of the audio scene on program. The sources published independently +are drawn into it, with no layer layout of their own. Each relay carries both broadcasts, +and GO LIVE's checks time the encoder and size the connection for both. Timestamps come from the studio's own clock, so a source that stalls repeats its last picture instead of holding up the stream. @@ -1466,6 +1552,7 @@ encode engine for them all. The Health window counts the sessions open: | The program's canvas size | 1 while on air | | Its half size | 1 while someone watches it | | Each published scene's own size and half size | as the program's | +| The portrait picture's, while the show goes out in both | as the program's | | The recording's own size | 1 while recording | | Each captured source or media file | 1 while used, as [above](#source-tracks) | @@ -1847,8 +1934,8 @@ Below that it shows: watches, since the relay pulls nothing otherwise. Source tracks never reach the relay, so they are left out of the media. -Then, for the program's broadcast and each published scene's, each with its -own viewers: +Then, for the program's broadcast, the portrait picture's, and each published +scene's, each with its own viewers: - **Watching**: the sessions subscribed to it, from the relays' `.stats` broadcasts. A relay shares those only with a connection allowed to diff --git a/quest/m0/destinations.md b/quest/m0/destinations.md index 2176be9f..e4b3af68 100644 --- a/quest/m0/destinations.md +++ b/quest/m0/destinations.md @@ -78,6 +78,14 @@ the go-live sheet and live control the studio has, which land them for MoQ relays. This quest adds its destinations to that sheet rather than building a second way to start a stream. +Proposed 2026-11-17 by +[Landscape and portrait at once](https://github.com/moq-dev/studio/pull/200), +waiting on the maintainer there: a show going out in both formats has two +broadcasts, the program's and `/portrait`. MoQ relays carry both, +and each destination here names the one it takes, landscape by default, so a +phone-first platform takes the portrait picture. That picture is another +encoder-sharing key: two destinations taking it share its encoder. + ## Required - [moq's egress dial changes are pinned](/quest/m0/pin-egress-dial.md) - RTMPS that keeps `tcUrl`, and a dial policy the caller supplies diff --git a/quest/m0/ux/README.md b/quest/m0/ux/README.md index 270c6ebc..4d277941 100644 --- a/quest/m0/ux/README.md +++ b/quest/m0/ux/README.md @@ -63,7 +63,7 @@ shows those pictures beside the frames they implement. from capture, dockable into spare room but never docked by default (rejected: a chat column in the main window; leaving chat to the browser). - **Formats:** ✅ one format per show in m0, landscape or portrait, and the - layout follows it. [Both at once](/quest/m1/dual-format.md) is m1 (rejected: + layout follows it. [Both at once](https://github.com/moq-dev/studio/pull/200) is m1 (rejected: designing every m0 page for two pictures). - **Layout follows the picture:** ✅ the picture is larger than any other part of the window but the title bar, as large as the window allows, and diff --git a/quest/m1/auto-place.md b/quest/m1/auto-place.md index 0b0a94e8..cb15dea9 100644 --- a/quest/m1/auto-place.md +++ b/quest/m1/auto-place.md @@ -1,11 +1,13 @@ -# [S] A camera that keeps out of the way remembers each app +# [M] A camera that keeps out of the way remembers each app, in each format ## Goal A layer with **Keep out of the way** on knows the app in front: it chooses again the moment another app comes to the front, rather than waiting to see the picture change, and the place it chose for an app is remembered per user, -so a known game is right from its first frame. +so a known game is right from its first frame. While the show goes out in +landscape and portrait at once, it keeps out of the way in each picture on +its own. The decision stays on the machine. Nothing is uploaded. @@ -34,6 +36,19 @@ ready; local first with a hosted model as an opt-in tie-breaker). Should the signals prove too weak, a small on-device model is a later quest, not a hosted one. +- **In each format:** while the show goes out in landscape and portrait at + once, a layer keeping out of the way places only its landscape picture; + its portrait place stays as set by hand or derived when **Both at once** + was turned on. Placing each picture on its own remains: the analysis reads + the portrait picture too, chooses among the portrait canvas's slots, and + writes its `portrait` place through moq-scene's `Item::placed`, with a + `Glide` per view (`View::reflows` already is one). A layer shown in one + format only is placed in that one alone. + +Decided 2026-11-18 (maintainer, [moq-dev/studio#200](https://github.com/moq-dev/studio/pull/200#issuecomment-6994130042)): +✅ landscape only in that PR, each format here (rejected: a placer per +format in that PR). + ## Required - [moq-video's foreground app is pinned](/quest/m1/pin-capture-foreground.md) - when the app in front changes, and the key its slot is remembered under diff --git a/quest/m1/dual-format-check.md b/quest/m1/dual-format-check.md new file mode 100644 index 00000000..13f16f69 --- /dev/null +++ b/quest/m1/dual-format-check.md @@ -0,0 +1,26 @@ +# [XS] Watch one show on a phone and a desk + +## Goal + +The maintainer has watched one show going out in both formats on a phone and +on a desktop at once, and each reads as made for its screen: the portrait +picture's main picture is cropped around what matters, the camera stacked +under it is not cut off, and overlays sit where a phone viewer sees them. If +it does not, how turning **Both at once** on places the portrait arrangement +changes until it does. + +## Plan + +[Landscape and portrait at once](https://github.com/moq-dev/studio/pull/200) +draws, encodes, and publishes the portrait picture at `/portrait`, +and its GPU and engine tests check the frames and the broadcast. Whether the +derived arrangement reads well on a real phone, and whether a viewer's player +picks the broadcast for its screen, are judgments no test makes. + +## Waiting + +- Maintainer: switch on **Both at once** in Settings, Show, on a show with a + game or a screen as its main picture and a camera, go live to a relay, + and watch `` on a desktop and `/portrait` on a phone. + Done when both read as intended, or the change the derivation needs is + written into this Plan. diff --git a/quest/m1/dual-format-encodes.md b/quest/m1/dual-format-encodes.md new file mode 100644 index 00000000..7c7c65d1 --- /dev/null +++ b/quest/m1/dual-format-encodes.md @@ -0,0 +1,32 @@ +# [S] Time two program encodes for real + +## Goal + +Setup and the go-live sheet say whether this machine keeps up with a show +going out in landscape and portrait at once from two encodes timed together, +not from one encode's time doubled. The numbers come from a real GPU encoder +with its session cap, such as NVENC on `nixos`'s RTX 3070 Ti, beside the +software encoders. + +## Plan + +Decided 2026-11-17 (maintainer, on +[moq-dev/studio#200](https://github.com/moq-dev/studio/pull/200)): ✅ +estimate two encodes as twice one until this lands (rejected: timing two +encodes in that PR). + +- `setup::Encoding::keeps_up_all` and `total_of` double one encode's frame + time and bitrate. Time the landscape and the turned canvas encoding at once + instead, as `setup::time` times one, and keep the doubled estimate only + where two sessions cannot open. +- A check that opens two hardware sessions holds both while it runs, so it + must finish before going live opens the program's, as the one-encode check + already does. +- Measure on `nixos` (NVENC, session cap 8, with CARLA and a transcode job + holding some) and on a software encoder: frame time and encoder load for + one encode and for two, per AGENTS.md. Record whether doubling over- or + under-estimates, and by how much. + +## Hosts + +- linux - the RTX 3070 Ti on `nixos` diff --git a/quest/m1/dual-format.md b/quest/m1/dual-format.md deleted file mode 100644 index 80065cfe..00000000 --- a/quest/m1/dual-format.md +++ /dev/null @@ -1,39 +0,0 @@ -# [L] Landscape and portrait at once - -## Goal - -A show can go out in both formats together. Each scene has a landscape and a -portrait arrangement of the same layers, the LIVE page shows the two -pictures side by side, each in its tally frame, and a scene pad takes both. -A layer says where it is shown: both, landscape, or portrait. The go-live -sheet sends each destination one format. See the `dual` frame of -[`mock/later.html`](/quest/m0/ux/mock/later.html). - -## Plan - -Decided 2026-11-09 (maintainer, /quest-plan): ✅ one format per show in m0 -and both at once here (rejected: designing every m0 page for two pictures; -never planning it). - -The engine can already do it the long way: a scene has its own size, can be -published as a broadcast of its own, and can draw the program through a -program source, cropped. This quest makes that one switch. - -- Settle the model here: one scene with an arrangement per format, or two - scenes linked by name. The first keeps a rename, a reorder, and a take from - ever going out of step. -- A portrait arrangement starts from the landscape one: the main picture - cropped around its subject, the camera stacked under it. It is then the - streamer's to move. -- A layer in a slot keeps no rect, so a scene laid out by slots needs only a - second `arrangement` for portrait (such as `portrait = { arrangement = - "row" }`), and the slots place themselves on the portrait canvas; only its - layers set free need a second rect. -- Each format is its own encode, so it costs an encoder session and its - bitrate; the checklist says whether the machine and the uplink carry both. -- Where a layer is shown is said under the pictures and in the inspector, - never on the layer, so nothing covers either picture. -- The pages follow: BUILD edits one arrangement at a time with the other - beside it, and Studio mode cues both. - -Human check: a phone and a desktop viewer watching the same show. diff --git a/rs/moq-scene/src/clip.rs b/rs/moq-scene/src/clip.rs index faae0a4d..f0bf54ee 100644 --- a/rs/moq-scene/src/clip.rs +++ b/rs/moq-scene/src/clip.rs @@ -136,6 +136,13 @@ impl Clip { }) .chain(drawn) .collect(), + // Copied from a show going out in both formats, its items + // may say where each picture shows them, which pasting + // keeps or drops as the destination does. + portrait: items + .items + .iter() + .any(|item| item.format.is_some() || item.portrait.is_some()), ..Collection::new() } .validate()? @@ -262,6 +269,11 @@ impl Draft { slot => (slot, item.rect), }; let filters = rename(&item.filters); + // Where it shows in portrait means nothing to a layer drawn in one + // format, and a slot places it there too. + let dual = self.collection().dual(layer); + let format = item.format.filter(|_| dual); + let portrait = item.portrait.filter(|_| dual && slot.is_none()); self.insert_item( layer, index, @@ -270,6 +282,8 @@ impl Draft { filters, slot, rect, + format, + portrait, ..item.clone() }, )?; @@ -364,6 +378,32 @@ mod tests { assert_eq!(pasted.top, collection.scenes[0].items); } + /// A layer copied from a show going out in both formats reads back with + /// its portrait place, which a paste into a show going out in landscape + /// alone drops. + #[test] + fn a_layer_with_a_portrait_place_copies() { + let mut collection = collection(); + collection.portrait = true; + let item = &mut collection.scenes[0].items[0]; + item.portrait = Some(crate::Placement { + rect: Rect::fill(&collection.canvas.turned()), + crop: Default::default(), + }); + item.format = Some(crate::Format::Portrait); + collection.validate().unwrap(); + let clip = Clip::items(&collection, collection.scenes[0].items.clone(), Path::new(DIR)); + let text = clip.to_toml().unwrap(); + let pasted = paste(&collection, &Layer::scene("Scene"), &text, DIR); + assert_eq!(pasted.scenes[0].items[1], collection.scenes[0].items[0]); + let mut landscape = collection.clone(); + landscape.portrait = false; + landscape.scenes[0].items.clear(); + let pasted = paste(&landscape, &Layer::scene("Scene"), &text, DIR); + assert_eq!(pasted.scenes[0].items[0].portrait, None); + assert_eq!(pasted.scenes[0].items[0].format, None); + } + /// A layer the studio placed on its own is the host's once copied. #[test] fn a_copy_is_the_hosts() { diff --git a/rs/moq-scene/src/collection.rs b/rs/moq-scene/src/collection.rs index 064500df..d6ce3394 100644 --- a/rs/moq-scene/src/collection.rs +++ b/rs/moq-scene/src/collection.rs @@ -11,6 +11,7 @@ use serde::{Deserialize, Serialize}; use serde_with::{DeserializeFromStr, SerializeDisplay}; use crate::Error; +use crate::format::{Format, Placement, Portrait}; /// Everything a studio composes: the program's canvas, the sources, and the /// scenes that place them. @@ -24,6 +25,12 @@ pub struct Collection { /// The program's size and frame rate: the size of every scene that sets /// none, and the frame rate every scene is drawn at. pub canvas: Canvas, + /// Whether the show goes out in portrait too, beside the landscape + /// canvas: every scene of the program's size drawn again on the canvas + /// turned on its side, as its [`Portrait`] arrangement and each item's + /// [`portrait`](Item::portrait) place it. Only a landscape canvas turns. + #[serde(default, skip_serializing_if = "is_false")] + pub portrait: bool, /// How the preview is taken to program, short of a cut. #[serde(default)] pub transition: Transition, @@ -77,6 +84,7 @@ impl Collection { pub fn new() -> Self { Self { canvas: Canvas::default(), + portrait: false, transition: Transition::default(), sources: BTreeMap::new(), audio: BTreeMap::new(), @@ -221,9 +229,9 @@ impl Collection { } /// Whether the scene called `scene` can go on program: it is the - /// program's size, and draws no program source, which it would draw into - /// itself. Refused with the reason otherwise, or for a scene that does - /// not exist. + /// program's size, following it while the show goes out in portrait too, + /// and draws no program source, which it would draw into itself. Refused + /// with the reason otherwise, or for a scene that does not exist. pub fn takes(&self, scene: &str) -> Result<(), Error> { match self.refuses(scene) { Some(why) => Err(Error::Invalid(why)), @@ -243,6 +251,13 @@ impl Collection { canvas.width, canvas.height, self.canvas.width, self.canvas.height )); } + // Only a scene that follows the program's size has a portrait picture. + if self.portrait && self.scene(scene).is_some_and(|scene| scene.size.is_some()) { + return Some(format!( + "`{scene}` has a size of its own, so it has no portrait picture and cannot go on program while \ + the show goes out in portrait too" + )); + } if self.draws_program(scene) { return Some(format!("`{scene}` draws the program, so it cannot go on program")); } @@ -286,26 +301,39 @@ impl Collection { } } - /// The visible items `layers` draw, then those of each scene they draw - /// through a scene source, and so on, each scene once. - pub fn drawn(&self, layers: impl IntoIterator) -> Vec<&Item> { - let mut items: Vec<&Item> = layers - .into_iter() - .flat_map(|layer| self.items(&layer).unwrap_or_default()) - .filter(|item| item.visible) - .collect(); + /// The visible items `layers` draw in `format`'s picture, then those of + /// each scene they draw through a scene source, and so on, each scene + /// once, leaving out those shown only in the other picture. In the + /// portrait picture, a scene of its own size and all it draws are drawn as + /// the landscape picture draws them. The portrait picture draws nothing + /// unless the show goes out in it. + pub fn drawn(&self, layers: impl IntoIterator, format: Format) -> Vec<&Item> { + if format == Format::Portrait && !self.portrait { + return Vec::new(); + } + let shown = |format: Format| move |item: &&Item| item.visible && item.shown_in(format); + let mut items: Vec<(&Item, Format)> = Vec::new(); + for layer in layers { + let format = self.format_of(&layer, format); + let found = self.items(&layer).unwrap_or_default(); + items.extend(found.iter().filter(shown(format)).map(|item| (item, format))); + } let mut nested = BTreeSet::new(); let mut next = 0; - while let Some(item) = items.get(next) { - if let Some(Nested::Scene(scene)) = self.nested(&item.source) - && nested.insert(scene) - && let Some(scene) = self.scene(scene) + while let Some(&(item, outer)) = items.get(next) { + if let Some(Nested::Scene(name)) = self.nested(&item.source) + && let Some(scene) = self.scene(name) { - items.extend(scene.items.iter().filter(|item| item.visible)); + // Drawn as the landscape picture draws it, a scene draws all it + // shows so. + let format = self.format_of(&Layer::scene(name), outer); + if nested.insert((name, format)) { + items.extend(scene.items.iter().filter(shown(format)).map(|item| (item, format))); + } } next += 1; } - items + items.into_iter().map(|(item, _)| item).collect() } /// Whether the scene called `scene` places a program source, itself or in @@ -641,6 +669,7 @@ impl Collection { self.validate_nesting()?; self.validate_published()?; + self.validate_formats()?; // Checked once the scenes it draws are, so a panic never finds out. if let Some(safe) = self.safe() && let Some(why) = self.refuses(safe) @@ -1102,6 +1131,11 @@ pub struct Scene { /// Where its items in a [`Slot`] go. #[serde(default, skip_serializing_if = "Arrangement::is_default")] pub arrangement: Arrangement, + /// How it is arranged in the portrait picture, while the show goes out in + /// [portrait](Collection::portrait). Only a scene of the program's size + /// has one. + #[serde(default, skip_serializing_if = "Portrait::is_default")] + pub portrait: Portrait, /// What the scene draws, back to front. #[serde(default)] pub items: Vec, @@ -1119,6 +1153,7 @@ impl Scene { audio: None, safe: false, arrangement: Arrangement::default(), + portrait: Portrait::default(), items: Vec::new(), } } @@ -1192,7 +1227,7 @@ impl Arrangement { /// Every arrangement. pub const ALL: [Self; 4] = [Self::Column, Self::Row, Self::Grid, Self::Inset]; - fn is_default(&self) -> bool { + pub(crate) fn is_default(&self) -> bool { *self == Self::default() } } @@ -1236,6 +1271,15 @@ pub struct Item { /// or failed, and glides the others to their new places. #[serde(default, skip_serializing_if = "Option::is_none")] pub slot: Option, + /// The one format it is shown in, or `None` for both, while the show goes + /// out in [portrait](Collection::portrait). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub format: Option, + /// Where it is drawn in the portrait picture, or `None` where it sits on + /// the landscape one, scaled to fit, as [`Collection::placement`] has it. + /// An item in a slot has none, since the portrait arrangement places it. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub portrait: Option, /// Whether the studio placed it on its own, as a guest appearing in the /// next tile, rather than the host: once that guest leaves, the next to /// appear may take it. Undo restores it with the rest of the item, and a @@ -1285,6 +1329,8 @@ impl Item { source: source.into(), rect, slot: None, + format: None, + portrait: None, auto: false, visible: true, locked: false, @@ -1308,6 +1354,8 @@ impl Item { Cow::Owned(Self { source: self.source.clone(), slot: self.slot, + format: self.format, + portrait: self.portrait, auto: self.auto, rect: Rect { x: self.rect.x + offset.x, @@ -1961,7 +2009,7 @@ impl Crop { } } - fn validate(&self) -> Result<(), String> { + pub(crate) fn validate(&self) -> Result<(), String> { let edges = [self.left, self.top, self.right, self.bottom]; if edges.iter().any(|edge| !(0.0..1.0).contains(edge)) || self.left + self.right >= 1.0 @@ -2064,7 +2112,7 @@ impl Rect { *self == Self::default() } - fn validate(&self, layer: &Layer, source: &str) -> Result<(), Error> { + pub(crate) fn validate(&self, layer: &Layer, source: &str) -> Result<(), Error> { let finite = [self.x, self.y, self.width, self.height].iter().all(|v| v.is_finite()); if !finite || self.width < 0.0 || self.height < 0.0 { return Err(Error::Invalid(format!( @@ -3627,15 +3675,68 @@ items = [ }, ]; collection.validate().unwrap(); - let drawn: Vec<&str> = collection - .drawn([Layer::scene("A")]) - .into_iter() - .map(|item| item.source.as_str()) - .collect(); - assert_eq!(drawn, ["b", "b", "logo", "c", "background"]); + let drawn = |collection: &Collection, format| -> Vec { + let drawn = collection.drawn([Layer::scene("A")], format); + drawn.into_iter().map(|item| item.source.clone()).collect() + }; + assert_eq!( + drawn(&collection, Format::Landscape), + ["b", "b", "logo", "c", "background"] + ); + assert!(drawn(&collection, Format::Portrait).is_empty(), "no portrait picture"); assert!(collection.draws_program("A")); assert!(collection.draws_program("C")); assert!(!collection.draws_program("Scene")); + + // While the show goes out in both, each picture draws the layers + // shown in it, in a scene drawn whole or through a scene source, + // which shows its portrait picture in the portrait one. + let turned = Rect::fill(&collection.canvas.turned()); + let portrait_only = |source| Item { + format: Some(Format::Portrait), + portrait: Some(Placement { + rect: turned, + crop: Crop::default(), + }), + ..Item::new(source, fill) + }; + collection.portrait = true; + collection.scenes[1].items.push(portrait_only("camera")); + collection.scenes[3].items.push(portrait_only("guest")); + collection.validate().unwrap(); + assert_eq!( + drawn(&collection, Format::Landscape), + ["b", "b", "logo", "c", "background"] + ); + assert_eq!( + drawn(&collection, Format::Portrait), + ["b", "b", "camera", "logo", "c", "background", "guest"] + ); + + // Shown only in landscape, B draws what it shows only in portrait in + // neither picture. + let mut landscape_only = collection.clone(); + for item in &mut landscape_only.scenes[1].items[..2] { + item.format = Some(Format::Landscape); + } + landscape_only.validate().unwrap(); + assert_eq!( + drawn(&landscape_only, Format::Landscape), + ["b", "b", "logo", "c", "background"] + ); + assert_eq!(drawn(&landscape_only, Format::Portrait), ["camera"]); + + // Of its own size, B, and C through it, are drawn in portrait as the + // landscape picture draws them. + collection.scenes[2].size = Some(Size { + width: 1280, + height: 720, + }); + collection.validate().unwrap(); + assert_eq!( + drawn(&collection, Format::Portrait), + ["b", "b", "camera", "logo", "c", "background"] + ); } /// A scene's own size is one the encoder can take, a published scene is diff --git a/rs/moq-scene/src/compositor.rs b/rs/moq-scene/src/compositor.rs index b57d515c..b9951da6 100644 --- a/rs/moq-scene/src/compositor.rs +++ b/rs/moq-scene/src/compositor.rs @@ -17,6 +17,7 @@ use crate::collection::{ }; use crate::dark::{Probe, Prober}; use crate::effect::{self, Effects, Frame, Lut, Picture, Prepared}; +use crate::format::Format; use crate::glide::{Glide, REFLOW, mixed, pair, stack}; use crate::matte::Matte; use crate::overlay::{Overlay, Overlays}; @@ -75,9 +76,12 @@ pub struct Compositor { queue: wgpu::Queue, effects: Effects, - program: Program, - /// The scene in preview, at its own size, as of the last render. - preview: Target, + /// The landscape picture, or the only one. + view: View, + /// The portrait picture, while the show goes out in portrait: the same + /// scenes on program and in preview, through the same transition, each + /// arranged as its portrait arrangement says. + portrait: Option, /// Whether the next render draws the preview, which it skips while no /// one looks. previewing: bool, @@ -87,10 +91,6 @@ pub struct Compositor { outputs: BTreeMap, layers: HashMap, - /// The textures scenes drawn through scene sources are drawn into, by - /// scene: as many as the last render drew it at different moments, - /// reused by the next. - nested: HashMap>, /// Lays out and paints overlays, started with the first one. overlays: Option, /// Reduces captures' pictures to see if they went dark, built with the @@ -115,10 +115,9 @@ pub struct Compositor { preview_shown: Duration, /// How long the program scene had been on program as of the last render. program_shown: Duration, - /// Whether the last render drew layers over the program. - layered: bool, switch: Option, - /// Each scene item keeping clear, by its scene and place in it. + /// Each scene item keeping clear, by its scene and place in it. It + /// places only the landscape picture: its portrait place stays as set. keepers: HashMap<(String, usize), Keeper>, /// Each scene item that stopped keeping clear, by its scene and place in /// it, for as long as it is there and keeps clear of nothing. @@ -128,11 +127,69 @@ pub struct Compositor { grabbed: Option<((String, usize, String), Instant)>, /// Reduces the pictures under them, made with the first. reducer: Option, +} + +/// One picture the show goes out in, as of the last render: its program, its +/// preview, and the scenes they draw. +struct View { + program: Program, + /// The scene in preview, at its own size. + preview: Target, + /// The textures scenes drawn through scene sources are drawn into, by + /// scene: as many as the last render drew it at different moments, + /// reused by the next. + nested: HashMap>, + /// Whether the last render drew layers over the program. + layered: bool, /// Each scene with items in slots, by name: where its arrangement put /// them as of the last render, and their glide there. reflows: HashMap, } +impl View { + fn new(device: &wgpu::Device, effects: &Effects, canvas: Canvas) -> Self { + Self { + program: Program::new(device, effects, canvas), + preview: Target::new(device, effects, canvas, "moq-scene preview"), + nested: HashMap::new(), + layered: false, + reflows: HashMap::new(), + } + } + + /// Draw at `canvas` from the next render on: a program of a new size + /// gets new textures of that size, blank until drawn. + fn fit(&mut self, device: &wgpu::Device, effects: &Effects, canvas: Canvas) { + if self.program.canvas.size() == canvas.size() { + self.program.canvas = canvas; + } else { + self.program = Program::new(device, effects, canvas); + } + } + + /// The program as viewers see it: with the layers of each source + /// published independently over it, while it has any showing. + fn layered(&self) -> &Target { + match &self.program.layered { + Some(layered) if self.layered => layered, + _ => &self.program.program, + } + } + + /// Keep where each scene's slots were through `renames`, so two scenes + /// swapping names do not glide from each other's. + fn rename(&mut self, renames: &BTreeMap>) { + let reflows = std::mem::take(&mut self.reflows); + for (scene, reflow) in reflows { + match renames.get(&scene) { + Some(Some(to)) => self.reflows.insert(to.clone(), reflow), + Some(None) => None, + None => self.reflows.insert(scene, reflow), + }; + } + } +} + /// A scene item keeping clear: where it goes, and its reads of the picture /// under it. struct Keeper { @@ -271,6 +328,9 @@ pub enum Thumbnail { /// The source called this: its newest picture through its own crop and /// filters, or what a scene or program source shows. Source(String), + /// The scene called this as the portrait picture shows it, while the + /// show goes out in portrait and the scene is the program's size. + Portrait(String), } /// A thumbnail named by [`Compositor::set_thumbnails`]. @@ -479,8 +539,8 @@ impl Compositor { let effects = Effects::new(device, queue, canvas.width, canvas.height); Ok(Self { - program: Program::new(device, &effects, canvas), - preview: Target::new(device, &effects, canvas, "moq-scene preview"), + view: View::new(device, &effects, canvas), + portrait: None, previewing: true, previewed: false, outputs: BTreeMap::new(), @@ -488,7 +548,6 @@ impl Compositor { queue: queue.clone(), effects, layers: HashMap::new(), - nested: HashMap::new(), overlays: None, prober: None, thumbnails: BTreeMap::new(), @@ -500,19 +559,17 @@ impl Compositor { program_start: None, preview_shown: Duration::ZERO, program_shown: Duration::ZERO, - layered: false, switch: None, keepers: HashMap::new(), parked: HashMap::new(), grabbed: None, reducer: None, - reflows: HashMap::new(), }) } /// The program's canvas. pub fn canvas(&self) -> Canvas { - self.program.canvas + self.view.program.canvas } /// Draw `collection`'s program and scenes at their sizes from the next @@ -524,7 +581,8 @@ impl Compositor { pub fn fit(&mut self, collection: &Collection) -> Result<(), Error> { let program = collection.canvas; let scenes = collection.scenes.iter().map(|scene| scene.canvas(program)); - let canvases: Vec = std::iter::once(program).chain(scenes).collect(); + let portrait = collection.portrait_canvas(); + let canvases: Vec = std::iter::once(program).chain(portrait).chain(scenes).collect(); for canvas in &canvases { fits(&self.device, *canvas)?; } @@ -532,11 +590,15 @@ impl Compositor { let width = canvases.iter().map(|canvas| canvas.width).max().unwrap_or_default(); let height = canvases.iter().map(|canvas| canvas.height).max().unwrap_or_default(); self.effects.fit(width, height); - if self.program.canvas.size() == program.size() { - self.program.canvas = program; - } else { - self.program = Program::new(&self.device, &self.effects, program); - } + self.view.fit(&self.device, &self.effects, program); + self.portrait = match (portrait, self.portrait.take()) { + (None, _) => None, + (Some(canvas), Some(mut view)) => { + view.fit(&self.device, &self.effects, canvas); + Some(view) + } + (Some(canvas), None) => Some(View::new(&self.device, &self.effects, canvas)), + }; Ok(()) } @@ -545,12 +607,20 @@ impl Compositor { /// size, with the pictures of their [layer layout](crate::layout) over /// both. pub fn preview(&self) -> &wgpu::Texture { - &self.preview.texture + &self.view.preview.texture } /// The canvas the preview was last drawn at. pub fn preview_canvas(&self) -> Canvas { - self.preview.canvas + self.view.preview.canvas + } + + /// The scene in preview as the portrait picture shows it, as + /// [`preview`](Self::preview) is, with the canvas it was last drawn at, + /// while the show goes out in portrait. + pub fn portrait_preview(&self) -> Option<(&wgpu::Texture, Canvas)> { + let view = self.portrait.as_ref()?; + Some((&view.preview.texture, view.preview.canvas)) } /// Whether the last [`render`](Self::render) drew the preview. @@ -569,7 +639,7 @@ impl Compositor { /// `moq_video::render::export::Exporter::copy` takes to hand it to an /// encoder. pub fn program(&self) -> &wgpu::Texture { - &self.program.program.texture + &self.view.program.program.texture } /// The program as viewers see it, as of the last [`render`](Self::render): @@ -577,10 +647,19 @@ impl Compositor { /// [`program`](Self::program) itself while it has none showing. Created /// as the program is. pub fn layered(&self) -> &wgpu::Texture { - match &self.program.layered { - Some(layered) if self.layered => &layered.texture, - _ => self.program(), - } + &self.view.layered().texture + } + + /// The portrait picture as viewers see it, as of the last + /// [`render`](Self::render), with its canvas, while the show goes out in + /// portrait: the scene on program, or the transition under way, as its + /// portrait arrangement places it. It carries no + /// [layer layout](crate::layout), so the pictures of sources published + /// independently are drawn into it, as in [`layered`](Self::layered). + /// Created as the program is. + pub fn portrait(&self) -> Option<(&wgpu::Texture, Canvas)> { + let view = self.portrait.as_ref()?; + Some((&view.layered().texture, view.program.canvas)) } /// How long the program scene had been on program as of the last @@ -787,19 +866,32 @@ impl Compositor { } /// The items of the scene called `scene` where its arrangement put them - /// as of the last [`render`](Self::render), in order, as - /// [`Collection::arranged`] gives them: `None` for a scene with no item - /// in a slot. What is drawn glides there when a slot opens or closes. - pub fn arranged(&self, scene: &str) -> Option<&[Item]> { - self.reflows.get(scene).map(|reflow| reflow.arranged.as_slice()) + /// in `format`'s picture as of the last [`render`](Self::render), in + /// order, as [`Collection::arranged`] gives them for + /// [`Collection::formatted`]: `None` for a scene with no item in a slot, + /// or a picture not drawn. What is drawn glides there when a slot opens or + /// closes. + pub fn arranged(&self, format: Format, scene: &str) -> Option<&[Item]> { + let reflow = self.view_of(format)?.reflows.get(scene)?; + Some(reflow.arranged.as_slice()) } /// The items of the scene called `scene` where the last - /// [`render`](Self::render) drew them, in order: those in slots part way - /// along their glide while a reflow is under way, as - /// [`arranged`](Self::arranged) otherwise. - pub fn placed(&self, scene: &str) -> Option<&[Item]> { - self.reflows.get(scene).map(|reflow| reflow.glide.placed()) + /// [`render`](Self::render) drew them in `format`'s picture, in order: + /// those in slots part way along their glide while a reflow is under + /// way, as [`arranged`](Self::arranged) otherwise. + pub fn placed(&self, format: Format, scene: &str) -> Option<&[Item]> { + let reflow = self.view_of(format)?.reflows.get(scene)?; + Some(reflow.glide.placed()) + } + + /// The picture of `format`: the portrait one, or the landscape one, or + /// the only one. + fn view_of(&self, format: Format) -> Option<&View> { + match format { + Format::Landscape => Some(&self.view), + Format::Portrait => self.portrait.as_ref(), + } } /// Whether a transition is under way. @@ -864,9 +956,12 @@ impl Compositor { None => self.outputs.insert(scene, output), }; } - // Each scene keeps where its slots were, and where its items keeping - // clear went, so two swapping names do not take each other's. - self.reflows = renamed(std::mem::take(&mut self.reflows), renames, |scene| scene); + self.view.rename(renames); + if let Some(portrait) = &mut self.portrait { + portrait.rename(renames); + } + // Each scene keeps where its items keeping clear went, so two + // swapping names do not take each other's. self.keepers = renamed(std::mem::take(&mut self.keepers), renames, |(scene, _)| scene); self.parked = renamed(std::mem::take(&mut self.parked), renames, |(scene, _)| scene); if let Some(((scene, _, _), _)) = &mut self.grabbed { @@ -1003,7 +1098,8 @@ impl Compositor { /// Draw the program, the preview unless no one looks at it, and each /// output being read as of `now`, from the newest picture each of - /// `sources` has. + /// `sources` has, and the portrait picture's program and preview too while + /// the show goes out in portrait. /// /// A source whose picture fails to draw keeps its last one and reports why /// through [`error`](Self::error); this fails only when a canvas itself @@ -1012,7 +1108,7 @@ impl Compositor { // A scene grown since the last `fit` would overrun what it sized. let (width, height) = self.effects.fitted(); let grown = collection.scenes.iter().find(|scene| { - let canvas = scene.canvas(self.program.canvas); + let canvas = scene.canvas(self.view.program.canvas); canvas.width > width || canvas.height > height }); if let Some(scene) = grown { @@ -1021,6 +1117,12 @@ impl Compositor { scene.name ))); } + let turned = collection.portrait_canvas(); + if turned.map(|canvas| canvas.size()) != self.portrait.as_ref().map(|view| view.program.canvas.size()) { + return Err(Error::Invalid( + "the portrait picture changed since the compositor was fit".into(), + )); + } // Before the pictures are pulled, so a stinger's clip starting now is // cleared in this frame rather than showing where its last play ended. @@ -1030,7 +1132,7 @@ impl Compositor { for output in self.outputs.values_mut() { output.start.get_or_insert(now); } - let framerate = self.program.canvas.framerate; + let framerate = self.view.program.canvas.framerate; for thumb in self.thumbnails.values_mut() { let period = u64::from((framerate / thumb.detail.rate.max(1)).max(1)); thumb.due = !thumb.drawn || (self.renders + thumb.phase).is_multiple_of(period); @@ -1039,15 +1141,23 @@ impl Compositor { } self.renders += 1; self.update(collection, sources, now); - // Every draw below reads the items keeping clear where they are - // placed, and the scenes as arranged around them. + // Every draw below reads the items keeping clear where they are placed + // in the landscape picture, and the scenes as arranged around them, + // each in its own picture. Keeping clear places only the landscape + // picture, so the portrait one draws each item where it was set. let watched = self.watched(collection, progress, now); let placed = self.place(collection, &watched, now); - let staged = self.reflow(&placed, sources, now); + let landscape = placed.formatted(Format::Landscape); + let staged = reflow(&mut self.view.reflows, &self.layers, landscape, sources, now); + let turned = self.portrait.as_mut().map(|view| { + let formatted = collection.formatted(Format::Portrait); + reflow(&mut view.reflows, &self.layers, formatted, sources, now) + }); let collection = &*staged; - self.restart(collection, preview_start, program_start); - self.paint(collection, now); - let thumbnails = self.thumbnail_targets(collection, now); + let turned = turned.as_deref(); + self.restart(collection, turned, preview_start, program_start); + self.paint(collection, turned, now); + let thumbnails = self.thumbnail_targets(collection, turned, now); let preview_since = now.saturating_duration_since(preview_start); self.preview_shown = preview_since; @@ -1061,30 +1171,14 @@ impl Compositor { (switch, from, progress) }); - // Each texture at its scene's size: a scene that does not exist, or no - // scene, at the program's. - let program_canvas = self.program.canvas; - let canvas_of = |scene: Option<&str>| collection.canvas(scene).unwrap_or(program_canvas); - let previewing = self.previewing; - let preview_canvas = canvas_of(preview.0); - // The top layer goes over a preview of what could go on program. - let preview_top = preview_canvas.size() == program_canvas.size(); - if previewing { - resize( - &mut self.preview, - &self.device, - &self.effects, - preview_canvas, - "moq-scene preview", - ); - } + let program_canvas = self.view.program.canvas; let mut outputs = Vec::new(); for (scene, output) in &mut self.outputs { output.drawn = output.draw; if !output.draw { continue; } - let canvas = canvas_of(Some(scene)); + let canvas = collection.canvas(Some(scene)).unwrap_or(program_canvas); match &mut output.target { Some(target) => resize(target, &self.device, &self.effects, canvas, "moq-scene output"), None => output.target = Some(Target::new(&self.device, &self.effects, canvas, "moq-scene output")), @@ -1096,166 +1190,38 @@ impl Compositor { )); } - // Each texture's scenes and top layer, and the scenes they draw. - let mut roots = Vec::new(); - if previewing { - roots.extend(preview.0.map(|scene| (Place::scene(scene), preview.1))); - if preview_top { - roots.push((Place::top(), Duration::ZERO)); - } - } - for On(scene, since) in on_program(program, switch) { - roots.extend(scene.map(|scene| (Place::scene(scene), since))); - } - roots.push((Place::top(), Duration::ZERO)); - for (On(scene, since), _) in &outputs { - roots.extend(scene.map(|scene| (Place::scene(scene), *since))); - } - for (shows, since, _) in &thumbnails { - if let Shows::Scene(scene) = shows { - roots.push((Place::scene(*scene), *since)); - } - } - // A source's thumbnail draws it whole, as an item of its own. - let wholes: Vec<(Item, Duration)> = thumbnails - .iter() - .filter_map(|(shows, since, target)| match shows { - Shows::Source(source) => Some((Item::new(*source, Rect::fill(&target.canvas)), *since)), - _ => None, - }) - .collect(); - let roots = roots - .iter() - .map(|(place, since)| (collection.items(place).unwrap_or_default(), *since)); - let wholes = wholes.iter().map(|(item, since)| (std::slice::from_ref(item), *since)); - let nesting = Nesting::of(collection, roots.chain(wholes)); - let targets = nested_targets(&mut self.nested, &self.device, &self.effects, collection, &nesting); - - // The pictures viewers draw over the program, which the preview draws - // over its scene too, so both show what viewers would see. - let program_layers = crate::layout::pictures(collection, program.0, true); - let preview_layers = crate::layout::pictures(collection, preview.0, preview_top); - self.layered = program_layers.iter().any(|item| item.visible); - if self.layered && self.program.layered.is_none() { - let target = Target::new(&self.device, &self.effects, program_canvas, "moq-scene layered"); - self.program.layered = Some(target); - } - + let (landscape, portrait): (Vec<_>, Vec<_>) = thumbnails + .into_iter() + .partition(|(format, ..)| *format == Format::Landscape); let mut frame = self.effects.frame(); - let mut draw = Draw { - collection, + let mut pass = Pass { + device: &self.device, effects: &mut self.effects, - library: Library(&self.layers), + layers: &self.layers, frame: &mut frame, - size: program_canvas, - prepared: HashMap::new(), - nesting: &nesting, - scenes: HashMap::new(), - program: None, + previewing: self.previewing, + preview, + program, + switch, }; - // What the program shows is drawn first, then the program, then what - // shows it, then everything else. - let nests = nesting.order.iter().zip(&targets).enumerate(); - let (late, early): (Vec<_>, Vec<_>) = nests.partition(|(_, (nest, _))| nest.program); - for (index, (nest, target)) in early { - draw.nest(index, nest, target); - } - draw.draw_program(&self.program, program, switch); - draw.program = Some(self.program.program.picture.clone()); - for (index, (nest, target)) in late { - draw.nest(index, nest, target); - } - if let Some(layered) = self.program.layered.as_ref().filter(|_| self.layered) { - draw.size = program_canvas; - draw.target(layered); - let canvas = Rect::fill(&program_canvas); - draw.picture(layered, &self.program.program, canvas, canvas, 1.0); - draw.layers(layered, &program_layers, program_since); - } - if previewing { - draw.draw_preview(&self.preview, preview, preview_top); - draw.layers(&self.preview, &preview_layers, preview_since); - } - for (on, target) in &outputs { - draw.size = target.canvas; - draw.target(target); - draw.scene(target, *on); - } - for (shows, since, target) in &thumbnails { - draw.size = target.canvas; - let fill = Rect::fill(&target.canvas); - match *shows { - Shows::Scene(scene) => { - draw.target(target); - draw.scene(target, On(Some(scene), *since)); - } - Shows::Program => { - // What viewers see, as the Program panel shows it: with the - // published layers over it. - let program = self.program.layered.as_ref().filter(|_| self.layered); - draw.target(target); - draw.picture(target, program.unwrap_or(&self.program.program), fill, fill, 1.0); - } - Shows::Preview(_) => { - draw.target(target); - draw.picture(target, &self.preview, fill, fill, 1.0); - } - Shows::Output(scene) => { - let output = outputs.iter().find(|(on, _)| on.0 == Some(scene)); - let (_, output) = output.expect("an output drawn this render"); - draw.target(target); - draw.picture(target, output, fill, fill, 1.0); - } - Shows::Source(source) => { - draw.frame - .target(&target.picture, target.canvas, wgpu::Color::TRANSPARENT); - draw.source(target, source, *since); - } - Shows::Nothing => unreachable!("a thumbnail showing nothing is not drawn"), - } - } - - // The picture under each item keeping clear, read a few times a - // second while its scene is drawn: on program, in preview, for an - // output, or through a scene source. - let mut reads = Vec::new(); - for ((scene, index), keeper) in &mut self.keepers { - keeper.watched = false; - let Some(&Watched { since, nested }) = watched.get(scene) else { - continue; - }; - let Some(shown) = collection.scene(scene) else { continue }; - // While its slots glide, a scene draws only its visible items, in - // the order they stack, so one keeping clear is found by what it - // shows and where it was placed, nearest its place in the file. - let rect = keeper.placer.placement().rect; - let mine = |item: &Item| item.keep_clear.is_some() && item.source == keeper.source && item.rect == rect; - let at = shown - .items - .iter() - .enumerate() - .filter(|(_, item)| mine(item)) - .min_by_key(|(at, _)| at.abs_diff(*index)) - .map(|(at, _)| at); - let Some(at) = at.filter(|&at| shown.items[at].visible) else { - continue; - }; - keeper.watched = true; - if !keeper.look.due(now) { - continue; - } - let canvas = canvas_of(Some(scene)); - let target = keeper.look.target(draw.effects, canvas); - draw.size = canvas; - // What a scene source draws starts transparent, as its nest does. - match nested { - true => draw.frame.target(&target.picture, canvas, wgpu::Color::TRANSPARENT), - false => draw.target(&target), - } - draw.under(&target, Place::scene(scene.as_str()), at, since); - reads.push((scene.clone(), *index)); + let keeping = Keeping { + keepers: &mut self.keepers, + watched: &watched, + now, + }; + let reads = pass.draw( + &mut self.view, + collection, + None, + &outputs, + &strip(landscape), + Some(keeping), + ); + if let (Some(view), Some(turned)) = (&mut self.portrait, turned) { + let program = self.view.program.program.picture.clone(); + pass.draw(view, turned, Some((collection, program)), &[], &strip(portrait), None); } - self.previewed = previewing; + self.previewed = self.previewing; self.effects.submit(frame); if !reads.is_empty() { @@ -1305,7 +1271,8 @@ impl Compositor { .iter() .filter_map(|On(scene, since)| Some((collection.items(&Place::scene((*scene)?))?, *since))); let top = (collection.items(&Place::top()).unwrap_or_default(), Duration::ZERO); - for nest in &Nesting::of(collection, items.chain([top])).order { + let roots = items.chain([top]).map(|(items, since)| (items, since, false)); + for nest in &Nesting::of(Pictures::single(collection), roots).order { let shown = Watched { since: nest.since, nested: true, @@ -1401,53 +1368,6 @@ impl Compositor { staged } - /// `collection` with each scene's items in slots where its arrangement - /// puts them at `now`, gliding there as a move transition does when a - /// slot opens or closes. A slot closes while its source is offline or - /// failed in `sources`, and the picture in it is cropped to cover it by - /// the size of its newest picture. - fn reflow<'a>(&mut self, collection: &'a Collection, sources: &Sources, now: Instant) -> Cow<'a, Collection> { - let present = |name: &str| { - let feed = sources.get(collection.capture(name)); - !feed.is_some_and(|feed| matches!(feed.status(), Status::Offline | Status::Failed(_))) - }; - let layers = &self.layers; - let size = |name: &str| match collection.nested(name) { - Some(Nested::Scene(scene)) => collection - .canvas(Some(scene)) - .map(|canvas| (canvas.width, canvas.height)), - Some(Nested::Program) => Some((collection.canvas.width, collection.canvas.height)), - None => { - let layer = layers.get(collection.capture(name))?; - layer.picture.as_ref().map(Picture::size) - } - }; - let mut staged = Cow::Borrowed(collection); - let mut reflows = HashMap::new(); - for (index, scene) in collection.scenes.iter().enumerate() { - let Some(arranged) = collection.arranged(&scene.name, present, size) else { - continue; - }; - let mut reflow = self.reflows.remove(&scene.name).unwrap_or_else(|| Reflow { - arranged: Vec::new(), - // Only the slots glide, so a layer set free that is being - // dragged follows the pointer. - glide: Glide::new(REFLOW, |item| item.slot.is_some()), - }); - // Only the slots start a glide: what each shows, whether it is - // open, and where. - let slotted = arranged.iter().filter(|item| item.slot.is_some()); - let key = slotted - .map(|item| (item.source.clone(), item.visible, item.rect)) - .collect(); - staged.to_mut().scenes[index].items = reflow.glide.show(key, &arranged, now).to_vec(); - reflow.arranged = arranged; - reflows.insert(scene.name.clone(), reflow); - } - self.reflows = reflows; - staged - } - /// Pull the newest picture from every source onto the GPU, and forget the /// sources the collection no longer has or has since edited. An overlay /// that arrives now starts its CSS timeline at `now`. @@ -1619,8 +1539,15 @@ impl Compositor { /// do. Program, outputs, and preview share one timeline, so a preview /// never restarts an overlay program or an output shows, through the scene /// a transition leaves, the top layer, or a stinger's masks included, nor - /// does an output going on restart one program shows. - fn restart(&mut self, collection: &Collection, preview_start: Instant, program_start: Instant) { + /// does an output going on restart one program shows. Program and preview + /// are both pictures', `collection` and `turned`, the portrait one. + fn restart( + &mut self, + collection: &Collection, + turned: Option<&Collection>, + preview_start: Instant, + program_start: Instant, + ) { let restarts = |layer: &Layer| matches!(layer.source, Source::Overlay { restart: true, .. }); if !self .layers @@ -1629,15 +1556,21 @@ impl Compositor { { return; } - let program = self.drawn(collection, self.program_scene.as_deref().map(Some)); - let on_air = self.on_air_sources(collection); - let preview = self.drawn(collection, self.preview_scene.as_deref().map(Some)); + let (mut program, mut on_air, mut preview) = (HashSet::new(), HashSet::new(), HashSet::new()); + for picture in each_picture(collection, turned) { + program.extend(self.drawn(picture, self.program_scene.as_deref().map(Some))); + on_air.extend(self.on_air_in(picture)); + preview.extend(self.drawn(picture, self.preview_scene.as_deref().map(Some))); + } // One not drawn shows nothing, so it leaves the overlay to the preview. let outputs: Vec<(HashSet<&str>, Instant)> = self .outputs .iter() .filter(|(_, output)| output.draw) - .filter_map(|(scene, output)| Some((self.drawn(collection, [Some(scene.as_str())]), output.start?))) + .filter_map(|(scene, output)| { + let shown = self.drawn(Pictures::single(collection), [Some(scene.as_str())]); + Some((shown, output.start?)) + }) .collect(); for (name, layer) in &mut self.layers { let name = name.as_str(); @@ -1668,9 +1601,9 @@ impl Compositor { /// where it moved since its last paint. One drawn nowhere waits: its CSS /// runs on the program clock, so it lands where it should whenever it is /// drawn again. - fn paint(&mut self, collection: &Collection, now: Instant) { + fn paint(&mut self, collection: &Collection, turned: Option<&Collection>, now: Instant) { let Some(overlays) = &self.overlays else { return }; - let shown = self.shown(collection); + let shown = self.shown(collection, turned); let painting: Vec<_> = self .layers .iter() @@ -1709,21 +1642,35 @@ impl Compositor { /// The sources whose pictures the next render draws: each visible item's /// of preview, program, and each output drawn, through the scenes they - /// draw, a stinger's clip under way, and each mask's, through clones. - fn shown<'a>(&self, collection: &'a Collection) -> HashSet<&'a str> { - let mut shown = self.on_air_sources(collection); - if self.previewing { - shown.extend(self.drawn(collection, [self.preview_scene.as_deref(), None])); + /// draw, in each picture, `collection` and `turned`, the portrait one, a + /// stinger's clip under way, and each mask's, through clones. + fn shown<'a>(&self, collection: &'a Collection, turned: Option<&'a Collection>) -> HashSet<&'a str> { + let mut shown = HashSet::new(); + for picture in each_picture(collection, turned) { + shown.extend(self.on_air_in(picture)); + if self.previewing { + shown.extend(self.drawn(picture, [self.preview_scene.as_deref(), None])); + } } let outputs = self.outputs.iter().filter(|(_, output)| output.draw); - shown.extend(self.drawn(collection, outputs.map(|(scene, _)| Some(scene.as_str())))); + let landscape = Pictures::single(collection); + shown.extend(self.drawn(landscape, outputs.map(|(scene, _)| Some(scene.as_str())))); let thumbnails = self.thumbnails.iter().filter(|(_, thumb)| thumb.due); for (thumbnail, _) in thumbnails { - match self.shows(collection, thumbnail) { - Shows::Scene(scene) => shown.extend(self.drawn(collection, [Some(scene)])), + let (format, shows) = self.shows(collection, turned, thumbnail); + let pictures = match (format, turned) { + (Format::Portrait, Some(turned)) => Pictures { + collection: turned, + landscape: Some(collection), + }, + _ => landscape, + }; + let collection = pictures.collection; + match shows { + Shows::Scene(scene) => shown.extend(self.drawn(pictures, [Some(scene)])), Shows::Source(source) => { if let Some(Nested::Scene(scene)) = collection.nested(source) { - shown.extend(self.drawn(collection, [Some(scene)])); + shown.extend(self.drawn(pictures, [Some(scene)])); } shown.insert(collection.capture(source)); let video = collection.video.get(source); @@ -1739,49 +1686,69 @@ impl Compositor { shown } - /// What `thumbnail` shows this render: a scene on program, in preview - /// while it is drawn, or drawn for an output is sampled from its texture. - fn shows<'a>(&self, collection: &'a Collection, thumbnail: &Thumbnail) -> Shows<'a> { - let scene = match thumbnail { - Thumbnail::Scene(scene) => scene, + /// What `thumbnail` shows this render, and in which picture: a scene on + /// program, in preview while it is drawn, or drawn for an output is + /// sampled from its texture. A scene in portrait shows nothing unless + /// `turned`, the portrait picture, is drawn and draws it. + fn shows<'a>( + &self, + collection: &'a Collection, + turned: Option<&'a Collection>, + thumbnail: &Thumbnail, + ) -> (Format, Shows<'a>) { + let (format, collection, scene) = match thumbnail { + Thumbnail::Scene(scene) => (Format::Landscape, collection, scene), + Thumbnail::Portrait(scene) => { + let turned = turned.filter(|turned| turned.dual(&Place::scene(scene.as_str()))); + let Some(turned) = turned else { + return (Format::Portrait, Shows::Nothing); + }; + (Format::Portrait, turned, scene) + } Thumbnail::Source(source) => { - return match collection.sources.get_key_value(source) { + let shows = match collection.sources.get_key_value(source) { Some((source, _)) => Shows::Source(source), None => Shows::Nothing, }; + return (Format::Landscape, shows); } }; let Some(scene) = collection.scene(scene).map(|scene| scene.name.as_str()) else { - return Shows::Nothing; + return (format, Shows::Nothing); }; - if self.program_scene.as_deref() == Some(scene) { + let shows = if self.program_scene.as_deref() == Some(scene) { Shows::Program } else if self.previewing && self.preview_scene.as_deref() == Some(scene) { Shows::Preview(scene) - } else if self.outputs.get(scene).is_some_and(|output| output.draw) { + } else if format == Format::Landscape && self.outputs.get(scene).is_some_and(|output| output.draw) { Shows::Output(scene) } else { Shows::Scene(scene) - } + }; + (format, shows) } - /// What each thumbnail due this render shows, from how long ago it was - /// named, and the texture it draws into, sized to what it shows. One - /// showing nothing lets its texture go. + /// What each thumbnail due this render shows, in which picture, from how + /// long ago it was named, and the texture it draws into, sized to what it + /// shows. One showing nothing lets its texture go. fn thumbnail_targets<'a>( &mut self, collection: &'a Collection, + turned: Option<&'a Collection>, now: Instant, - ) -> Vec<(Shows<'a>, Duration, Target)> { - let due: Vec<(Thumbnail, Shows)> = self + ) -> Vec<(Format, Shows<'a>, Duration, Target)> { + let due: Vec<(Thumbnail, (Format, Shows))> = self .thumbnails .iter() .filter(|(_, thumb)| thumb.due) - .map(|(thumbnail, _)| (thumbnail.clone(), self.shows(collection, thumbnail))) + .map(|(thumbnail, _)| (thumbnail.clone(), self.shows(collection, turned, thumbnail))) .collect(); - let program = self.program.canvas; let mut jobs = Vec::with_capacity(due.len()); - for (thumbnail, shows) in due { + for (thumbnail, (format, shows)) in due { + let (collection, program) = match (format, turned, &self.portrait) { + (Format::Portrait, Some(turned), Some(view)) => (turned, view.program.canvas), + _ => (collection, self.view.program.canvas), + }; let thumb = self.thumbnails.get_mut(&thumbnail).expect("listed above"); let since = now.saturating_duration_since(thumb.start.unwrap_or(now)); let canvas = match shows { @@ -1818,49 +1785,379 @@ impl Compositor { } _ => Target::sized(&self.device, &self.effects, canvas, size, "moq-scene thumbnail"), }; - thumb.target = Some(target.clone()); - jobs.push((shows, since, target)); + thumb.target = Some(target.clone()); + jobs.push((format, shows, since, target)); + } + jobs + } + + /// The sources whose pictures program draws: each visible item's in the + /// scene on program, the scene a transition leaves, and the top layer, in + /// each picture the show goes out in, through the scenes they draw, a + /// stinger's clip under way, and each mask's, through clones. + pub fn on_air_sources<'a>(&self, collection: &'a Collection) -> HashSet<&'a str> { + let pictures: Vec<_> = match collection.portrait { + true => Format::ALL.iter().map(|format| collection.formatted(*format)).collect(), + false => vec![Cow::Borrowed(collection)], + }; + let mut shown = HashSet::new(); + let (landscape, turned) = (&pictures[0], pictures.get(1).map(|turned| &**turned)); + for picture in each_picture(landscape, turned) { + // Named as `collection` names them, rather than as the pictures do. + let names = self.on_air_in(picture).into_iter(); + let names = names.filter_map(|name| collection.sources.get_key_value(name)); + shown.extend(names.map(|(name, _)| name.as_str())); + } + shown + } + + /// The sources whose pictures program draws in one picture, as it places + /// its scenes, as [`on_air_sources`](Self::on_air_sources) says. + fn on_air_in<'a>(&self, pictures: Pictures<'a>) -> HashSet<&'a str> { + let collection = pictures.collection; + let mut shown = self.drawn(pictures, self.on_air().map(Some).chain([None])); + // Drawn through its source's filters alone. + if let Some((name, _)) = self.stinger().and_then(|name| collection.sources.get_key_value(name)) { + shown.insert(name); + let video = collection.video.get(name); + masks( + collection, + &mut shown, + video.into_iter().flat_map(|video| &video.filters), + ); + } + shown + } + + /// The sources whose pictures `scenes` draw in one picture, `None` being + /// the top layer: each visible item's, through the scenes they draw, and + /// each mask's, through clones. + fn drawn<'a, 'b>( + &self, + pictures: Pictures<'a>, + scenes: impl IntoIterator>, + ) -> HashSet<&'a str> { + let collection = pictures.collection; + let places = scenes.into_iter().map(|scene| Place { + scene: scene.map(str::to_string), + }); + let mut shown = HashSet::new(); + for item in pictures.drawn(places, false) { + shown.insert(collection.capture(&item.source)); + masks(collection, &mut shown, collection.filters(item)); + } + shown + } +} + +/// The landscape picture, `collection`, then the portrait one, `turned`, +/// while the show goes out in it. +fn each_picture<'a>(collection: &'a Collection, turned: Option<&'a Collection>) -> impl Iterator> { + let turned = turned.map(|turned| Pictures { + collection: turned, + landscape: Some(collection), + }); + std::iter::once(Pictures::single(collection)).chain(turned) +} + +/// `thumbnails` without the picture each is in. +fn strip<'a>(thumbnails: Vec<(Format, Shows<'a>, Duration, Target)>) -> Vec<(Shows<'a>, Duration, Target)> { + let thumbnails = thumbnails.into_iter(); + thumbnails + .map(|(_, shows, since, target)| (shows, since, target)) + .collect() +} + +/// What a render draws in each picture: the same scenes on program and in +/// preview, onto that picture's textures. +struct Pass<'a, 's> { + device: &'a wgpu::Device, + effects: &'a mut Effects, + layers: &'a HashMap, + frame: &'a mut Frame, + previewing: bool, + preview: On<'s>, + program: On<'s>, + switch: Option<(&'a Switch, On<'s>, f32)>, +} + +impl Pass<'_, '_> { + /// Draw `view`'s program, its preview unless no one looks at it, + /// `outputs`, and `thumbnails`, from `collection` as that picture places + /// its scenes, the scenes they draw through scene sources included: in + /// the portrait picture, a scene source shows that scene's portrait + /// picture. In the portrait one, `landscape` is the landscape picture's + /// collection and drawn program, which a scene of its own size shows + /// what it draws from. With `keeping`, it also draws the picture under + /// each item keeping clear that is due a read, and returns which. + fn draw( + &mut self, + view: &mut View, + collection: &Collection, + landscape: Option<(&Collection, Picture)>, + outputs: &[(On, Target)], + thumbnails: &[(Shows, Duration, Target)], + keeping: Option, + ) -> Vec<(String, usize)> { + let (landscape, landscape_program) = landscape.unzip(); + let pictures = Pictures { collection, landscape }; + let (preview, program, switch) = (self.preview, self.program, self.switch); + // Each texture at its scene's size: a scene that does not exist, or no + // scene, at the program's. + let program_canvas = view.program.canvas; + let canvas_of = |scene: Option<&str>| collection.canvas(scene).unwrap_or(program_canvas); + let previewing = self.previewing; + let preview_canvas = canvas_of(preview.0); + // The top layer goes over a preview of what could go on program. + let preview_top = preview_canvas.size() == program_canvas.size(); + if previewing { + resize( + &mut view.preview, + self.device, + self.effects, + preview_canvas, + "moq-scene preview", + ); + } + + // Each texture's scenes and top layer, and the scenes they draw. + let mut roots = Vec::new(); + if previewing { + roots.extend(preview.0.map(|scene| (Place::scene(scene), preview.1))); + if preview_top { + roots.push((Place::top(), Duration::ZERO)); + } + } + for On(scene, since) in on_program(program, switch) { + roots.extend(scene.map(|scene| (Place::scene(scene), since))); + } + roots.push((Place::top(), Duration::ZERO)); + for (On(scene, since), _) in outputs { + roots.extend(scene.map(|scene| (Place::scene(scene), *since))); + } + for (shows, since, _) in thumbnails { + if let Shows::Scene(scene) = shows { + roots.push((Place::scene(*scene), *since)); + } + } + // A source's thumbnail draws it whole, as an item of its own. + let wholes: Vec<(Item, Duration)> = thumbnails + .iter() + .filter_map(|(shows, since, target)| match shows { + Shows::Source(source) => Some((Item::new(*source, Rect::fill(&target.canvas)), *since)), + _ => None, + }) + .collect(); + let roots = roots.iter().map(|(place, since)| { + let landscaped = place + .scene + .as_deref() + .is_some_and(|scene| pictures.landscaped(false, scene)); + let items = pictures.of(landscaped).items(place).unwrap_or_default(); + (items, *since, landscaped) + }); + let wholes = wholes + .iter() + .map(|(item, since)| (std::slice::from_ref(item), *since, false)); + let nesting = Nesting::of(pictures, roots.chain(wholes)); + let targets = nested_targets(&mut view.nested, self.device, self.effects, pictures, &nesting); + + // The pictures viewers draw over the program, which the preview draws + // over its scene too, so both show what viewers would see. + let program_layers = crate::layout::pictures(collection, program.0, true); + let preview_layers = crate::layout::pictures(collection, preview.0, preview_top); + view.layered = program_layers.iter().any(|item| item.visible); + if view.layered && view.program.layered.is_none() { + let target = Target::new(self.device, self.effects, program_canvas, "moq-scene layered"); + view.program.layered = Some(target); + } + + let mut draw = Draw { + collection, + pictures, + landscaped: false, + prepared_landscaped: HashMap::new(), + effects: self.effects, + library: Library(self.layers), + frame: self.frame, + size: program_canvas, + prepared: HashMap::new(), + nesting: &nesting, + scenes: HashMap::new(), + program: None, + landscape_program, + }; + // What the program shows is drawn first, then the program, then what + // shows it, then everything else. + let nests = nesting.order.iter().zip(&targets).enumerate(); + let (late, early): (Vec<_>, Vec<_>) = nests.partition(|(_, (nest, _))| nest.program); + for (index, (nest, target)) in early { + draw.nest(index, nest, target); + } + draw.draw_program(&view.program, program, switch); + draw.program = Some(view.program.program.picture.clone()); + for (index, (nest, target)) in late { + draw.nest(index, nest, target); + } + if let Some(layered) = view.program.layered.as_ref().filter(|_| view.layered) { + draw.size = program_canvas; + draw.target(layered); + let canvas = Rect::fill(&program_canvas); + draw.picture(layered, &view.program.program, canvas, canvas, 1.0); + draw.layers(layered, &program_layers, program.1); + } + if previewing { + draw.draw_preview(&view.preview, preview, preview_top); + draw.layers(&view.preview, &preview_layers, preview.1); + } + for (on, target) in outputs { + draw.size = target.canvas; + draw.target(target); + draw.scene(target, *on); + } + for (shows, since, target) in thumbnails { + draw.size = target.canvas; + let fill = Rect::fill(&target.canvas); + match *shows { + Shows::Scene(scene) => { + draw.target(target); + draw.scene(target, On(Some(scene), *since)); + } + Shows::Program => { + // What viewers see, as the Program panel shows it: with the + // published layers over it. + draw.target(target); + draw.picture(target, view.layered(), fill, fill, 1.0); + } + Shows::Preview(_) => { + draw.target(target); + draw.picture(target, &view.preview, fill, fill, 1.0); + } + Shows::Output(scene) => { + let output = outputs.iter().find(|(on, _)| on.0 == Some(scene)); + let (_, output) = output.expect("an output drawn this render"); + draw.target(target); + draw.picture(target, output, fill, fill, 1.0); + } + Shows::Source(source) => { + draw.frame + .target(&target.picture, target.canvas, wgpu::Color::TRANSPARENT); + draw.source(target, source, *since); + } + Shows::Nothing => unreachable!("a thumbnail showing nothing is not drawn"), + } + } + + // The picture under each item keeping clear, read a few times a + // second while its scene is drawn: on program, in preview, for an + // output, or through a scene source. + let mut reads = Vec::new(); + let Some(Keeping { keepers, watched, now }) = keeping else { + return reads; + }; + for ((scene, index), keeper) in keepers { + keeper.watched = false; + let Some(&Watched { since, nested }) = watched.get(scene) else { + continue; + }; + let Some(shown) = collection.scene(scene) else { continue }; + // While its slots glide, a scene draws only its visible items, in + // the order they stack, so one keeping clear is found by what it + // shows and where it was placed, nearest its place in the file. + let rect = keeper.placer.placement().rect; + let mine = |item: &Item| item.keep_clear.is_some() && item.source == keeper.source && item.rect == rect; + let at = shown + .items + .iter() + .enumerate() + .filter(|(_, item)| mine(item)) + .min_by_key(|(at, _)| at.abs_diff(*index)) + .map(|(at, _)| at); + let Some(at) = at.filter(|&at| shown.items[at].visible) else { + continue; + }; + keeper.watched = true; + if !keeper.look.due(now) { + continue; + } + let canvas = canvas_of(Some(scene)); + let target = keeper.look.target(draw.effects, canvas); + draw.size = canvas; + // What a scene source draws starts transparent, as its nest does. + match nested { + true => draw.frame.target(&target.picture, canvas, wgpu::Color::TRANSPARENT), + false => draw.target(&target), + } + draw.under(&target, Place::scene(scene.as_str()), at, since); + reads.push((scene.clone(), *index)); } - jobs + reads } +} - /// The sources whose pictures program draws: each visible item's in the - /// scene on program, the scene a transition leaves, and the top layer, - /// through the scenes they draw, a stinger's clip under way, and each - /// mask's, through clones. - pub fn on_air_sources<'a>(&self, collection: &'a Collection) -> HashSet<&'a str> { - let mut shown = self.drawn(collection, self.on_air().map(Some).chain([None])); - // Drawn through its source's filters alone. - if let Some((name, _)) = self.stinger().and_then(|name| collection.sources.get_key_value(name)) { - shown.insert(name); - let video = collection.video.get(name); - masks( - collection, - &mut shown, - video.into_iter().flat_map(|video| &video.filters), - ); - } - shown - } +/// The items keeping clear whose pictures a [`Pass`] reads, and each scene +/// the render draws, as [`Compositor::watched`] finds them, as of `now`. +struct Keeping<'a> { + keepers: &'a mut HashMap<(String, usize), Keeper>, + watched: &'a HashMap, + now: Instant, +} - /// The sources whose pictures `scenes` draw, `None` being the top layer: - /// each visible item's, through the scenes they draw, and each mask's, - /// through clones. - fn drawn<'a, 'b>( - &self, - collection: &'a Collection, - scenes: impl IntoIterator>, - ) -> HashSet<&'a str> { - let places = scenes.into_iter().map(|scene| Place { - scene: scene.map(str::to_string), - }); - let mut shown = HashSet::new(); - for item in collection.drawn(places) { - shown.insert(collection.capture(&item.source)); - masks(collection, &mut shown, collection.filters(item)); +/// `staged`, one picture's collection, with each scene's items in slots +/// where its arrangement puts them at `now`, gliding there as a move +/// transition does when a slot opens or closes, following `reflows`, that +/// picture's. A slot closes while its source is offline or failed in +/// `sources`, and the picture in it is cropped to cover it by the size of +/// its newest picture in `layers`, or of the scene it shows in that picture. +fn reflow<'a>( + reflows: &mut HashMap, + layers: &HashMap, + mut staged: Cow<'a, Collection>, + sources: &Sources, + now: Instant, +) -> Cow<'a, Collection> { + let collection: &Collection = &staged; + let present = |name: &str| { + let feed = sources.get(collection.capture(name)); + !feed.is_some_and(|feed| matches!(feed.status(), Status::Offline | Status::Failed(_))) + }; + let size = |name: &str| match collection.nested(name) { + Some(Nested::Scene(scene)) => collection + .canvas(Some(scene)) + .map(|canvas| (canvas.width, canvas.height)), + Some(Nested::Program) => Some((collection.canvas.width, collection.canvas.height)), + None => { + let layer = layers.get(collection.capture(name))?; + layer.picture.as_ref().map(Picture::size) } - shown + }; + let mut kept = HashMap::new(); + let mut placed = Vec::new(); + for (index, scene) in collection.scenes.iter().enumerate() { + let Some(arranged) = collection.arranged(&scene.name, present, size) else { + continue; + }; + let mut reflow = reflows.remove(&scene.name).unwrap_or_else(|| Reflow { + arranged: Vec::new(), + // Only the slots glide, so a layer set free that is being + // dragged follows the pointer. + glide: Glide::new(REFLOW, |item| item.slot.is_some()), + }); + // Only the slots start a glide: what each shows, whether it is + // open, and where. + let slotted = arranged.iter().filter(|item| item.slot.is_some()); + let key = slotted + .map(|item| (item.source.clone(), item.visible, item.rect)) + .collect(); + placed.push((index, reflow.glide.show(key, &arranged, now).to_vec())); + reflow.arranged = arranged; + kept.insert(scene.name.clone(), reflow); + } + for (index, items) in placed { + staged.to_mut().scenes[index].items = items; } + *reflows = kept; + staged } /// `target` at `canvas`'s size, made anew if it was another, labeled `label`. @@ -1899,18 +2196,21 @@ fn nested_targets( pool: &mut HashMap>, device: &wgpu::Device, effects: &Effects, - collection: &Collection, + pictures: Pictures, nesting: &Nesting, ) -> Vec { let mut free = std::mem::take(pool); let mut targets = Vec::with_capacity(nesting.order.len()); for nest in &nesting.order { + let collection = pictures.of(nest.landscaped); let canvas = collection.canvas(Some(nest.scene)).unwrap_or(collection.canvas); - let reused = free.get_mut(nest.scene).and_then(Vec::pop); - let target = match reused { - Some(target) if target.canvas.size() == canvas.size() => target, - _ => Target::new(device, effects, canvas, "moq-scene nested"), - }; + // One of its size: the portrait picture may draw a scene both turned + // and, through a scene of its own size, as the landscape one does. + let reused = free.get_mut(nest.scene).and_then(|free| { + let at = free.iter().position(|target| target.canvas.size() == canvas.size())?; + Some(free.swap_remove(at)) + }); + let target = reused.unwrap_or_else(|| Target::new(device, effects, canvas, "moq-scene nested")); pool.entry(nest.scene.to_string()).or_default().push(target.clone()); targets.push(target); } @@ -2047,14 +2347,87 @@ fn on_program<'a>(program: On<'a>, switch: Option<(&Switch, On<'a>, f32)>) -> Ve } } +/// The scenes one picture draws: `collection`, as that picture places them, +/// and, in the portrait picture, `landscape`, the landscape one. A scene of +/// its own size is drawn in one picture, the same in both, so in the portrait +/// picture it and every scene it draws, the program included, are drawn as +/// the landscape picture draws them. +#[derive(Clone, Copy)] +struct Pictures<'a> { + collection: &'a Collection, + landscape: Option<&'a Collection>, +} + +impl<'a> Pictures<'a> { + /// The landscape picture, or a picture of one format alone. + fn single(collection: &'a Collection) -> Self { + Self { + collection, + landscape: None, + } + } + + /// Whether the scene called `scene` is drawn as the landscape picture + /// draws it, drawn by one that is, as `outer` says: in the portrait + /// picture, a scene of its own size and all it draws. + fn landscaped(&self, outer: bool, scene: &str) -> bool { + self.landscape.is_some() && (outer || !self.collection.dual(&Place::scene(scene))) + } + + /// The collection a scene drawn as `landscaped` says is read from. + fn of(&self, landscaped: bool) -> &'a Collection { + match (landscaped, self.landscape) { + (true, Some(landscape)) => landscape, + _ => self.collection, + } + } + + /// The visible items `places` draw, drawn by one drawn as the landscape + /// picture draws it if `outer`, then those of each scene they draw + /// through a scene source, and so on, each scene once, as + /// [`Collection::drawn`] has them, read as [`of`](Self::of) says. + fn drawn(&self, places: impl IntoIterator, outer: bool) -> Vec<&'a Item> { + let mut items: Vec<(&'a Item, bool)> = Vec::new(); + for place in places { + let landscaped = match place.scene.as_deref() { + Some(scene) => self.landscaped(outer, scene), + None => outer && self.landscape.is_some(), + }; + let found = self.of(landscaped).items(&place).unwrap_or_default(); + items.extend(found.iter().filter(|item| item.visible).map(|item| (item, landscaped))); + } + let mut nested = HashSet::new(); + let mut next = 0; + while let Some(&(item, outer)) = items.get(next) { + if let Some(Nested::Scene(scene)) = self.collection.nested(&item.source) { + let landscaped = self.landscaped(outer, scene); + if nested.insert((scene, landscaped)) + && let Some(found) = self.of(landscaped).scene(scene) + { + items.extend( + found + .items + .iter() + .filter(|item| item.visible) + .map(|item| (item, landscaped)), + ); + } + } + next += 1; + } + items.into_iter().map(|(item, _)| item).collect() + } +} + /// The scenes a render draws through scene sources, each at the moment of its /// keyframes it is shown at. struct Nesting<'a> { /// In an order that draws each after the scenes it shows. order: Vec>, /// Which of `order` draws a scene shown in a scene that went on a while - /// ago, by the scene's name and that while. - drawn: HashMap<(&'a str, Duration), usize>, + /// ago, by the scene's name, that while, and whether it is drawn as the + /// landscape picture draws it. + drawn: HashMap<(&'a str, Duration, bool), usize>, } /// A scene drawn through a scene source. @@ -2068,12 +2441,16 @@ struct Nest<'a> { moment: Option, /// Whether it shows the program, itself or through the scenes it draws. program: bool, + /// Whether it is drawn as the landscape picture draws it, as + /// [`Pictures::landscaped`] says. + landscaped: bool, } impl<'a> Nesting<'a> { - /// The scenes each of `roots`, a run of items, draws, from when the scene - /// they are in went on, and those they draw in turn. - fn of<'r>(collection: &'a Collection, roots: impl IntoIterator) -> Self + /// The scenes each of `roots`, a run of `pictures`' items, draws, from + /// when the scene they are in went on, and those they draw in turn, each + /// with whether the run is drawn as the landscape picture draws it. + fn of<'r>(pictures: Pictures<'a>, roots: impl IntoIterator) -> Self where 'a: 'r, { @@ -2082,21 +2459,23 @@ impl<'a> Nesting<'a> { drawn: HashMap::new(), }; let mut walked = HashMap::new(); - for (items, since) in roots { - nesting.walk(collection, items, since, &mut walked); + for (items, since, landscaped) in roots { + nesting.walk(pictures, items, since, landscaped, &mut walked); } nesting } /// Add the scenes `items` show, from when the scene they are in went on - /// `since` ago. `walked` holds each scene and moment added, or being + /// `since` ago, drawn as the landscape picture draws them if + /// `landscaped`. `walked` holds each scene and moment added, or being /// walked. fn walk<'r>( &mut self, - collection: &'a Collection, + pictures: Pictures<'a>, items: &'r [Item], since: Duration, - walked: &mut HashMap<(&'a str, Option), Option>, + landscaped: bool, + walked: &mut HashMap<(&'a str, Option, bool), Option>, ) where 'a: 'r, { @@ -2108,6 +2487,8 @@ impl<'a> Nesting<'a> { next: usize, /// Whether any item walked shows the program. program: bool, + /// Whether it is drawn as the landscape picture draws it. + landscaped: bool, } // A stack of its own rather than the call stack, which a long enough @@ -2117,6 +2498,7 @@ impl<'a> Nesting<'a> { items, next: 0, program: false, + landscaped, }]; while let Some(walk) = stack.last_mut() { let items = walk.items; @@ -2124,6 +2506,7 @@ impl<'a> Nesting<'a> { let Some(Walk { nest: Some((scene, moment)), program, + landscaped, .. }) = stack.pop() else { @@ -2134,10 +2517,11 @@ impl<'a> Nesting<'a> { since, moment, program, + landscaped, }); let index = self.order.len() - 1; - walked.insert((scene, moment), Some(index)); - self.drawn.insert((scene, since), index); + walked.insert((scene, moment, landscaped), Some(index)); + self.drawn.insert((scene, since, landscaped), index); stack.last_mut().expect("a nest is walked from items").program |= program; continue; }; @@ -2145,7 +2529,7 @@ impl<'a> Nesting<'a> { if !item.visible { continue; } - let scene = match collection.nested(&item.source) { + let scene = match pictures.collection.nested(&item.source) { Some(Nested::Scene(scene)) => scene, Some(Nested::Program) => { walk.program = true; @@ -2153,28 +2537,31 @@ impl<'a> Nesting<'a> { } None => continue, }; - if let Some(&index) = self.drawn.get(&(scene, since)) { + let landscaped = pictures.landscaped(walk.landscaped, scene); + if let Some(&index) = self.drawn.get(&(scene, since, landscaped)) { walk.program |= self.order[index].program; continue; } - let moment = moment(collection, scene, since); - match walked.get(&(scene, moment)) { + let moment = moment(pictures, scene, landscaped, since); + match walked.get(&(scene, moment, landscaped)) { Some(Some(index)) => { let index = *index; - self.drawn.insert((scene, since), index); + self.drawn.insert((scene, since, landscaped), index); walk.program |= self.order[index].program; } // A scene drawing itself, which validation refuses: it shows // nothing there rather than walking forever. Some(None) => {} None => { - walked.insert((scene, moment), None); - let items = collection.scene(scene).map_or(&[][..], |scene| scene.items.as_slice()); + walked.insert((scene, moment, landscaped), None); + let found = pictures.of(landscaped).scene(scene); + let items = found.map_or(&[][..], |scene| scene.items.as_slice()); stack.push(Walk { nest: Some((scene, moment)), items, next: 0, program: false, + landscaped, }); } } @@ -2185,15 +2572,23 @@ impl<'a> Nesting<'a> { /// The moment of its keyframes the scene called `scene` is drawn at, `since` /// the scene drawing it went on, or `None` once every item it draws, through /// the scenes it draws, holds its last keyframe. -fn moment(collection: &Collection, scene: &str, since: Duration) -> Option { - let items = collection.drawn([Place::scene(scene)]); +fn moment(pictures: Pictures, scene: &str, landscaped: bool, since: Duration) -> Option { + let items = pictures.drawn([Place::scene(scene)], landscaped); let moving = items.iter().any(|item| item.settled(since) != Settled::After); moving.then_some(since) } /// One render's draws onto its targets. struct Draw<'a> { + /// The collection the items drawn now are read from: one of `pictures`. collection: &'a Collection, + pictures: Pictures<'a>, + /// Whether the items drawn now are drawn as the landscape picture draws + /// them, as [`Pictures::landscaped`] says. + landscaped: bool, + /// `prepared` for the items drawn while `landscaped` swaps, so an item + /// placed differently in each picture is not shared. + prepared_landscaped: HashMap<(Place, usize, Settled), Option>, effects: &'a mut Effects, library: Library<'a>, frame: &'a mut Frame, @@ -2210,16 +2605,33 @@ struct Draw<'a> { scenes: HashMap, /// The program, once drawn. program: Option, + /// The landscape picture's program, which the portrait picture shows + /// through a scene of its own size. + landscape_program: Option, } impl Draw<'_> { /// Draw the scene `nest` names, the `index`th of the nesting, into /// `target`, transparent where it draws nothing, for items to show. fn nest(&mut self, index: usize, nest: &Nest, target: &Target) { + let outer = self.landscape(nest.landscaped); self.size = target.canvas; self.frame.target(&target.picture, self.size, wgpu::Color::TRANSPARENT); self.layer(target, Place::scene(nest.scene), nest.since); self.scenes.insert(index, target.picture.clone()); + self.landscape(outer); + } + + /// Read the items drawn from now on as the landscape picture draws them, + /// or not, as [`Pictures::landscaped`] says; whether they were. + fn landscape(&mut self, landscaped: bool) -> bool { + let was = self.landscaped; + if landscaped != was { + self.landscaped = landscaped; + self.collection = self.pictures.of(landscaped); + std::mem::swap(&mut self.prepared, &mut self.prepared_landscaped); + } + was } /// Draw `preview` showing the scene `on`, under the top layer if `top`. @@ -2326,12 +2738,18 @@ impl Draw<'_> { /// Draw `place`'s items under its `index`th onto `target`, as /// [`layer`](Self::layer) draws them: what that item covers. fn under(&mut self, target: &Target, place: Place, index: usize, since: Duration) { + let landscaped = place + .scene + .as_deref() + .is_some_and(|scene| self.pictures.landscaped(self.landscaped, scene)); + let outer = self.landscape(landscaped); let items = self.collection.items(&place).unwrap_or_default(); let under = items.iter().enumerate().take(index); for (index, item) in under.filter(|(_, item)| item.visible) { let key = (place.clone(), index, self.settled(item, since)); self.item(target, key, &item.posed(since), since); } + self.landscape(outer); } /// Draw `layers`, pictures of the layer layout, onto `target`, back to @@ -2366,7 +2784,7 @@ impl Draw<'_> { Some(Nested::Scene(scene)) => self .nesting .drawn - .get(&(scene, since)) + .get(&(scene, since, self.pictures.landscaped(self.landscaped, scene))) .is_some_and(|&index| self.nesting.order[index].moment.is_some()), _ => false, }; @@ -2569,9 +2987,11 @@ impl Draw<'_> { fn shows(&self, item: &Item, since: Duration) -> Option { match self.collection.nested(&item.source) { Some(Nested::Scene(scene)) => { - let index = self.nesting.drawn.get(&(scene, since))?; + let landscaped = self.pictures.landscaped(self.landscaped, scene); + let index = self.nesting.drawn.get(&(scene, since, landscaped))?; self.scenes.get(index).cloned() } + Some(Nested::Program) if self.landscaped => self.landscape_program.clone(), Some(Nested::Program) => self.program.clone(), // Its own track carries a published source to viewers instead, // as it does a clone's, which shows the same capture, and only @@ -3134,6 +3554,56 @@ mod tests { .await; } + /// While the show goes out in both, a camera keeping clear places only the + /// landscape picture: in the portrait one it stays where the place saved + /// in the scene file puts it, fitted, rather than following its move. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just render-sw"] + async fn keeping_clear_places_only_the_landscape_picture() { + local(async { + let mut collection = busy(); + collection.portrait = true; + collection.validate().unwrap(); + let mut studio = Studio::new(collection).await; + studio.compositor.fit(&studio.collection).unwrap(); + studio.compositor.set_preview(Some("busy".into())); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + let camera = studio.collection.scenes[0].items.len() - 1; + let block = camera - 1; + let mut now = Instant::now(); + let mut moved = None; + for frame in 0..40u32 { + studio.collection.scenes[0].items[block].rect.x = 240.0 + 6.0 * (frame % 7) as f32; + studio.render(now); + studio.device.poll(wgpu::PollType::wait_indefinitely()).unwrap(); + if studio + .compositor + .placement("busy", camera, "camera") + .unwrap() + .slot + .is_some() + { + moved = Some(now); + break; + } + now = now.saturating_add(Duration::from_millis(100)); + } + let moved = moved.expect("it chose a slot"); + studio.render(moved.saturating_add(crate::KeepClear::default().glide)); + let program = studio.program(); + assert_eq!(texel(&program, 320, 41, 41), BLUE, "moved in landscape"); + + // Fitted from where the scene file saves it: 36x36, its center + // at about (157, 247) of the 180x320 portrait canvas. + let (texture, _) = studio.compositor.portrait().unwrap(); + let texture = texture.clone(); + let turned = studio.read(&texture); + assert_eq!(texel(&turned, 180, 157, 247), BLUE, "where it was saved"); + assert_ne!(texel(&turned, 180, 23, 73), BLUE, "not where it moved"); + }) + .await; + } + /// Each scene's keepers follow a rename, so two scenes swapping names /// keep their own, and a removed scene's are dropped. #[test] @@ -3593,6 +4063,334 @@ mod tests { .await; } + /// An 8x4 show going out in portrait too, at 4x8. Scene "a": red filling + /// each picture, blue over the landscape one's right half alone, and + /// green over the portrait one's bottom half alone. Scene "b": blue + /// filling the landscape picture, fitted in the portrait one. + fn dual() -> Collection { + let mut collection = collection(); + collection.canvas = Canvas { + width: 8, + height: 4, + framerate: 30, + }; + collection.portrait = true; + collection + .sources + .insert("green".into(), Source::Color { color: Rgba(GREEN) }); + let rect = |x, y, width, height| Rect { x, y, width, height }; + let placed = |rect| { + Some(crate::Placement { + rect, + crop: Crop::default(), + }) + }; + collection.scenes = vec![ + Scene { + items: vec![ + Item { + portrait: placed(rect(0.0, 0.0, 4.0, 8.0)), + ..Item::new("red", rect(0.0, 0.0, 8.0, 4.0)) + }, + Item { + format: Some(Format::Landscape), + ..Item::new("blue", rect(4.0, 0.0, 4.0, 4.0)) + }, + Item { + format: Some(Format::Portrait), + portrait: placed(rect(0.0, 4.0, 4.0, 4.0)), + ..Item::new("green", rect(0.0, 0.0, 8.0, 4.0)) + }, + ], + ..Scene::new("a") + }, + Scene { + items: vec![Item::new("blue", rect(0.0, 0.0, 8.0, 4.0))], + ..Scene::new("b") + }, + ]; + collection.validate().unwrap(); + collection + } + + /// Each picture draws the scene on program as it is arranged there, a + /// layer shown in one format only in that one, and a cut and a fade go + /// through both at once. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just render-sw"] + async fn both_pictures_follow_the_program() { + local(async { + let mut studio = Studio::new(dual()).await; + studio.compositor.fit(&studio.collection).unwrap(); + let portrait = |studio: &mut Studio| { + let (texture, canvas) = studio.compositor.portrait().unwrap(); + assert_eq!((canvas.width, canvas.height), (4, 8)); + let texture = texture.clone(); + studio.read(&texture) + }; + let start = Instant::now(); + studio.compositor.set_preview(Some("a".into())); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + studio.render(start); + let landscape = studio.program(); + assert_eq!(pixel(&landscape, 1, 1), RED); + assert_eq!(pixel(&landscape, 6, 1), BLUE, "shown in landscape"); + let turned = portrait(&mut studio); + assert_eq!(texel(&turned, 4, 1, 1), RED); + assert_eq!(texel(&turned, 4, 1, 6), GREEN, "shown in portrait"); + assert!(!turned.as_chunks::<4>().0.contains(&BLUE), "shown in landscape alone"); + + // "b" fills the landscape picture and is fitted, 4x2, across the + // middle of the portrait one. + studio.compositor.set_preview(Some("b".into())); + studio + .compositor + .transition(&studio.collection, FADE, &studio.sources, start) + .unwrap(); + studio.render(start.saturating_add(Duration::from_millis(500))); + let turned = portrait(&mut studio); + let fading = texel(&turned, 4, 1, 6); + assert!((126..=129).contains(&fading[1]) && fading[0] == 0, "{fading:?}"); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + studio.render(start.saturating_add(Duration::from_millis(600))); + let turned = portrait(&mut studio); + assert_eq!(texel(&turned, 4, 1, 3), BLUE); + assert_eq!(texel(&turned, 4, 1, 1), BLACK); + assert_eq!(texel(&turned, 4, 1, 6), BLACK); + }) + .await; + } + + /// A layer showing a scene of the program's size shows the scene's + /// portrait arrangement in the portrait picture, 9:16, at a place shaped + /// for it: one covering the landscape canvas covers the portrait one. + /// The landscape picture shows its landscape arrangement. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just render-sw"] + async fn a_scene_source_shows_the_portrait_arrangement_in_portrait() { + local(async { + let mut collection = dual(); + collection + .sources + .insert("scene a".into(), Source::Scene { scene: "a".into() }); + let item = Item::new("scene a", Rect::fill(&collection.canvas)); + collection.scenes.push(Scene { + items: vec![item.clone()], + ..Scene::new("c") + }); + collection.validate().unwrap(); + let placed = collection.placement(&item, Format::Portrait); + assert_eq!(placed.rect, Rect::fill(&collection.canvas.turned())); + let mut studio = Studio::new(collection).await; + studio.compositor.fit(&studio.collection).unwrap(); + studio.compositor.set_preview(Some("c".into())); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + studio.render(Instant::now()); + let landscape = studio.program(); + assert_eq!(pixel(&landscape, 1, 1), RED); + assert_eq!(pixel(&landscape, 6, 1), BLUE); + assert!( + !landscape.as_chunks::<4>().0.contains(&GREEN), + "a's portrait-only layer" + ); + let (texture, _) = studio.compositor.portrait().unwrap(); + let texture = texture.clone(); + let turned = studio.read(&texture); + // "a" in portrait: red, green over its bottom half. + assert_eq!(texel(&turned, 4, 1, 1), RED); + assert_eq!(texel(&turned, 4, 1, 6), GREEN); + assert!(!turned.as_chunks::<4>().0.contains(&BLUE), "a's landscape-only layer"); + }) + .await; + } + + /// A scene of its own size is drawn in one picture, the same in both: in + /// the portrait picture, a scene it shows is drawn as the landscape + /// picture draws it, rather than turned and squeezed into its place. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just render-sw"] + async fn a_scene_of_its_own_size_shows_the_landscape_picture_in_portrait() { + local(async { + let mut collection = dual(); + for (name, scene) in [("scene a", "a"), ("scene w", "w")] { + let scene = scene.into(); + collection.sources.insert(name.into(), Source::Scene { scene }); + } + let size = crate::Size { width: 16, height: 8 }; + let wide = Canvas { + width: 16, + height: 8, + framerate: 30, + }; + collection.scenes.push(Scene { + size: Some(size), + items: vec![Item::new("scene a", Rect::fill(&wide))], + ..Scene::new("w") + }); + // "a" shown directly too, behind "w", so the portrait picture draws + // it both turned and as the landscape picture does. + let behind = Rect { + x: 3.0, + y: 1.5, + width: 2.0, + height: 0.5, + }; + collection.scenes.push(Scene { + items: vec![ + Item::new("scene a", behind), + Item::new("scene w", Rect::fill(&collection.canvas)), + ], + ..Scene::new("c") + }); + collection.validate().unwrap(); + let mut studio = Studio::new(collection).await; + studio.compositor.fit(&studio.collection).unwrap(); + studio.compositor.set_preview(Some("c".into())); + studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); + studio.render(Instant::now()); + let (texture, _) = studio.compositor.portrait().unwrap(); + let texture = texture.clone(); + let turned = studio.read(&texture); + // "w" fitted, 4x2, across the middle: "a" in landscape, red with + // blue over its right half. + assert_eq!(texel(&turned, 4, 0, 3), RED); + assert_eq!(texel(&turned, 4, 3, 4), BLUE); + assert_eq!(texel(&turned, 4, 1, 1), BLACK); + assert!(!turned.as_chunks::<4>().0.contains(&GREEN), "a's portrait-only layer"); + + // Each of "a"'s two pictures keeps its texture from one render to + // the next. + let nested = |studio: &Studio| { + let view = studio.compositor.portrait.as_ref().unwrap(); + let mut targets: Vec = + view.nested["a"].iter().map(|target| target.texture.clone()).collect(); + targets.sort_by_key(|texture| texture.width()); + targets + }; + let drawn = nested(&studio); + assert_eq!(drawn.len(), 2, "turned and as in landscape"); + for _ in 0..2 { + studio.render(Instant::now()); + assert_eq!(nested(&studio), drawn); + } + }) + .await; + } + + /// In the portrait picture, a scene of its own size moves as the + /// landscape picture draws what it shows: a layer moving in landscape + /// alone keeps it moving, so its draws at two moments are not shared. + #[test] + fn a_scene_of_its_own_size_moves_as_the_landscape_picture_does() { + let mut collection = dual(); + for (name, scene) in [("scene a", "a"), ("scene w", "w")] { + let scene = scene.into(); + collection.sources.insert(name.into(), Source::Scene { scene }); + } + collection.scenes.push(Scene { + size: Some(crate::Size { width: 16, height: 8 }), + items: vec![Item::new( + "scene a", + Rect { + x: 0.0, + y: 0.0, + width: 16.0, + height: 8.0, + }, + )], + ..Scene::new("w") + }); + // "d", drawn in both, shows "w". + collection.scenes.push(Scene { + items: vec![Item::new("scene w", Rect::fill(&collection.canvas))], + ..Scene::new("d") + }); + // Blue, shown in landscape alone, slides in. + collection.scenes[0].items[1].keyframes = slide_in(); + collection.validate().unwrap(); + let landscape = collection.formatted(Format::Landscape); + let portrait = collection.formatted(Format::Portrait); + let pictures = Pictures { + collection: &portrait, + landscape: Some(&landscape), + }; + let since = Duration::from_millis(500); + assert!(pictures.landscaped(false, "w")); + assert_eq!(moment(pictures, "w", true, since), Some(since)); + assert_eq!(moment(pictures, "d", false, since), Some(since), "through `w`"); + assert_eq!(moment(pictures, "a", false, since), None, "still in portrait"); + assert_eq!(moment(Pictures::single(&landscape), "a", false, since), Some(since)); + } + + /// A slot showing a scene in the portrait picture is cropped for the + /// scene's portrait picture, which the scene source shows there. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just render-sw"] + async fn a_slot_showing_a_scene_crops_its_portrait_picture_in_portrait() { + local(async { + let mut collection = dual(); + collection + .sources + .insert("scene b".into(), Source::Scene { scene: "b".into() }); + collection.scenes.push(Scene { + items: vec![Item { + slot: Some(crate::Slot::Main), + ..Item::new("scene b", Rect::default()) + }], + ..Scene::new("s") + }); + collection.validate().unwrap(); + let mut studio = Studio::new(collection).await; + studio.compositor.fit(&studio.collection).unwrap(); + studio.render(Instant::now()); + let turned = studio.collection.formatted(Format::Portrait); + let landscape = turned.arranged("s", |_| true, |_| Some((8, 4))).unwrap(); + let portrait = turned.arranged("s", |_| true, |_| Some((4, 8))).unwrap(); + assert_ne!(landscape, portrait); + let arranged = studio.compositor.arranged(Format::Portrait, "s").unwrap(); + assert_eq!(arranged, portrait.as_slice()); + }) + .await; + } + + /// A scene's portrait thumbnail draws it as the portrait picture + /// arranges it, and the portrait preview the scene in preview. + #[tokio::test] + #[ignore = "needs a Vulkan device; run by just render-sw"] + async fn a_portrait_thumbnail_and_preview_draw_the_portrait_arrangement() { + local(async { + let mut studio = Studio::new(dual()).await; + studio.compositor.fit(&studio.collection).unwrap(); + let detail = Detail { side: 8, rate: 30 }; + studio.compositor.set_thumbnails([ + (Thumbnail::Portrait("a".into()), detail), + (Thumbnail::Scene("a".into()), detail), + ]); + studio.compositor.set_preview(Some("a".into())); + studio.render(Instant::now()); + let texture = studio + .compositor + .thumbnail(&Thumbnail::Portrait("a".into())) + .unwrap() + .clone(); + assert_eq!((texture.width(), texture.height()), (4, 8)); + let thumbnail = studio.read(&texture); + assert_eq!(texel(&thumbnail, 4, 1, 1), RED); + assert_eq!(texel(&thumbnail, 4, 1, 6), GREEN); + let texture = studio + .compositor + .thumbnail(&Thumbnail::Scene("a".into())) + .unwrap() + .clone(); + assert_eq!((texture.width(), texture.height()), (8, 4)); + let (texture, _) = studio.compositor.portrait_preview().unwrap(); + let texture = texture.clone(); + let preview = studio.read(&texture); + assert_eq!(texel(&preview, 4, 1, 6), GREEN); + }) + .await; + } + /// Halfway through a slide to the left, eased to half the canvas, the /// outgoing scene's right half shows on the left and the incoming scene's /// left half on the right; a wipe leaves the outgoing scene in place and @@ -3780,7 +4578,7 @@ mod tests { studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); let start = Instant::now(); studio.render(start); - let before = studio.compositor.arranged("slots").unwrap().to_vec(); + let before = studio.compositor.arranged(Format::Landscape, "slots").unwrap().to_vec(); let width = 192; let program = studio.program(); for (item, color) in before.iter().zip([RED, BLUE, GREEN]) { @@ -3795,7 +4593,7 @@ mod tests { studio.collection.scenes[0].items[1].visible = false; studio.render(start); - let after = studio.compositor.arranged("slots").unwrap().to_vec(); + let after = studio.compositor.arranged(Format::Landscape, "slots").unwrap().to_vec(); assert!(!after[1].visible, "a hidden item takes no slot"); assert_ne!(after[2].rect, before[2].rect, "green moves up"); @@ -3857,8 +4655,8 @@ mod tests { studio.collection.apply(&edit).unwrap(); studio.render(start.saturating_add(REFLOW / 2)); for scene in ["slots", "row"] { - let arranged = studio.compositor.arranged(scene).unwrap().to_vec(); - let placed = studio.compositor.placed(scene).unwrap().to_vec(); + let arranged = studio.compositor.arranged(Format::Landscape, scene).unwrap().to_vec(); + let placed = studio.compositor.placed(Format::Landscape, scene).unwrap().to_vec(); assert_eq!(placed, arranged, "`{scene}` is where it was, with no glide"); } }) @@ -3881,7 +4679,7 @@ mod tests { studio.compositor.set_preview(Some("slots".into())); studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); studio.render(Instant::now()); - let arranged = studio.compositor.arranged("slots").unwrap().to_vec(); + let arranged = studio.compositor.arranged(Format::Landscape, "slots").unwrap().to_vec(); assert!(!arranged[0].visible, "an unfilled role takes no slot"); assert!(!arranged[1].visible, "an offline guest takes no slot"); let (_, alone) = Arrangement::Column.places(studio.collection.canvas, false, 1); @@ -3896,7 +4694,7 @@ mod tests { let filled = Instant::now(); studio.render(filled); studio.render(filled.saturating_add(REFLOW)); - let arranged = studio.compositor.arranged("slots").unwrap().to_vec(); + let arranged = studio.compositor.arranged(Format::Landscape, "slots").unwrap().to_vec(); assert!(arranged[0].visible, "a filled role takes the main slot"); let (x, y) = center(&arranged[0].rect); assert_eq!( @@ -4223,7 +5021,7 @@ mod tests { ..Default::default() }, ); - let shown = studio.compositor.shown(&collection); + let shown = studio.compositor.shown(&collection, None); assert!(shown.contains("sting") && shown.contains("red"), "{shown:?}"); sting(&mut studio, Filter::Brightness { amount: 0.0 }); @@ -5774,7 +6572,7 @@ mod tests { assert_eq!(pixel(&program, 3, 4), BLUE, "the red scene's corner"); assert_eq!(pixel(&program, 6, 6), BLUE, "under it"); assert_eq!(pixel(&program, 0, 0), RED, "shown twice"); - assert_eq!(studio.compositor.nested.get("red").map(Vec::len), Some(1)); + assert_eq!(studio.compositor.view.nested.get("red").map(Vec::len), Some(1)); studio.compositor.set_preview(Some("blue".into())); studio.compositor.cut(&studio.collection, &studio.sources).unwrap(); @@ -5782,7 +6580,7 @@ mod tests { let program = studio.program(); assert_eq!(pixel(&program, 1, 1), RED, "through the nested scene"); assert_eq!(pixel(&program, 6, 1), BLUE); - let drawn: Vec<&String> = studio.compositor.nested.keys().collect(); + let drawn: Vec<&String> = studio.compositor.view.nested.keys().collect(); assert_eq!(drawn, ["corner"], "only what is drawn"); }) .await; @@ -5966,12 +6764,12 @@ mod tests { assert_eq!((pixel(&program, 5, 1), pixel(&program, 6, 1)), (BLUE, BLACK)); let preview = studio.preview(); assert_eq!((pixel(&preview, 1, 1), pixel(&preview, 2, 1)), (BLUE, BLACK)); - assert_eq!(studio.compositor.nested["blue"].len(), 2, "one per moment"); + assert_eq!(studio.compositor.view.nested["blue"].len(), 2, "one per moment"); studio.render(at(2000)); assert_eq!(pixel(&studio.program(), 7, 1), BLUE); assert_eq!(pixel(&studio.preview(), 7, 1), BLUE); - assert_eq!(studio.compositor.nested["blue"].len(), 1, "settled alike"); + assert_eq!(studio.compositor.view.nested["blue"].len(), 1, "settled alike"); }) .await; } @@ -6511,6 +7309,30 @@ mod bench { .await; } + /// What going out in portrait too costs, over the program and the same + /// scene in preview: landscape only, both formats at once, and the long + /// way before it, a published portrait scene placing every source. + #[tokio::test] + #[ignore = "a benchmark; run by hand on a GPU"] + async fn both_formats() { + local(async { + println!("sources case frame passes copies"); + for sources in [4, 16] { + let landscape = collection(sources, 0, sources); + let mut both = landscape.clone(); + both.portrait = true; + both.validate().unwrap(); + let published = collection(sources, 1, sources); + let scene = landscape.scenes[0].name.clone(); + for (case, collection) in [("landscape", landscape), ("both", both), ("published", published)] { + let (frame, (passes, copies)) = cost(collection, &scene).await; + println!("{sources:>7} {case:<10} {frame:>7.2?} {passes:>6} {copies:>6}"); + } + } + }) + .await; + } + /// What drawing sources published independently as layers costs, over the /// program and the same scene in preview: none, then the last of the /// sources, then the last four. diff --git a/rs/moq-scene/src/edit.rs b/rs/moq-scene/src/edit.rs index 25893a56..1869e1c2 100644 --- a/rs/moq-scene/src/edit.rs +++ b/rs/moq-scene/src/edit.rs @@ -59,19 +59,22 @@ pub enum Change { after: Option, }, /// The item at `index` of `layer`. Adding one inserts it there, in front of - /// the ones before it. + /// the ones before it. Boxed, as an item is far larger than any other + /// change. Item { layer: Layer, index: usize, #[serde(default, skip_serializing_if = "Option::is_none")] - before: Option, + before: Option>, #[serde(default, skip_serializing_if = "Option::is_none")] - after: Option, + after: Option>, }, /// [`Collection::transition`]. Transition { before: Transition, after: Transition }, /// [`Collection::canvas`]. Canvas { before: Canvas, after: Canvas }, + /// [`Collection::portrait`]. + Portrait { before: bool, after: bool }, /// [`Collection::audio_scenes`], whole. AudioScenes { before: Vec, @@ -127,6 +130,10 @@ impl Change { before: after, after: before, }, + Self::Portrait { before, after } => Self::Portrait { + before: after, + after: before, + }, Self::AudioScenes { before, after } => Self::AudioScenes { before: after, after: before, @@ -145,6 +152,7 @@ impl Change { | Self::Item { .. } | Self::Transition { .. } | Self::Canvas { .. } + | Self::Portrait { .. } | Self::AudioScenes { .. } => None, } } @@ -160,6 +168,7 @@ impl Change { Self::Item { before, after, .. } => before == after, Self::Transition { before, after } => before == after, Self::Canvas { before, after } => before == after, + Self::Portrait { before, after } => before == after, Self::AudioScenes { before, after } => before == after, } } @@ -261,6 +270,9 @@ impl Change { (Self::Canvas { before, after }, Self::Canvas { before: b, after: a }) if after == *b => { Self::Canvas { before, after: *a } } + (Self::Portrait { before, after }, Self::Portrait { before: b, after: a }) if after == *b => { + Self::Portrait { before, after: *a } + } (Self::AudioScenes { before, after }, Self::AudioScenes { before: b, after: a }) if after == *b => { Self::AudioScenes { before, @@ -291,6 +303,7 @@ impl Change { Ok(()) } Self::Scene { index, before, after } => { + let (before, after) = (before.as_ref(), after.as_ref()); splice(&mut collection.scenes, *index, before, after).map_err(|err| conflict(format!("scene {err}"))) } Self::Item { @@ -299,6 +312,7 @@ impl Change { before, after, } => { + let (before, after) = (before.as_deref(), after.as_deref()); let spliced = collection.edit_items(layer, |items| splice(items, *index, before, after)); let spliced = spliced.ok_or_else(|| conflict(format!("there is no {layer}")))?; spliced.map_err(|err| conflict(format!("layer {err} in {layer}"))) @@ -317,6 +331,13 @@ impl Change { collection.canvas = *after; Ok(()) } + Self::Portrait { before, after } => { + if collection.portrait != *before { + return Err(conflict("whether the show goes out in portrait changed".into())); + } + collection.portrait = *after; + Ok(()) + } Self::AudioScenes { before, after } => { if collection.audio_scenes != *before { return Err(conflict("the audio scenes changed".into())); @@ -353,8 +374,8 @@ fn replace( fn splice( list: &mut Vec, index: usize, - before: &Option, - after: &Option, + before: Option<&T>, + after: Option<&T>, ) -> Result<(), String> { match (before, after) { (None, Some(after)) if index <= list.len() => list.insert(index, after.clone()), @@ -761,7 +782,7 @@ impl Draft { self.insert_scene(to, scene) } - fn scene_at(&self, index: usize) -> Result { + pub(crate) fn scene_at(&self, index: usize) -> Result { let scene = self.collection.scenes.get(index).cloned(); scene.ok_or_else(|| conflict(format!("there is no scene {index}"))) } @@ -772,7 +793,7 @@ impl Draft { layer: layer.clone(), index, before: None, - after: Some(item), + after: Some(Box::new(item)), }) } @@ -782,7 +803,7 @@ impl Draft { self.change(Change::Item { layer: layer.clone(), index, - before: Some(before.clone()), + before: Some(Box::new(before.clone())), after: None, })?; Ok(before) @@ -794,8 +815,8 @@ impl Draft { self.change(Change::Item { layer: layer.clone(), index, - before: Some(before), - after: Some(item), + before: Some(Box::new(before)), + after: Some(Box::new(item)), }) } diff --git a/rs/moq-scene/src/format.rs b/rs/moq-scene/src/format.rs new file mode 100644 index 00000000..2bb4d454 --- /dev/null +++ b/rs/moq-scene/src/format.rs @@ -0,0 +1,1163 @@ +//! Landscape and portrait at once: while a show goes out in +//! [portrait](Collection::portrait), each scene of the program's size has a +//! portrait arrangement of the same layers, drawn on the canvas turned on its +//! side. + +use std::borrow::Cow; + +use serde::{Deserialize, Serialize}; + +use crate::Error; +use crate::collection::{Arrangement, Canvas, Collection, Crop, Item, Layer, Nested, Rect, Scene, Size, Source}; +use crate::edit::{Change, Draft}; + +/// Where the portrait picture goes out, beneath the program's broadcast, so +/// no scene published while the show goes out in portrait takes the name. +pub const PORTRAIT: &str = "portrait"; + +/// How much of the portrait canvas's height the main picture takes when a +/// camera is stacked under it. +const MAIN_SHARE: f32 = 0.6; + +/// How much of the landscape canvas an item covers to count as a backdrop, +/// which covers the portrait canvas too. +const BACKDROP: f32 = 0.9; + +/// One of the two shapes a show goes out in. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash, PartialOrd, Ord, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum Format { + /// Wider than tall: the canvas as it is. + Landscape, + /// Taller than wide: the canvas turned on its side. + Portrait, +} + +impl Format { + /// Both, landscape first. + pub const ALL: [Self; 2] = [Self::Landscape, Self::Portrait]; + + /// The other one. + pub fn other(self) -> Self { + match self { + Self::Landscape => Self::Portrait, + Self::Portrait => Self::Landscape, + } + } + + /// Its name, capitalized, as a label says it. + pub fn name(self) -> &'static str { + match self { + Self::Landscape => "Landscape", + Self::Portrait => "Portrait", + } + } +} + +/// Where an item is drawn in the portrait picture. +#[derive(Clone, Copy, Debug, Default, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct Placement { + /// Where on the portrait canvas, stretched to fill it. + pub rect: Rect, + /// Trimmed from what its source's crop leaves, in place of the item's own + /// crop. + #[serde(default, skip_serializing_if = "Crop::is_none")] + pub crop: Crop, +} + +impl Placement { + /// Where an item placed here on the `landscape` canvas sits on the + /// portrait one without a place of its own there: as far across and down, + /// scaled by as much as the narrower side shrinks, so it keeps its shape + /// and fits, its picture `turn` times as wide for its height there, as + /// [`Collection::turn`] says. + fn fitted(self, landscape: Canvas, turn: f32) -> Self { + let (from, to) = (Rect::fill(&landscape), Rect::fill(&landscape.turned())); + self.moved(&from, &to, turn) + } + + /// Moved from on `from` to on `to` as [`fitted`](Self::fitted) moves it + /// from one canvas to the other: its center as far across and down, and + /// scaled to fit, uncropped, as tall again as a picture `turn` times as + /// wide for its height needs. + fn moved(self, from: &Rect, to: &Rect, turn: f32) -> Self { + let rect = self.rect; + let scale = (to.width / from.width).min(to.height / from.height); + let (width, height) = (rect.width * scale, rect.height * scale / turn); + let center = ( + to.x + (rect.x + rect.width / 2.0 - from.x) / from.width * to.width, + to.y + (rect.y + rect.height / 2.0 - from.y) / from.height * to.height, + ); + Self { + rect: Rect { + x: (center.0 - width / 2.0).round(), + y: (center.1 - height / 2.0).round(), + width: width.round(), + height: height.round(), + }, + crop: self.crop, + } + } +} + +/// How a scene is arranged in the portrait picture. +#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct Portrait { + /// Where its items in a slot go: in a row under the main one unless set, + /// since a column beside it leaves a tall canvas mostly empty. + #[serde(default = "row", skip_serializing_if = "is_row")] + pub arrangement: Arrangement, +} + +impl Default for Portrait { + fn default() -> Self { + Self { + arrangement: Arrangement::Row, + } + } +} + +impl Portrait { + pub(crate) fn is_default(&self) -> bool { + *self == Self::default() + } +} + +fn row() -> Arrangement { + Arrangement::Row +} + +fn is_row(arrangement: &Arrangement) -> bool { + *arrangement == Arrangement::Row +} + +impl Canvas { + /// The canvas turned on its side: its width and height swapped. + pub fn turned(self) -> Self { + Self { + width: self.height, + height: self.width, + framerate: self.framerate, + } + } +} + +impl Item { + /// Whether it is shown in `format`'s picture. + pub fn shown_in(&self, format: Format) -> bool { + self.format.is_none_or(|only| only == format) + } + + /// The item drawn at `placement` in `format`'s picture, as it was in the + /// other. An item in a slot has one crop in both, which this sets. + pub fn placed(self, format: Format, placement: Placement) -> Self { + match format { + Format::Portrait if self.slot.is_none() => Self { + portrait: Some(placement), + ..self + }, + _ => Self { + rect: placement.rect, + crop: placement.crop, + ..self + }, + } + } + + /// Where it is drawn in the landscape picture. + fn landscaped(&self) -> Placement { + Placement { + rect: self.rect, + crop: self.crop, + } + } + + /// Whether it says anything about the portrait picture. + fn formats(&self) -> bool { + self.format.is_some() || self.portrait.is_some() + } +} + +impl Collection { + /// The portrait picture's canvas, while the show goes out in portrait. + pub fn portrait_canvas(&self) -> Option { + self.portrait.then(|| self.canvas.turned()) + } + + /// The canvas `format`'s picture is drawn on: the program's, or turned. + pub fn canvas_of(&self, format: Format) -> Canvas { + match format { + Format::Landscape => self.canvas, + Format::Portrait => self.canvas.turned(), + } + } + + /// Where `item` is drawn in `format`'s picture: its rect and crop, or its + /// portrait place, or failing that where it sits on the landscape picture, + /// fitted, its picture turned as [`turn`](Self::turn) says. + pub fn placement(&self, item: &Item, format: Format) -> Placement { + match format { + Format::Landscape => item.landscaped(), + Format::Portrait => item + .portrait + .unwrap_or_else(|| item.landscaped().fitted(self.canvas, self.turn(item))), + } + } + + /// `item` as `format`'s picture draws it at rest: at its place there, and + /// hidden unless shown in it. An item in a slot stays where the + /// arrangement puts it. + pub fn portrayed(&self, item: &Item, format: Format) -> Item { + let mut portrayed = item.clone(); + if item.slot.is_none() { + let placement = self.placement(item, format); + portrayed.rect = placement.rect; + portrayed.crop = placement.crop; + } + portrayed.visible &= item.shown_in(format); + portrayed + } + + /// How many times as wide for its height `item`'s picture is in the + /// portrait picture as in the landscape one: narrower, as the canvas + /// turns, for one showing the program or a scene drawn in both, which + /// shows its portrait picture there; as wide for any other. + pub fn turn(&self, item: &Item) -> f32 { + let turns = match self.nested(&item.source) { + Some(Nested::Program) => self.portrait, + Some(Nested::Scene(scene)) => self.dual(&Layer::scene(scene)), + None => false, + }; + match turns { + true => (self.canvas.height as f32 / self.canvas.width as f32).powi(2), + false => 1.0, + } + } + + /// Whether `layer` is drawn in both pictures: the top layer, or a scene of + /// the program's size, while the show goes out in portrait. + pub fn dual(&self, layer: &Layer) -> bool { + self.portrait + && match &layer.scene { + None => true, + Some(scene) => self.scene(scene).is_some_and(|scene| scene.size.is_none()), + } + } + + /// The collection as `format`'s picture draws it, while the show goes out + /// in portrait: in portrait, on the canvas turned, each scene of the + /// program's size arranged as its portrait arrangement says and each item + /// at its portrait place; in either, an item shown only in the other is + /// hidden. A scene of its own size is drawn as it is. Itself otherwise. + pub fn formatted(&self, format: Format) -> Cow<'_, Self> { + if !self.portrait { + return Cow::Borrowed(self); + } + if format == Format::Landscape { + let hides = self + .layers() + .any(|(_, items)| items.iter().any(|item| !item.shown_in(format))); + if !hides { + return Cow::Borrowed(self); + } + } + let mut view = self.clone(); + view.canvas = self.canvas_of(format); + for (scene, at) in view.scenes.iter_mut().zip(&self.scenes) { + if scene.size.is_some() { + continue; + } + if format == Format::Portrait { + scene.arrangement = scene.portrait.arrangement; + } + for (item, at) in scene.items.iter_mut().zip(&at.items) { + *item = self.portrayed(at, format); + } + } + for (item, at) in view.top.iter_mut().zip(&self.top) { + *item = self.portrayed(at, format); + } + Cow::Owned(view) + } + + /// The picture `layer` is edited in when `format` is asked for: `format` + /// while the layer is drawn in both, landscape otherwise. + pub fn format_of(&self, layer: &Layer, format: Format) -> Format { + match self.dual(layer) { + true => format, + false => Format::Landscape, + } + } + + /// Check the portrait picture's rules: only a landscape canvas turns, and + /// only a scene of the program's size, or the top layer, says anything + /// about it, an item in a slot or shown only in landscape taking no place + /// there; nor does a scene publish where it goes out. + pub(crate) fn validate_formats(&self) -> Result<(), Error> { + if self.portrait && self.canvas.width <= self.canvas.height { + return Err(Error::Invalid(format!( + "the show goes out in portrait too, which turns a landscape canvas, not {}x{}", + self.canvas.width, self.canvas.height + ))); + } + for scene in &self.scenes { + let why = match (self.portrait, scene.size) { + (false, _) => "the show does not go out in portrait", + (true, Some(_)) => "it has a size of its own, so it is drawn in one format", + (true, None) => "", + }; + if why.is_empty() { + continue; + } + if !scene.portrait.is_default() { + return Err(Error::Invalid(format!( + "scene `{}` has a portrait arrangement, but {why}", + scene.name + ))); + } + if let Some(item) = scene.items.iter().find(|item| item.formats()) { + return Err(Error::Invalid(format!( + "`{}` in scene `{}` says where it shows in portrait, but {why}", + item.source, scene.name + ))); + } + } + if !self.portrait + && let Some(item) = self.top.iter().find(|item| item.formats()) + { + return Err(Error::Invalid(format!( + "always-on-top `{}` says where it shows in portrait, but the show does not go out in portrait", + item.source + ))); + } + for (layer, items) in self.layers() { + for item in items { + let Some(placement) = &item.portrait else { continue }; + let on = format!("`{}` in {layer}", item.source); + if item.slot.is_some() { + return Err(Error::Invalid(format!( + "{on} is in a slot, which places it in portrait too, so it takes no portrait place" + ))); + } + if item.format == Some(Format::Landscape) { + return Err(Error::Invalid(format!( + "{on} is shown only in landscape, so it takes no portrait place" + ))); + } + placement.rect.validate(&layer, &item.source)?; + placement + .crop + .validate() + .map_err(|err| Error::Invalid(format!("{on} in portrait: {err}")))?; + } + } + if self.portrait + && let Some(scene) = self + .scenes + .iter() + .find(|scene| scene.published && scene.name == PORTRAIT) + { + return Err(Error::Invalid(format!( + "scene `{}` is published where the portrait picture goes out; rename it to publish it", + scene.name + ))); + } + Ok(()) + } + + /// The items of a layer as a show turning to portrait first places them: + /// the main picture, the largest covering a quarter of the canvas or more, + /// cropped to the top of the portrait canvas, or all of it without a + /// camera, and the largest camera stacked under it, keeping its shape if + /// it has an outline; a backdrop covering the landscape canvas covers the + /// portrait one, cropped, or, a page or overlay, which is mostly + /// see-through, is fitted whole, never the main picture. What sits on the + /// main picture keeps its place on it. The rest sit where they are, fitted, until moved there. A layer + /// showing a scene drawn in both, or the program, shows its portrait + /// picture, shaped for the portrait canvas: covering the landscape canvas, + /// it covers the portrait one whole, rather than being the main picture. + fn turned_on(&self, items: &[Item]) -> Vec { + let landscape = self.canvas; + let portrait = landscape.turned(); + let whole = Rect::fill(&portrait); + let canvas = landscape.width as f32 * landscape.height as f32; + let area = |item: &Item| covered(&item.rect, &landscape) / canvas; + let free: Vec = (0..items.len()) + .filter(|&at| { + let item = &items[at]; + item.slot.is_none() && item.visible && item.shown_in(Format::Portrait) && area(item) > 0.0 + }) + .collect(); + let largest = |among: &mut dyn Iterator| { + // The frontmost of two as large, which is the one seen. + among.max_by(|a, b| area(&items[*a]).total_cmp(&area(&items[*b])).then(a.cmp(b))) + }; + let source = |at: usize| self.sources.get(self.capture(&items[at].source)); + // A page or overlay is mostly see-through, so it is fitted whole + // rather than cropped. + let see_through = |at: usize| matches!(source(at), Some(Source::Browser { .. } | Source::Overlay { .. })); + let turned = |at: usize| self.turn(&items[at]) != 1.0; + // One covering the landscape canvas that is see-through, or shows its + // own portrait picture, covers or fits the portrait canvas whole + // rather than being cropped to the main picture. + let whole_page = |at: usize| area(&items[at]) >= BACKDROP && (turned(at) || see_through(at)); + let main = + largest(&mut free.iter().copied().filter(|&at| !whole_page(at))).filter(|&at| area(&items[at]) >= 0.25); + let camera = largest(&mut free.iter().copied().filter(|&at| { + Some(at) != main && matches!(source(at), Some(Source::Camera { .. })) && area(&items[at]) < BACKDROP + })); + + let mut placed = items.to_vec(); + if let Some(main) = main { + let top = match camera { + Some(_) => Rect { + height: (portrait.height as f32 * MAIN_SHARE).round(), + ..whole + }, + None => whole, + }; + placed[main].portrait = Some(self.covering(&items[main], top)); + // What sits on the main picture, such as a lower third, keeps + // its place on it, rather than landing on the camera. + let on = items[main].rect; + for &at in free.iter().filter(|&&at| Some(at) != Some(main) && Some(at) != camera) { + let item = &items[at]; + let center = ( + item.rect.x + item.rect.width / 2.0, + item.rect.y + item.rect.height / 2.0, + ); + let inside = + (on.x..on.x + on.width).contains(¢er.0) && (on.y..on.y + on.height).contains(¢er.1); + if camera.is_some() && inside && area(item) < BACKDROP { + placed[at].portrait = Some(item.landscaped().moved(&on, &top, self.turn(item))); + } + } + if let Some(camera) = camera { + let under = Rect { + y: top.height, + height: whole.height - top.height, + ..whole + }; + let item = &items[camera]; + placed[camera].portrait = Some(match item.shape.is_square() { + true => self.covering(item, under), + false => self.fitting(item, under), + }); + } + } + for &at in &free { + let item = &items[at]; + if Some(at) != main && Some(at) != camera && area(item) >= BACKDROP && !see_through(at) { + placed[at].portrait = Some(self.covering(item, whole)); + } + } + placed + } +} + +impl Scene { + /// The scene at `size`, its own, or the program's for `None`. A scene of + /// its own size is drawn in one picture, so it keeps no portrait + /// arrangement or place, and its layers shown only in portrait are + /// removed, as turning portrait off removes them. + pub fn sized(mut self, size: Option) -> Self { + if size.is_some() { + self.single(); + } + Self { size, ..self } + } + + /// Forget the portrait picture: its arrangement, each layer's place + /// there, and the layers shown only there. + fn single(&mut self) { + self.portrait = Portrait::default(); + self.items.retain(|item| item.shown_in(Format::Landscape)); + for item in &mut self.items { + item.format = None; + item.portrait = None; + } + } +} + +impl Draft { + /// Give the scene at `index` a size of its own, or the program's with + /// `None`, as [`Scene::sized`] does. While the show goes out in both, a + /// layer elsewhere showing it shows its portrait picture as long as it + /// follows the program's size, and its own size otherwise, so one placed + /// in the portrait picture, whose picture changes shape there, is fitted + /// where it was placed, as wide for its height as the picture became. + pub fn set_size(&mut self, index: usize, size: Option) -> Result<(), Error> { + let before = self.collection().clone(); + let scene = self.scene_at(index)?; + let name = scene.name.clone(); + self.set_scene(index, scene.sized(size))?; + let after = self.collection().clone(); + let (Some(from), Some(to)) = (before.portrait_shape_of(&name), after.portrait_shape_of(&name)) else { + return Ok(()); + }; + if from == to { + return Ok(()); + } + for (layer, items) in after.layers() { + for (index, item) in items.iter().enumerate() { + let Some(placement) = item.portrait else { continue }; + // A place shrunk to nothing has no shape to keep. + if placement.rect.width <= 0.0 || placement.rect.height <= 0.0 { + continue; + } + if after.nested(&item.source) != Some(Nested::Scene(&name)) { + continue; + } + // Cropped as it was, the picture shown there changes shape as + // the whole does. + let shape = placement.rect.width / placement.rect.height * to / from; + let placement = Placement { + rect: fit(shape, placement.rect), + crop: placement.crop, + }; + let item = Item { + portrait: Some(placement), + ..item.clone() + }; + self.set_item(&layer, index, item)?; + } + } + Ok(()) + } + + /// Send the show out in portrait too, or no longer. Turned on, each + /// layer's portrait arrangement starts from its landscape one: the main + /// picture cropped to the top, the camera stacked under it, and slots in + /// a row unless they were in a grid or inset. A scene given the program's + /// size as its own follows the program's instead, so it has a portrait + /// picture and can stay on program. Turned off, the layers shown only in + /// portrait are removed and every portrait place dropped. + pub fn set_portrait(&mut self, on: bool) -> Result<(), Error> { + let before = self.collection().portrait; + if before == on { + return Ok(()); + } + let collection = self.collection().clone(); + let program = (collection.canvas.width, collection.canvas.height); + let pinned = |scene: &Scene| on && scene.size.is_some_and(|size| (size.width, size.height) == program); + // The show as it goes out in both, so a layer showing a scene that + // will be drawn in both is placed for its portrait picture. + let mut turning = collection.clone(); + turning.portrait = true; + for scene in turning.scenes.iter_mut().filter(|scene| pinned(scene)) { + scene.size = None; + } + for (index, scene) in collection.scenes.iter().enumerate() { + if scene.size.is_some() && !pinned(scene) { + continue; + } + let mut after = Scene { + size: None, + ..scene.clone() + }; + match on { + true => { + after.items = turning.turned_on(&scene.items); + if matches!(scene.arrangement, Arrangement::Grid | Arrangement::Inset) { + after.portrait.arrangement = scene.arrangement; + } + } + false => after.single(), + } + if after != *scene { + self.set_scene(index, after)?; + } + } + let top = Layer::top(); + match on { + true => { + for (index, item) in turning.turned_on(&collection.top).into_iter().enumerate() { + if item != collection.top[index] { + self.set_item(&top, index, item)?; + } + } + } + false => { + for (index, item) in collection.top.iter().enumerate().rev() { + if !item.shown_in(Format::Landscape) { + self.remove_item(&top, index)?; + } else if item.formats() { + let item = Item { + format: None, + portrait: None, + ..item.clone() + }; + self.set_item(&top, index, item)?; + } + } + } + } + self.change(Change::Portrait { before, after: on }) + } +} + +/// How much of `canvas` `rect` covers, in square pixels. +fn covered(rect: &Rect, canvas: &Canvas) -> f32 { + let width = (rect.x + rect.width).min(canvas.width as f32) - rect.x.max(0.0); + let height = (rect.y + rect.height).min(canvas.height as f32) - rect.y.max(0.0); + width.max(0.0) * height.max(0.0) +} + +impl Collection { + /// The shape `item`'s picture shows in the portrait picture, as its rect + /// shows it in the landscape one, turned as [`turn`](Self::turn) says. + fn portrait_shape(&self, item: &Item) -> f32 { + item.rect.width / item.rect.height * self.turn(item) + } + + /// `item` covering `rect` in the portrait picture, its picture cropped + /// evenly to `rect`'s shape on top of its own crop, so it shows + /// unstretched as it did. + fn covering(&self, item: &Item, rect: Rect) -> Placement { + let (shown, wanted) = (self.portrait_shape(item), rect.width / rect.height); + // A shape turned to the canvas's own, in floating point, trims nothing. + let trim = match (shown / wanted - 1.0).abs() < 1e-4 { + true => Crop::default(), + false => Crop::covering(shown, wanted), + }; + Placement { + rect, + crop: item.crop.then(&trim), + } + } + + /// `item` as large as fits in `rect` in the portrait picture, keeping its + /// shape, centered. + fn fitting(&self, item: &Item, rect: Rect) -> Placement { + Placement { + rect: fit(self.portrait_shape(item), rect), + crop: item.crop, + } + } + + /// How many times as wide for its height the portrait picture draws the + /// scene called `scene`, whole: as the turned canvas while it is drawn in + /// both, as its own size otherwise; `None` while the show does not go out + /// in portrait, or the scene is gone. + fn portrait_shape_of(&self, scene: &str) -> Option { + let canvas = match self.dual(&Layer::scene(scene)) { + true => self.portrait_canvas()?, + false => self.portrait.then(|| self.canvas(Some(scene))).flatten()?, + }; + Some(canvas.width as f32 / canvas.height as f32) + } +} + +/// As large as a picture `shape` times as wide for its height fits in +/// `rect`, centered. +fn fit(shape: f32, rect: Rect) -> Rect { + let width = rect.width.min(rect.height * shape); + let height = width / shape; + Rect { + x: (rect.x + (rect.width - width) / 2.0).round(), + y: (rect.y + (rect.height - height) / 2.0).round(), + width: width.round(), + height: height.round(), + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::collection::{Rgba, Scene, Shape, Size, Slot}; + + fn color() -> Source { + Source::Color { + color: Rgba([255, 0, 0, 255]), + } + } + + /// A 1920x1080 show: a game filling the canvas, a camera in its corner, + /// and a logo, in scene "Game"; and a slotted scene "Guests". + fn show() -> Collection { + let mut collection = Collection::new(); + collection.sources.insert("game".into(), color()); + collection.sources.insert("camera".into(), Source::Camera { id: None }); + collection.sources.insert("logo".into(), color()); + let fill = Rect::fill(&collection.canvas); + let corner = Rect { + x: 1440.0, + y: 720.0, + width: 480.0, + height: 270.0, + }; + let logo = Rect { + x: 1600.0, + y: 40.0, + width: 280.0, + height: 120.0, + }; + let slotted = |slot| Item { + slot: Some(slot), + ..Item::new("camera", Rect::default()) + }; + collection.scenes = vec![ + Scene { + items: vec![ + Item::new("game", fill), + Item::new("camera", corner), + Item::new("logo", logo), + ], + ..Scene::new("Game") + }, + Scene { + items: vec![slotted(Slot::Main), slotted(Slot::Tile)], + ..Scene::new("Guests") + }, + ]; + collection.validate().unwrap(); + collection + } + + fn turned(collection: &Collection) -> Collection { + let mut draft = collection.draft("Both"); + draft.set_portrait(true).unwrap(); + let edit = draft.finish().unwrap(); + let mut turned = collection.clone(); + turned.apply(&edit).unwrap(); + turned + } + + #[test] + fn turning_on_crops_the_main_picture_over_the_camera() { + let turned = turned(&show()); + assert!(turned.portrait); + let items = &turned.scenes[0].items; + let game = items[0].portrait.unwrap(); + assert_eq!( + game.rect, + Rect { + x: 0.0, + y: 0.0, + width: 1080.0, + height: 1152.0 + } + ); + // The 16:9 game cropped evenly to the 1080x1152 band, unstretched. + let kept = 1.0 - game.crop.left - game.crop.right; + assert!((1920.0 * kept / 1080.0 - 1080.0 / 1152.0).abs() < 1e-3, "{game:?}"); + assert!((game.crop.left - game.crop.right).abs() < 1e-6); + let camera = items[1].portrait.unwrap(); + assert_eq!( + camera.rect, + Rect { + x: 0.0, + y: 1152.0, + width: 1080.0, + height: 768.0 + } + ); + // The logo on the game keeps its place on it, above the camera. + assert_eq!( + items[2].portrait.unwrap().rect, + Rect { + x: 900.0, + y: 73.0, + width: 158.0, + height: 68.0 + } + ); + turned.validate().unwrap(); + } + + /// A backdrop as large as the game, behind it, is not the main picture: + /// the game in front is, and the backdrop covers the portrait canvas. + /// A layer showing a scene drawn in both shows its portrait picture, + /// 9:16, in the portrait one: covering the landscape canvas, such as a + /// scene of overlays, it covers the portrait canvas whole rather than + /// being the main picture; a smaller one is shaped for it wherever it + /// goes, set or fitted. A scene of its own size shows one picture. + #[test] + fn a_scene_source_is_placed_for_its_portrait_picture() { + let mut show = show(); + show.sources + .insert("overlays".into(), Source::Scene { scene: "Guests".into() }); + show.sources + .insert("wide".into(), Source::Scene { scene: "Wide".into() }); + show.scenes.push(Scene { + size: Some(Size { + width: 1280, + height: 720, + }), + ..Scene::new("Wide") + }); + let fill = Rect::fill(&show.canvas); + let pip = Rect { + x: 0.0, + y: 0.0, + width: 960.0, + height: 540.0, + }; + let items = &mut show.scenes[0].items; + items.push(Item::new("overlays", fill)); + items.push(Item::new("wide", pip)); + show.validate().unwrap(); + let turned = turned(&show); + let items = &turned.scenes[0].items; + let whole = Rect::fill(&show.canvas.turned()); + assert_eq!( + items[0].portrait.unwrap().rect.height, + 1152.0, + "the game is the main picture" + ); + let overlays = items[3].portrait.unwrap(); + assert_eq!((overlays.rect, overlays.crop), (whole, Crop::default())); + // On the game, the scene of its own size keeps its 16:9 shape. + let wide = items[4].portrait.unwrap().rect; + assert!((wide.width / wide.height - 16.0 / 9.0).abs() < 0.01, "{wide:?}"); + + // Without a place of its own, a scene drawn in both is fitted turned: + // a quarter of the landscape canvas is a quarter of the portrait one. + let quarter = Item::new("overlays", pip); + let fitted = turned.placement(&quarter, Format::Portrait).rect; + assert_eq!( + fitted, + Rect { + x: 0.0, + y: 0.0, + width: 540.0, + height: 960.0 + } + ); + let program = Item::new("program", pip); + let mut with_program = turned.clone(); + with_program.sources.insert("program".into(), Source::Program); + assert_eq!(with_program.placement(&program, Format::Portrait).rect, fitted); + assert_eq!(turned.turn(&Item::new("wide", pip)), 1.0); + assert_eq!(turned.turn(&Item::new("camera", pip)), 1.0); + } + + #[test] + fn the_frontmost_of_two_as_large_is_the_main_picture() { + let mut show = show(); + let fill = show.scenes[0].items[0].rect; + show.scenes[0].items.insert(0, Item::new("logo", fill)); + let turned = turned(&show); + let items = &turned.scenes[0].items; + assert_eq!(items[0].portrait.unwrap().rect, Rect::fill(&show.canvas.turned())); + assert_eq!(items[1].portrait.unwrap().rect.height, 1152.0); + } + + #[test] + fn a_round_camera_keeps_its_shape() { + let mut show = show(); + show.scenes[0].items[1].rect = Rect { + x: 1500.0, + y: 700.0, + width: 300.0, + height: 300.0, + }; + show.scenes[0].items[1].shape = Shape { + corners: [0.5; 4], + ..Shape::default() + }; + let turned = turned(&show); + let camera = turned.scenes[0].items[1].portrait.unwrap(); + assert_eq!(camera.rect.width, camera.rect.height); + assert_eq!(camera.rect.width, 768.0); + assert_eq!(camera.rect.y, 1152.0); + assert_eq!(camera.rect.x, 156.0); + } + + #[test] + fn without_a_camera_the_main_picture_covers_the_canvas() { + let mut show = show(); + show.scenes[0].items.remove(1); + let turned = turned(&show); + let game = turned.scenes[0].items[0].portrait.unwrap(); + assert_eq!(game.rect, Rect::fill(&show.canvas.turned())); + } + + #[test] + fn slots_go_in_a_row_unless_gridded() { + let mut show = show(); + show.scenes[1].arrangement = Arrangement::Grid; + let mut copy = show.scenes[1].clone(); + copy.name = "Row".into(); + copy.arrangement = Arrangement::Column; + show.scenes.push(copy); + let turned = turned(&show); + assert_eq!(turned.scenes[1].portrait.arrangement, Arrangement::Grid); + assert_eq!(turned.scenes[2].portrait.arrangement, Arrangement::Row); + assert!(turned.scenes[1].items.iter().all(|item| item.portrait.is_none())); + } + + #[test] + fn a_scene_of_its_own_size_is_left_alone() { + let mut show = show(); + show.scenes[0].size = Some(Size { + width: 1080, + height: 1920, + }); + let turned = turned(&show); + assert_eq!(turned.scenes[0], show.scenes[0]); + } + + /// A scene of its own size has no portrait picture, so it does not go on + /// program while the show goes out in both, even at the program's size; + /// one given the program's size before turning on follows it instead. + #[test] + fn only_a_scene_following_the_program_goes_on_in_both() { + let mut show = show(); + let program = Size { + width: 1920, + height: 1080, + }; + show.scenes[0].size = Some(program); + show.takes("Game").unwrap(); + let mut turned = turned(&show); + assert_eq!( + turned.scenes[0].size, None, + "pinned at the program's size, so it follows it" + ); + assert!(turned.scenes[0].items[0].portrait.is_some()); + turned.takes("Game").unwrap(); + + turned.scenes[0] = turned.scenes[0].clone().sized(Some(program)); + turned.validate().unwrap(); + let why = turned.refuses("Game").unwrap(); + assert!(why.contains("size of its own"), "{why}"); + } + + #[test] + fn the_portrait_view_places_each_item_there() { + let mut turned = turned(&show()); + turned.scenes[0].items[2].format = Some(Format::Landscape); + let view = turned.formatted(Format::Portrait); + assert_eq!(view.canvas, turned.canvas.turned()); + let items = &view.scenes[0].items; + assert_eq!(items[0].rect, turned.scenes[0].items[0].portrait.unwrap().rect); + assert!(!items[2].visible, "shown only in landscape"); + assert_eq!(view.scenes[1].arrangement, Arrangement::Row); + // The landscape view hides nothing shown in landscape, so it is the + // collection itself. + assert!(matches!(turned.formatted(Format::Landscape), Cow::Borrowed(_))); + turned.scenes[0].items[2].format = Some(Format::Portrait); + let view = turned.formatted(Format::Landscape); + assert!(!view.scenes[0].items[2].visible); + assert!(view.scenes[0].items[0].visible); + } + + #[test] + fn fitted_keeps_its_place_and_shape() { + let landscape = Canvas::default(); + let logo = Rect { + x: 1600.0, + y: 40.0, + width: 280.0, + height: 120.0, + }; + let fitted = Placement { + rect: logo, + crop: Crop::default(), + } + .fitted(landscape, 1.0); + // 9/16 the size, its center as far across and down. + assert_eq!(fitted.rect.width, 158.0); + assert_eq!(fitted.rect.height, 68.0); + let center = fitted.rect.x + fitted.rect.width / 2.0; + assert!((center / 1080.0 - 1740.0 / 1920.0).abs() < 1e-3); + let center = fitted.rect.y + fitted.rect.height / 2.0; + assert!((center / 1920.0 - 100.0 / 1080.0).abs() < 1e-3); + } + + #[test] + fn turning_off_drops_the_portrait_picture() { + let mut turned = turned(&show()); + turned.scenes[0].items.push(Item { + format: Some(Format::Portrait), + ..Item::new("logo", Rect::fill(&turned.canvas)) + }); + turned.top.push(Item { + format: Some(Format::Portrait), + ..Item::new("logo", Rect::fill(&turned.canvas)) + }); + turned.top.push(Item { + format: Some(Format::Landscape), + ..Item::new("logo", Rect::fill(&turned.canvas)) + }); + turned.validate().unwrap(); + let mut draft = turned.draft("Landscape"); + draft.set_portrait(false).unwrap(); + let edit = draft.finish().unwrap(); + let mut off = turned.clone(); + off.apply(&edit).unwrap(); + assert_eq!(off.scenes, show().scenes); + assert_eq!(off.top.len(), 1); + assert_eq!(off.top[0].format, None); + // Undone whole. + off.apply(&edit.inverse()).unwrap(); + assert_eq!(off, turned); + } + + #[test] + fn refuses_what_the_portrait_picture_cannot_draw() { + let refused = |edit: &dyn Fn(&mut Collection)| { + let mut collection = turned(&show()); + edit(&mut collection); + collection.validate().unwrap_err().to_string() + }; + assert!(refused(&|c| c.canvas = c.canvas.turned()).contains("turns a landscape canvas")); + assert!(refused(&|c| c.scenes[1].items[0].portrait = Some(Placement::default())).contains("in a slot")); + assert!(refused(&|c| c.scenes[0].items[0].format = Some(Format::Landscape)).contains("only in landscape")); + assert!( + refused(&|c| { + c.portrait = false; + }) + .contains("does not go out in portrait") + ); + assert!( + refused(&|c| { + c.scenes[0].name = PORTRAIT.into(); + c.scenes[0].published = true; + }) + .contains("rename it") + ); + } + + /// A scene given its own size is drawn in one picture, so it forgets the + /// portrait one in the same edit, which validates. + #[test] + fn a_scene_given_its_own_size_forgets_the_portrait_picture() { + let mut turned = turned(&show()); + turned.scenes[0].items[2].format = Some(Format::Portrait); + turned.validate().unwrap(); + let size = Size { + width: 1080, + height: 1920, + }; + let sized = turned.scenes[0].clone().sized(Some(size)); + assert_eq!(sized.size, Some(size)); + assert_eq!(sized.items.len(), 2, "the portrait-only logo removed"); + assert!(sized.items.iter().all(|item| !item.formats())); + turned.scenes[0] = sized; + turned.validate().unwrap(); + } + + /// While the show goes out in both, a scene of overlays given a size of + /// its own shows one 16:9 picture, so a layer covering the portrait + /// canvas with its portrait picture is fitted there instead; given the + /// program's size back, its 9:16 picture is fitted in the box it was + /// placed in. Each is one edit with the resize. + #[test] + fn resizing_a_scene_fits_the_layers_showing_it_in_portrait() { + let mut show = show(); + show.sources + .insert("overlays".into(), Source::Scene { scene: "Guests".into() }); + show.scenes[0] + .items + .push(Item::new("overlays", Rect::fill(&show.canvas))); + let turned = turned(&show); + let whole = Rect::fill(&show.canvas.turned()); + assert_eq!(turned.scenes[0].items[3].portrait.unwrap().rect, whole); + let resize = |collection: &Collection, size: Option| { + let mut draft = collection.draft("Size"); + draft.set_size(1, size).unwrap(); + let edit = draft.finish().unwrap(); + let mut resized = collection.clone(); + resized.apply(&edit).unwrap(); + resized.validate().unwrap(); + resized + }; + let own = resize( + &turned, + Some(Size { + width: 1280, + height: 720, + }), + ); + let banner = Rect { + x: 0.0, + y: 656.0, + width: 1080.0, + height: 608.0, + }; + let overlays = own.scenes[0].items[3].portrait.unwrap(); + assert_eq!(overlays.rect, banner, "its 16:9 picture fitted unstretched"); + assert_eq!( + own.scenes[0].items[0], turned.scenes[0].items[0], + "the game is left alone" + ); + + // Given a 9:16 size of its own, its picture keeps its shape there. + let tall = resize( + &turned, + Some(Size { + width: 1080, + height: 1920, + }), + ); + assert_eq!(tall.scenes[0].items[3].portrait.unwrap().rect, whole); + + let back = resize(&own, None); + assert_eq!( + back.scenes[0].items[3].portrait.unwrap().rect, + Rect { + x: 369.0, + y: 656.0, + width: 342.0, + height: 608.0 + }, + "its 9:16 picture fitted in the box it was placed in" + ); + + // A place shrunk to nothing has no shape to keep, so it stays as is. + for (width, height) in [(0.0, 608.0), (1080.0, 0.0), (0.0, 0.0)] { + let mut empty = turned.clone(); + let placement = empty.scenes[0].items[3].portrait.as_mut().unwrap(); + placement.rect.width = width; + placement.rect.height = height; + let rect = placement.rect; + let own = resize( + &empty, + Some(Size { + width: 1280, + height: 720, + }), + ); + assert_eq!(own.scenes[0].items[3].portrait.unwrap().rect, rect); + } + } + + /// A web page covering the canvas in front of the game is mostly + /// see-through, so the game stays the main picture and the page is + /// fitted whole rather than cropped to the main picture's band. + #[test] + fn a_page_covering_the_canvas_is_not_the_main_picture() { + let mut show = show(); + let page = Source::Browser { + url: "https://example.com".into(), + width: 1920, + height: 1080, + }; + show.sources.insert("page".into(), page); + show.scenes[0].items.push(Item::new("page", Rect::fill(&show.canvas))); + show.validate().unwrap(); + let turned = turned(&show); + let items = &turned.scenes[0].items; + assert_eq!( + items[0].portrait.unwrap().rect.height, + 1152.0, + "the game is the main picture" + ); + assert_eq!(items[3].portrait, None, "the page is fitted"); + } + + #[test] + fn round_trips_through_the_scene_file() { + let mut turned = turned(&show()); + turned.scenes[0].items[2].format = Some(Format::Landscape); + turned.scenes[0].items[2].portrait = None; + let text = turned.to_toml().unwrap(); + assert!(text.contains("portrait = true"), "{text}"); + assert!(text.contains("format = \"landscape\""), "{text}"); + assert_eq!(Collection::from_toml(&text).unwrap(), turned); + } +} diff --git a/rs/moq-scene/src/layout.rs b/rs/moq-scene/src/layout.rs index 95a0ec7f..0f0d1064 100644 --- a/rs/moq-scene/src/layout.rs +++ b/rs/moq-scene/src/layout.rs @@ -53,8 +53,16 @@ impl Layout { /// The layout of `collection` with the scene called `scene` on program /// since `start` on the media clock, or no scene for `None`. pub fn new(collection: &Collection, scene: Option<&str>, start: Duration) -> Self { + // As the landscape picture places them, which is the one the layout + // goes with: the portrait picture flattens them and has none. let pictures = pictures(collection, scene, true) .into_iter() + .map(|item| Item { + visible: item.visible && item.shown_in(crate::Format::Landscape), + format: None, + portrait: None, + ..item + }) .map(Picture::new) .collect(); let heard = collection.program_audio().map(|audio| collection.heard(audio)); @@ -324,6 +332,27 @@ mod tests { assert_eq!(top.pictures.len(), 1, "only the top layer's"); } + /// In a show going out in both formats, the layout places each picture + /// as the landscape picture does, which it goes with: hidden there when + /// shown in portrait alone, and saying nothing of the portrait one. + #[test] + fn lays_out_the_landscape_picture_of_both() { + let mut collection = collection(); + collection.portrait = true; + let cam = &mut collection.scenes[0].items[1]; + cam.format = Some(crate::Format::Portrait); + cam.portrait = Some(crate::Placement { + rect: rect(4.0), + crop: Default::default(), + }); + collection.validate().unwrap(); + let layout = Layout::new(&collection, Some("live"), Duration::ZERO); + let cam = &layout.pictures[0].item; + assert!(!cam.visible, "shown in portrait alone"); + assert_eq!((cam.format, cam.portrait), (None, None)); + assert_eq!(cam.rect, rect(0.0)); + } + /// Each source's own crop and filters fold into its picture, a clone's /// its own rather than the source's it shows. #[test] diff --git a/rs/moq-scene/src/lib.rs b/rs/moq-scene/src/lib.rs index 997f492f..e73169aa 100644 --- a/rs/moq-scene/src/lib.rs +++ b/rs/moq-scene/src/lib.rs @@ -53,6 +53,7 @@ mod dark; mod edit; mod effect; pub mod file; +mod format; mod glide; mod grant; pub mod layout; @@ -74,6 +75,7 @@ pub use collection::{ }; pub use compositor::{Compositor, Detail, FORMAT, Output, Thumbnail}; pub use edit::{Change, Draft, Edit, History}; +pub use format::{Format, PORTRAIT, Placement, Portrait}; pub use grant::Grants; pub use scale::Scaler; diff --git a/rs/moq-scene/src/source.rs b/rs/moq-scene/src/source.rs index 29d3b57b..61e531fe 100644 --- a/rs/moq-scene/src/source.rs +++ b/rs/moq-scene/src/source.rs @@ -313,6 +313,10 @@ pub struct Sources { feeds: BTreeMap, /// Each audio scene's mix, by its name. mixes: BTreeMap, + /// The program's audio scene with its sources published independently + /// mixed in too, by its name, while the show goes out in portrait and + /// one of them is heard: see [`whole`](Self::whole). + whole: Option<(String, Mix)>, /// The sources on program, as last set. on_program: BTreeSet, /// The picked displays and windows the last sync ran, by name, whose @@ -327,6 +331,7 @@ impl Sources { context, feeds: BTreeMap::new(), mixes: BTreeMap::new(), + whole: None, on_program: BTreeSet::new(), picked: None, } @@ -370,11 +375,11 @@ impl Sources { /// [`Collection::validate`] refuses. pub fn sync(&mut self, collection: &Collection) -> Vec { self.forget_grants(collection); - let mixes = &self.mixes; + let (mixes, whole) = (&self.mixes, &self.whole); self.feeds.retain(|name, feed| { let keep = collection.sources.get(name) == Some(&feed.source); if !keep { - leave(mixes, feed); + leave(mixes, whole, feed); } keep }); @@ -390,10 +395,22 @@ impl Sources { .collect(); for name in orphaned { let feed = self.feeds.remove(&name).expect("listed above"); - leave(&self.mixes, &feed); + leave(&self.mixes, &self.whole, &feed); } // A mix ends once its last handle drops, which may be a reader's. self.mixes.retain(|name, _| collection.audio_scene(name).is_some()); + // The portrait picture carries no layout, so its sound has the + // sources published independently mixed in, when any is heard. + let whole = collection + .portrait_canvas() + .and(collection.program_audio()) + .filter(|audio| { + let heard = collection.heard(audio); + heard.iter().any(|name| collection.published.contains(*name)) + }); + if self.whole.as_ref().map(|(name, _)| name.as_str()) != whole { + self.whole = whole.map(|name| (name.to_string(), self.context.bus.mix())); + } for scene in &collection.audio_scenes { if !self.mixes.contains_key(&scene.name) { self.mixes.insert(scene.name.clone(), self.context.bus.mix()); @@ -458,6 +475,17 @@ impl Sources { mix.unroute(strip); } } + if let Some((scene, mix)) = &self.whole { + let routed = heard + .get(scene.as_str()) + .is_some_and(|heard| heard.contains(name.as_str())); + if mix.routes(strip) != routed { + match routed { + true => mix.route(strip), + false => mix.unroute(strip), + } + } + } } } drivers @@ -498,7 +526,7 @@ impl Sources { /// every mix at once, as an edited source does. pub fn restart(&mut self, name: &str) { if let Some(feed) = self.feeds.remove(name) { - leave(&self.mixes, &feed); + leave(&self.mixes, &self.whole, &feed); } } @@ -517,7 +545,7 @@ impl Sources { /// and the next [`sync`](Self::sync) starts the sources again. pub fn stop(&mut self) { for feed in std::mem::take(&mut self.feeds).into_values() { - leave(&self.mixes, &feed); + leave(&self.mixes, &self.whole, &feed); } } @@ -532,6 +560,17 @@ impl Sources { self.mixes.get(scene) } + /// The sound of a picture that carries no layer layout, such as the + /// portrait one: the program's audio scene, as [`mix`](Self::mix) has + /// it, with the sources published independently that it hears mixed in, + /// which reach the landscape picture's viewers by their own tracks. + pub fn whole(&self, collection: &Collection) -> Option<&Mix> { + match &self.whole { + Some((_, mix)) => Some(mix), + None => self.mix(collection.program_audio()?), + } + } + /// Whether the audio scene called `scene` plays the source called `name`: /// it has sound, is not muted, and the audio scene reaches it. pub fn plays(&self, scene: &str, name: &str) -> bool { @@ -548,11 +587,11 @@ impl Sources { } } -/// Take a stopping `feed` out of every one of `mixes`, so it leaves nothing -/// behind in them, nor plays over a replacement. -fn leave(mixes: &BTreeMap, feed: &Feed) { +/// Take a stopping `feed` out of every one of `mixes`, and `whole`, so it +/// leaves nothing behind in them, nor plays over a replacement. +fn leave(mixes: &BTreeMap, whole: &Option<(String, Mix)>, feed: &Feed) { let Some(strip) = &feed.strip else { return }; - for mix in mixes.values() { + for mix in mixes.values().chain(whole.iter().map(|(_, mix)| mix)) { mix.unroute(strip); } } @@ -4669,6 +4708,46 @@ mod media_tests { .await; } + /// A sound published independently reaches the landscape picture's + /// viewers by its own track, so the program's mix leaves it out; the + /// portrait picture, which carries no layout, has it mixed in. + #[tokio::test(start_paused = true)] + async fn the_portrait_picture_hears_what_is_published_independently() { + local(async { + let dir = tempfile::tempdir().unwrap(); + let (context, _origin) = context(dir.path()); + let mut sources = Sources::new(context); + let mut collection = media(dir.path(), &[("tone.webm", TONE, true, false)]); + collection.published.insert("tone.webm".into()); + collection.validate().unwrap(); + let drivers = sources.sync(&collection); + start(&sources, drivers).await; + let strip = sources.get("tone.webm").unwrap().strip().unwrap().clone(); + let program = sources.mix(crate::AudioScene::DEFAULT).unwrap(); + assert!(!program.routes(&strip), "by its own track"); + assert!( + std::ptr::eq(sources.whole(&collection).unwrap(), program), + "without portrait, the program's" + ); + + collection.portrait = true; + collection.validate().unwrap(); + assert!(sources.sync(&collection).is_empty()); + let whole = sources.whole(&collection).unwrap(); + assert!(whole.routes(&strip), "mixed into the portrait picture's"); + assert!(!sources.mix(crate::AudioScene::DEFAULT).unwrap().routes(&strip)); + + // Restarted, it leaves that mix too, and comes back in. + sources.restart("tone.webm"); + assert!(!sources.whole(&collection).unwrap().routes(&strip)); + let drivers = sources.sync(&collection); + start(&sources, drivers).await; + let strip = sources.get("tone.webm").unwrap().strip().unwrap().clone(); + assert!(sources.whole(&collection).unwrap().routes(&strip)); + }) + .await; + } + /// A file the studio cannot read fails its source, saying why, and only a /// media source plays from its start. #[tokio::test] diff --git a/rs/moq-studio/shots/dual.toml b/rs/moq-studio/shots/dual.toml new file mode 100644 index 00000000..85e3cac6 --- /dev/null +++ b/rs/moq-studio/shots/dual.toml @@ -0,0 +1,75 @@ +# The show the dual-format views render on: one show going out in landscape +# and portrait at once. Each scene of the program's size has a portrait +# arrangement of the same layers: the game cropped into the top of the +# portrait picture with the camera under it, as turning Both on places them, +# a logo shown only in landscape, and a scene laid out by slots in a row in +# portrait. Still sources only, so each view draws the same picture. +version = 1 +portrait = true + +[canvas] +width = 1920 +height = 1080 +framerate = 30 + +[sources.Main] +kind = "role" +source = "game" + +[sources.Camera] +kind = "role" +source = "host" + +[sources.game] +kind = "color" +color = "#3b1d6e" + +[sources.host] +kind = "image" +path = "person.jpg" + +[sources.Dana] +kind = "clone" +source = "host" + +[sources.backdrop] +kind = "color" +color = "#101828" + +[sources.bar] +kind = "color" +color = "#2563ebe6" + +[sources.logo] +kind = "color" +color = "#f59e0b" + +[video.Dana] +filters = [{ kind = "hue_rotate", degrees = 140.0 }] + +[[scenes]] +name = "Gameplay" +items = [ + { source = "backdrop", rect = { x = 0, y = 0, width = 1920, height = 1080 }, portrait = { rect = { x = 0, y = 0, width = 1080, height = 1920 } } }, + { source = "game", rect = { x = 0, y = 0, width = 1920, height = 1080 }, portrait = { rect = { x = 0, y = 0, width = 1080, height = 1152 } } }, + { source = "host", rect = { x = 1440, y = 600, width = 400, height = 400 }, crop = { left = 0.21875, right = 0.21875 }, portrait = { rect = { x = 0, y = 1152, width = 1080, height = 768 }, crop = { left = 0.1, right = 0.1 } } }, + { source = "bar", rect = { x = 80, y = 860, width = 900, height = 120 }, portrait = { rect = { x = 45, y = 948, width = 506, height = 68 } } }, + { source = "logo", rect = { x = 1760, y = 40, width = 120, height = 120 }, format = "landscape" }, +] + +[[scenes]] +name = "Guests" +portrait = { arrangement = "row" } +items = [ + { source = "backdrop", rect = { x = 0, y = 0, width = 1920, height = 1080 }, portrait = { rect = { x = 0, y = 0, width = 1080, height = 1920 } } }, + { source = "Main", slot = "main" }, + { source = "Camera", slot = "tile" }, + { source = "Dana", slot = "tile" }, +] + +[[scenes]] +name = "Break" +items = [ + { source = "backdrop", rect = { x = 0, y = 0, width = 1920, height = 1080 }, portrait = { rect = { x = 0, y = 0, width = 1080, height = 1920 } } }, + { source = "bar", rect = { x = 460, y = 480, width = 1000, height = 120 } }, +] diff --git a/rs/moq-studio/src/engine.rs b/rs/moq-studio/src/engine.rs index a57000f9..6282b1b0 100644 --- a/rs/moq-studio/src/engine.rs +++ b/rs/moq-studio/src/engine.rs @@ -403,17 +403,29 @@ impl State { /// mask's it reads. fn placed<'a>(&self, scenes: impl IntoIterator>) -> BTreeSet { let collection = &self.collection; - let layers = scenes.into_iter().map(|scene| moq_scene::Layer { - scene: scene.map(str::to_string), - }); - let items = collection.drawn(layers); + let items = scenes.into_iter().flat_map(|scene| self.drawn(scene)); items - .into_iter() .flat_map(|item| std::iter::once(&item.source).chain(masks(collection.filters(item)))) .map(|source| collection.capture(source).to_string()) .collect() } + /// The visible items the scene called `scene`, or the top layer for + /// `None`, draws where viewers see it, through the scenes they draw: in + /// each picture the show goes out in while it is on air, or for the top + /// layer, and in the landscape one alone for a scene drawn beside program. + fn drawn(&self, scene: Option<&str>) -> Vec<&moq_scene::Item> { + let on_air = scene.is_none_or(|scene| self.compositor.on_air().any(|on| on == scene)); + let layer = moq_scene::Layer { + scene: scene.map(str::to_string), + }; + let formats = moq_scene::Format::ALL.into_iter(); + let formats = formats.filter(|format| on_air || *format == moq_scene::Format::Landscape); + formats + .flat_map(|format| self.collection.drawn([layer.clone()], format)) + .collect() + } + /// Put the preview on program at once; see [`Compositor::cut`]. pub fn cut(&mut self) -> anyhow::Result<()> { self.compositor.cut(&self.collection, &self.sources)?; @@ -506,10 +518,7 @@ impl State { let roots = self.compositor.live().chain(published.map(|found| found.name.as_str())); let roots = roots.map(Some).chain([None]); for root in roots { - let layer = moq_scene::Layer { - scene: root.map(str::to_string), - }; - let items = collection.drawn([layer]); + let items = self.drawn(root); if items .iter() .any(|item| collection.nested(&item.source) == Some(moq_scene::Nested::Scene(scene))) @@ -795,11 +804,12 @@ fn reordered(before: &[moq_scene::Item], after: &[moq_scene::Item]) -> bool { } /// Refuse edits, each read as its [`Collection::renames`], that remove a -/// scene `on_air` program draws, so a slip cannot put black on air, that -/// leave one drawing the program in `collection`, which it would draw into -/// itself, or that make one another size than the program, which it would -/// not fill. A reload that leaves one out still drops it, having nothing to -/// show instead. +/// scene `on_air` program draws, so a slip cannot put black on air, or that +/// leave one in `collection` unable to go on program, as +/// [`Collection::refuses`] says: drawing the program into itself, another +/// size than the program, or of its own size while the show goes out in +/// portrait too, with no portrait picture. A reload that leaves one out +/// still drops it, having nothing to show instead. fn keep_on_air<'a>( on_air: impl IntoIterator, renames: &'a [BTreeMap>], @@ -813,15 +823,13 @@ fn keep_on_air<'a>( None => {} } } - anyhow::ensure!( - !collection.draws_program(scene), - "`{scene}` is on air, so it cannot draw the program, which it would draw into itself" - ); - let size = collection.canvas(Some(scene)).map(|canvas| canvas.size()); - anyhow::ensure!( - size.is_none_or(|size| size == collection.canvas.size()), - "`{scene}` is on air, so it stays the program's size; cut to another scene first" - ); + // A reload that leaves it out drops it, having nothing to show. + if collection.scene(scene).is_none() { + continue; + } + if let Some(why) = collection.refuses(scene) { + anyhow::bail!("`{scene}` is on air, so cut to another scene first: {why}"); + } } Ok(()) } @@ -837,7 +845,8 @@ pub struct Stats { /// hold. pub recording: Arc, /// Each published scene's video, by the scene's name, while it is - /// published. + /// published, and the portrait picture's, as [`moq_scene::PORTRAIT`], + /// while the show goes out in both. published: Mutex>>, /// The video encoders open: the program's, each published scene's, the /// recording's, and the sources'. @@ -1899,37 +1908,60 @@ impl Engine { // On air, each published scene's line starts once it is // published, in a broadcast of its own, and ends once it no - // longer is. + // longer is, as does the portrait picture's while the show + // goes out in portrait. let published = published(&state.collection); + let portrait = state.collection.portrait_canvas(); + let extras: Vec<(Extra, Canvas)> = published + .iter() + .map(|(name, canvas)| (Extra::Scene(name.clone()), *canvas)) + .chain(portrait.map(|canvas| (Extra::Portrait, canvas))) + .collect(); if let Some(air) = &mut air { - let gone: Vec = air + // One whose name now carries the other kind, such as a scene + // named as the portrait picture before it went out in both, + // ends too, so its viewers see it end. + let gone: Vec = air .lines .keys() - .filter(|line| !published.iter().any(|(name, _)| name == *line)) + .filter(|line| !extras.iter().any(|(extra, _)| extra == *line)) .cloned() .collect(); - for name in gone { - if let Some(Some((line, broadcast))) = air.lines.remove(&name) { + for extra in gone { + if let Some(Some((line, broadcast))) = air.lines.remove(&extra) { retired.push(line.end(Some(broadcast))); } - self.stats.published.lock().unwrap().remove(&name); + self.stats.published.lock().unwrap().remove(extra.name()); } - for (name, canvas) in &published { - let started = !air.lines.contains_key(name); + for (extra, canvas) in &extras { + let name = extra.name(); + let started = !air.lines.contains_key(extra); let extras = &air.extras; - let line = air.lines.entry(name.clone()).or_insert_with(|| { + let line = air.lines.entry(extra.clone()).or_insert_with(|| { let video = Arc::new(VideoStats::default()); - self.stats.published.lock().unwrap().insert(name.clone(), video.clone()); + self.stats + .published + .lock() + .unwrap() + .insert(name.to_string(), video.clone()); extras .start(name, *canvas, video.clone()) .inspect_err(|err| video.fail(format!("{err:#}"))) .ok() }); - // Its audio follows the audio scene it carries from now on. + // Its audio follows the audio scene it carries from now on: + // the portrait picture's, the program's, with the sources + // published independently in it, as it carries no layout. + let mix = match extra { + Extra::Portrait => state.sources.whole(&state.collection), + Extra::Scene(name) => state + .collection + .published_audio(name) + .and_then(|audio| state.sources.mix(audio)), + }; if (synced || started) && let Some((_, broadcast)) = line - && let Some(audio) = state.collection.published_audio(name) - && let Some(mix) = state.sources.mix(audio) + && let Some(mix) = mix { broadcast.audio.set(mix.clone()); } @@ -1949,7 +1981,8 @@ impl Engine { let cameras = &state.cameras; let lines = air.as_ref().map(|air| &air.lines); let wants = |name: &str| { - matches!(lines.and_then(|lines| lines.get(name)), Some(Some((line, _))) if line.wants()) + let extra = Extra::Scene(name.to_string()); + matches!(lines.and_then(|lines| lines.get(&extra)), Some(Some((line, _))) if line.wants()) || cameras .get(&Some(name.to_string())) .is_some_and(crate::vcam::Camera::wanted) @@ -1991,13 +2024,15 @@ impl Engine { } let program = (compositor.program().clone(), compositor.canvas()); let texture = |scene: &str| compositor.output(scene).and_then(moq_scene::Output::texture).cloned(); + let portrait = compositor.portrait().map(|(texture, canvas)| (texture.clone(), canvas)); Drawn { published: published .into_iter() .map(|(name, canvas)| { let texture = texture(&name); - (name, texture, canvas) + (Extra::Scene(name), texture, canvas) }) + .chain(portrait.map(|(texture, canvas)| (Extra::Portrait, Some(texture), canvas))) .collect(), recorded: match recorded { Some(scene) => (texture(&scene), recorded_canvas), @@ -2020,8 +2055,8 @@ impl Engine { recording.feed(texture.as_ref(), recorded, timestamp); if let Some(air) = &mut air { air.main.feed(Some(&program), canvas, timestamp); - for (name, texture, canvas) in drawn.published { - if let Some(Some((line, _))) = air.lines.get_mut(&name) { + for (extra, texture, canvas) in drawn.published { + if let Some(Some((line, _))) = air.lines.get_mut(&extra) { line.feed(texture.as_ref(), canvas, timestamp); } } @@ -2726,16 +2761,37 @@ impl Line { } } -/// Each published scene's line and broadcast, by name, or `None` once it -/// failed to start, which is not retried until the scene is published again. -type Lines = BTreeMap>; +/// Each published scene's line and broadcast, and the portrait picture's, or +/// `None` once it failed to start, which is not retried until it starts +/// again. +type Lines = BTreeMap>; + +/// A picture going out as a broadcast of its own, beneath the program's. +#[derive(Clone, Debug, PartialEq, Eq, PartialOrd, Ord)] +enum Extra { + /// A published scene, by name. + Scene(String), + /// The portrait picture, while the show goes out in both. + Portrait, +} + +impl Extra { + /// Where it goes out, beneath the program's broadcast. + fn name(&self) -> &str { + match self { + Self::Scene(name) => name, + Self::Portrait => moq_scene::PORTRAIT, + } + } +} /// What one tick drew, for the encoders. struct Drawn { /// The program, at its canvas. program: (wgpu::Texture, Canvas), - /// Each published scene by name, unless not drawn, at its canvas. - published: Vec<(String, Option, Canvas)>, + /// Each published scene, unless not drawn, at its canvas, and the + /// portrait picture, while the show goes out in portrait. + published: Vec<(Extra, Option, Canvas)>, /// What recordings hold, unless not drawn, at its canvas. recorded: (Option, Canvas), } @@ -4578,8 +4634,23 @@ mod tests { drawing.apply(&edit).unwrap(); let renames = [collection.renames(&edit)]; let err = keep_on_air([live.as_str()], &renames, &drawing).unwrap_err(); - assert!(err.to_string().contains("cannot draw the program"), "{err}"); + assert!(err.to_string().contains("draws the program"), "{err}"); keep_on_air(["Renamed"], &renames, &drawing).unwrap(); + + // Nor, while the show goes out in portrait too, one given a size of + // its own, even the program's, which has no portrait picture. + let mut both = collection.clone(); + both.portrait = true; + let canvas = both.canvas; + let on_air = both.scenes.iter_mut().find(|scene| scene.name == live).unwrap(); + on_air.size = Some(moq_scene::Size { + width: canvas.width, + height: canvas.height, + }); + let err = keep_on_air([live.as_str()], &[], &both).unwrap_err(); + assert!(err.to_string().contains("no portrait picture"), "{err}"); + both.portrait = false; + keep_on_air([live.as_str()], &[], &both).unwrap(); } async fn subscribe( @@ -4884,6 +4955,113 @@ mod tests { assert_eq!(engine.stats.sessions.open(), 0); } + /// A show going out in portrait too sends its portrait picture as a + /// broadcast of its own beneath the program's, turned on its side, with + /// the program's audio scene, holding an encode session of its own; + /// turning portrait off ends it, giving the session back. A scene + /// published where it goes out, until one edit unpublishes it and turns + /// portrait on, ends its broadcast, so its viewers see it end rather than + /// the portrait picture take its place. + #[tokio::test(flavor = "multi_thread")] + #[ignore = "needs a Vulkan device and an H.264 encoder; run by just render-sw"] + async fn the_portrait_picture_has_its_own_broadcast() { + let _ = moq_tokio::crypto::install_default(); + let fixture = moq_relay::test_relay().await.expect("test relay"); + let ready = fixture.relay.ready(); + let relay = fixture.relay.cluster().origin.consume(); + tokio::spawn(fixture.relay.run()); + ready.wait().await.expect("relay ready"); + + let mut client = moq_tokio::connect::Config::default(); + client.tls.fingerprint = vec![fixture.fingerprint.clone()]; + let client = client.init(Default::default()).expect("client"); + + let dir = tempfile::tempdir().unwrap(); + let tone = include_bytes!("../../moq-scene/src/test_data/tone.webm"); + std::fs::write(dir.path().join("tone.webm"), tone).unwrap(); + let gpu = Gpu::new(wgpu::Backends::all()).await.expect("a GPU"); + let (autosave, _) = Autosave::open(dir.path().join("show.toml")).unwrap(); + let relays = vec![Relay { + client, + url: fixture.url.clone(), + }]; + let mut loaded = red_canvas(); + let collection = &mut loaded.collection; + // The program's audio scene plays a tone over and over. + let tone = moq_scene::Source::Media { + path: "tone.webm".into(), + looping: true, + restart: false, + }; + collection.sources.insert("tone".into(), tone); + collection.audio_scenes[0].sounds.insert("tone".into()); + let mut published = collection.scenes[0].clone(); + published.name = moq_scene::PORTRAIT.into(); + published.published = true; + collection.scenes.push(published); + let engine = start(gpu, relays, dir.path(), autosave, loaded) + .await + .expect("the studio starts"); + + let path = format!("studio/{}", moq_scene::PORTRAIT); + let (scene, catalogs, catalog) = routed(&relay, &path).await; + let renditions = catalog.video.renditions.iter(); + let mut wide = renditions.filter(|(_, config)| config.coded_width == Some(640)); + let (video, _) = wide.next().expect("the scene at the program's size"); + let track = served(&scene, video).await; + { + // Red fills the landscape picture, and is fitted across the + // middle of the portrait one. + let mut state = engine.state.lock().unwrap(); + let mut draft = state.collection().draft("Both"); + let unpublished = moq_scene::Scene { + published: false, + ..state.collection().scenes[1].clone() + }; + draft.set_scene(1, unpublished).unwrap(); + draft.set_portrait(true).unwrap(); + state.commit(draft, None).unwrap(); + } + ended(catalogs, track).await; + kio::clock::timeout(TIMEOUT, scene.closed()) + .await + .expect("the scene's broadcast ends"); + + let (broadcast, catalogs, catalog) = routed(&relay, &path).await; + assert!(catalog.audio.enabled().next().is_some(), "its audio"); + let (video, config) = catalog + .video + .renditions + .iter() + .find(|(_, config)| config.coded_width == Some(360)) + .expect("the canvas turned"); + assert_eq!(config.coded_height, Some(640)); + let video = video.clone(); + let mut tall = Viewer::watch(&broadcast, &catalog, moq_video::Size::new(360, 640)).await; + tall.sees(RED).await; + // The tone peaks near 0.1 (-20 dBFS). + tall.hears(Duration::ZERO, Duration::ZERO, 0.05).await; + assert_eq!(engine.stats.sessions.open(), 2, "one full size each"); + + let track = served(&broadcast, &video).await; + { + let mut state = engine.state.lock().unwrap(); + let mut draft = state.collection().draft("Landscape"); + draft.set_portrait(false).unwrap(); + state.commit(draft, None).unwrap(); + } + ended(catalogs, track).await; + kio::clock::timeout(TIMEOUT, broadcast.closed()) + .await + .expect("the portrait broadcast ends"); + assert_eq!(engine.stats.sessions.open(), 1, "its session given back"); + let stopping = engine.clone(); + tokio::task::spawn_blocking(move || stopping.stop()).await.unwrap(); + kio::clock::timeout(TIMEOUT, engine.close()) + .await + .expect("the studio closes"); + } + /// What a viewer of one broadcast takes in, frame by frame as it arrives, /// each stamped with its timestamp: the color at the center of each /// picture, and the loudest channel of each sample. diff --git a/rs/moq-studio/src/kit/widgets.rs b/rs/moq-studio/src/kit/widgets.rs index 5320ed9f..1426215b 100644 --- a/rs/moq-studio/src/kit/widgets.rs +++ b/rs/moq-studio/src/kit/widgets.rs @@ -1104,6 +1104,9 @@ pub struct Pad<'a> { width: f32, /// The thumbnail's shape, width over height: the show's. shape: f32, + /// The scene's portrait thumbnail and its shape, drawn at the right of + /// the landscape one, while the show goes out in both. + portrait: Option<(Option, f32)>, pose: Option, } @@ -1120,10 +1123,21 @@ impl<'a> Pad<'a> { dark: false, width: 124.0, shape: 16.0 / 9.0, + portrait: None, pose: None, } } + /// Beside the landscape thumbnail, the portrait one of `shape`, once + /// there is one, for a scene going out in both. The pad's own + /// [`shape`](Self::shape) is then the pair's. + pub fn portrait(mut self, picture: Option, shape: f32) -> Self { + if shape.is_finite() && shape > 0.0 { + self.portrait = Some((picture, shape)); + } + self + } + /// The scene's thumbnail, once there is one, fit to the pad whatever its /// shape. pub fn picture(mut self, picture: Option) -> Self { @@ -1215,9 +1229,18 @@ impl Widget for Pad<'_> { painter.rect(rect, 10, fill, Stroke::new(2.0, border), StrokeKind::Inside); let thumb = Rect::from_min_size(rect.min + vec2(4.0, 4.0), picture); painter.rect_filled(thumb, 7, BG); - if let Some(texture) = self.picture { - let fit = texture.size * (thumb.size() / texture.size).min_elem(); - let at = Rect::from_center_size(thumb.center(), fit); + // The landscape picture, and the portrait one at its right. + let mut pictures = vec![(thumb, self.picture)]; + if let Some((texture, shape)) = self.portrait { + let wide = (thumb.height() * shape).min(thumb.width() / 2.0); + let right = thumb.with_min_x(thumb.right() - wide); + pictures[0].0 = thumb.with_max_x(right.left() - 3.0); + pictures.push((right, texture)); + } + for (room, texture) in pictures { + let Some(texture) = texture else { continue }; + let fit = texture.size * (room.size() / texture.size).min_elem(); + let at = Rect::from_center_size(room.center(), fit); egui::Image::new(texture).corner_radius(7).paint_at(ui, at); } if self.refused.is_some() { diff --git a/rs/moq-studio/src/setup.rs b/rs/moq-studio/src/setup.rs index ec7ba09b..61ba1b75 100644 --- a/rs/moq-studio/src/setup.rs +++ b/rs/moq-studio/src/setup.rs @@ -85,7 +85,20 @@ impl Encoding { /// Whether the encoder keeps up with `canvas` in real time, within [`PACE`]. pub(crate) fn keeps_up(&self, canvas: Canvas) -> bool { - self.frame <= budget(canvas) + self.keeps_up_all(canvas, 1) + } + + /// Whether it keeps up with `encodes` pictures of `canvas` at once, such + /// as a show's landscape and portrait ones: each costs about a frame of + /// the one timed, as a turned picture has as many pixels. + pub(crate) fn keeps_up_all(&self, canvas: Canvas, encodes: u32) -> bool { + self.frame * encodes <= budget(canvas) + } + + /// What `encodes` pictures at this bitrate send, each a broadcast with + /// its own audio. + pub(crate) fn total_of(&self, encodes: u32) -> Rate { + Rate::from_bps(self.total().as_bps() * u64::from(encodes)) } /// How many times its [`budget`] a frame of `canvas` took, scaled to compare @@ -934,6 +947,14 @@ mod tests { assert_eq!(pick(&encoders, None), CANVASES[2], "the default does not keep up"); } + /// The portrait picture is a broadcast of its own, audio included. + #[test] + fn both_formats_send_the_audio_twice() { + let encoding = encoders(DEFAULTS)[1].1.clone().unwrap(); + assert_eq!(encoding.total_of(1), encoding.total()); + assert_eq!(encoding.total_of(2), Rate::from_kbps(2 * (4354 + 128))); + } + #[test] fn proposes_the_closest_to_keeping_up_when_none_does() { let encoders = timed([100, 80, 60, 40, 30]); diff --git a/rs/moq-studio/src/shots.rs b/rs/moq-studio/src/shots.rs index 21b1e9fc..5c3af1ea 100644 --- a/rs/moq-studio/src/shots.rs +++ b/rs/moq-studio/src/shots.rs @@ -111,8 +111,11 @@ enum View { /// A scene, by name, open for editing off air on BUILD. OffAir(&'static str), /// BUILD with the front layer selected, once it kept out of the way: - /// moved off what mattered and landed. - KeepClear, + /// moved off what mattered and landed. If `dual`, with the show going + /// out in both formats. + KeepClear { + dual: bool, + }, /// BUILD with the game's window gone dark selected, and what to do about /// it open from its badge. Dark, @@ -133,11 +136,39 @@ enum View { /// BUILD's Add tab on the drop views' show, with a file dragged or /// dropped on the picture. Drop(Drop), + /// On the dual-format show, `page` with the layer of the scene on + /// program at `index`, front first, selected, BUILD editing `format`'s + /// picture, in Studio mode with the scene `cued` cued, if any, and the + /// scene `off_air` open off air, if any. + Dual { + page: Page, + format: moq_scene::Format, + index: Option, + cued: Option<&'static str>, + off_air: Option<&'static str>, + }, /// The command palette, open by its key over LIVE with a few keys bound, /// with what is typed into it. Palette(&'static str), } +/// The studio on the dual-format views' show, at the mock-ups' size, turned +/// to `page` with the camera's layer selected, editing `format`'s picture. +fn dual(page: Page, format: moq_scene::Format) -> Open { + Open::Studio { + view: View::Dual { + page, + format, + index: Some(2), + cued: None, + off_air: None, + }, + size: SIZE, + show: "dual.toml", + relay: false, + } +} + /// A file over the picture, at the pointer. #[derive(Clone, Copy)] enum Drop { @@ -297,10 +328,14 @@ views! { /// A facecam keeping out of the way of a game: what matters glowing on the /// picture, the places it may go, where it was and the glide to where it /// went, the legend under the picture, and its settings in the inspector. - build_keep_clear: Live, Open::Studio { view: View::KeepClear, size: SIZE, show: "keep.toml", relay: false }; + build_keep_clear: Live, Open::Studio { view: View::KeepClear { dual: false }, size: SIZE, show: "keep.toml", relay: false }; /// The same in the smallest window: the legend's labels shortened to fit /// beside the note, or the note left out. - build_keep_clear_compact: Live, Open::Studio { view: View::KeepClear, size: crate::ui::MIN_SIZE, show: "keep.toml", relay: false }; + build_keep_clear_compact: Live, Open::Studio { view: View::KeepClear { dual: false }, size: crate::ui::MIN_SIZE, show: "keep.toml", relay: false }; + /// The same going out in both formats: the legend between the landscape + /// picture, the only one it moves in, and the tabs, the portrait picture + /// beside it showing the facecam where its rect rests, fitted. + build_keep_clear_dual: Live, Open::Studio { view: View::KeepClear { dual: true }, size: SIZE, show: "keep.toml", relay: false }; /// A window gone dark: its row, the layer on the picture, its scene's /// pad, and the health dot turn amber, and its badge opens what happened /// and the fixes. @@ -319,6 +354,37 @@ views! { build_drop_refused: Still, dropping(Drop::Refused); /// The logo dropped lands where it was let go, at its own size, selected. build_dropped: Still, dropping(Drop::Placed); + /// A show going out in both formats: the landscape and portrait + /// pictures side by side, each in its tally frame with its size under + /// it, where the selected camera is shown between them, and each pad + /// showing the pair, as the `dual` mock-up frame has it. + live_dual: Still, dual(Page::Live, moq_scene::Format::Landscape); + /// It in a half-size window: the chip of where the camera is shown keeps + /// clear of what is said under the landscape picture, without its words. + live_dual_compact: Still, Open::Studio { view: View::Dual { page: Page::Live, format: moq_scene::Format::Landscape, index: Some(2), cued: None, off_air: None }, size: HALF, show: "dual.toml", relay: false }; + /// Studio mode on it: NEXT shows the cued scene in both formats. + live_dual_studio: Still, Open::Studio { view: View::Dual { page: Page::Live, format: moq_scene::Format::Landscape, index: None, cued: Some("Guests"), off_air: None }, size: SIZE, show: "dual.toml", relay: false }; + /// BUILD on it: the landscape picture edited, the portrait one small + /// beside it, and under them the tabs that pick which and where the + /// selected camera is shown, which the inspector says too. + build_dual: Still, dual(Page::Build, moq_scene::Format::Landscape); + /// BUILD on it in a half-size window: the chip keeps clear of the tabs, + /// or leaves the choice to the inspector where there is no room. + build_dual_compact: Still, Open::Studio { view: View::Dual { page: Page::Build, format: moq_scene::Format::Landscape, index: Some(2), cued: None, off_air: None }, size: HALF, show: "dual.toml", relay: false }; + /// BUILD editing the portrait picture: the camera's handles at its + /// portrait place, the landscape picture small beside it. + build_dual_portrait: Still, dual(Page::Build, moq_scene::Format::Portrait); + /// A scene laid out by slots on it, open off air in portrait: its own + /// arrangement there, a row, which the inspector picks. + build_dual_slots: Still, Open::Studio { view: View::Dual { page: Page::Build, format: moq_scene::Format::Portrait, index: None, cued: None, off_air: Some("Guests") }, size: SIZE, show: "dual.toml", relay: false }; + /// Settings, Show on it: Both at once on. + settings_show_dual: Still, Open::Studio { view: View::Setting(Setting::Show), size: SIZE, show: "dual.toml", relay: false }; + /// GO LIVE's sheet on it: both broadcasts to each relay, and the encoder + /// and connection checked for both pictures. + going_dual: Live, Open::Studio { view: View::Going, size: SIZE, show: "dual.toml", relay: true }; + /// On air on it: both pictures in their ON AIR frames, each with what + /// its video sends under it. + on_air_dual: Live, Open::Studio { view: View::OnAir, size: SIZE, show: "dual.toml", relay: true }; /// MIX: a strip per sound and per audio scene, with their sends, and a /// small live picture in the sound's place. mix: Still, mixing(MixView::Strips, None, SIZE); @@ -1564,6 +1630,9 @@ fn shoot(open: &Open, gpu: &engine::Gpu, runtime: &tokio::runtime::Runtime, show Open::Kit { .. } | Open::Library => unreachable!("drawn without an engine"), Open::Studio { show, view, relay, .. } => { let mut text = std::fs::read_to_string(shows.join(show)).unwrap(); + if let View::KeepClear { dual: true } = view { + text = text.replacen("version = 1\n", "version = 1\nportrait = true\n", 1); + } if let View::Review { replays: true, .. } = view { text.push_str("\n[sources.replay]\nkind = \"replay\"\n\n[sources.slowmo]\nkind = \"replay\"\n"); } @@ -1597,7 +1666,7 @@ fn shoot(open: &Open, gpu: &engine::Gpu, runtime: &tokio::runtime::Runtime, show View::Mix(view, opened) => studio.mix(*view, *opened), View::MixMenu => studio.mix(MixView::Strips, None), View::Build { side, inspect, index } => studio.build(*side, *inspect, *index), - View::KeepClear => studio.build(Side::Layers, Inspect::Layer, Some(0)), + View::KeepClear { .. } => studio.build(Side::Layers, Inspect::Layer, Some(0)), View::Menu => studio.build(Side::Layers, Inspect::Layer, Some(1)), View::Setting(setting) | View::OnAirSetting(setting) => studio.open(*setting), View::Restart | View::CustomCanvas => studio.open(Setting::Show), @@ -1630,6 +1699,24 @@ fn shoot(open: &Open, gpu: &engine::Gpu, runtime: &tokio::runtime::Runtime, show } } View::Drop(_) => studio.build(Side::Add, Inspect::Layer, None), + View::Dual { + page, + format, + index, + cued, + off_air, + } => { + studio.build(Side::Layers, Inspect::Layer, *index); + studio.edit_format(*format); + if let Some(cued) = cued { + studio.studio_mode(true); + studio.cue(cued); + } + studio.turn(*page); + if let Some(scene) = off_air { + studio.edit_off_air(scene); + } + } View::Palette(_) => { let key = |code, name: &str| hotkeys::Binding { modifiers: egui::Modifiers::NONE, @@ -1715,7 +1802,7 @@ fn shoot(open: &Open, gpu: &engine::Gpu, runtime: &tokio::runtime::Runtime, show } })) .expect("the room fails to join at launch"), - View::KeepClear => runtime + View::KeepClear { .. } => runtime .block_on(kio::clock::timeout(SETTLE, async { loop { let landed = { @@ -1738,6 +1825,25 @@ fn shoot(open: &Open, gpu: &engine::Gpu, runtime: &tokio::runtime::Runtime, show harness.run_steps(2); })) .expect("the facecam moves off the minimap and lands"), + View::OnAir if engine.state.lock().unwrap().collection().portrait => runtime + .block_on(kio::clock::timeout(SETTLE, async { + // Each picture's rate shows once health hears what its video + // sent. + let sending = |name: Option<&str>| { + let report = engine.health.lock().unwrap(); + let broadcast = report.broadcasts.iter().find(|found| found.name.as_deref() == name); + broadcast.is_some_and(|broadcast| { + let mut tracks = broadcast.tracks.iter(); + tracks.any(|track| track.label.starts_with("video") && track.sent.is_some()) + }) + }; + while !(sending(None) && sending(Some(moq_scene::PORTRAIT))) { + harness.step(); + kio::clock::sleep(Duration::from_millis(20)).await; + } + harness.run_steps(2); + })) + .expect("both pictures send"), View::OnAirHealth => { let pill = ["Healthy", "Starting", "Degraded", "Unhealthy"] .into_iter() diff --git a/rs/moq-studio/src/ui.rs b/rs/moq-studio/src/ui.rs index 8553a5e1..57fcd3ea 100644 --- a/rs/moq-studio/src/ui.rs +++ b/rs/moq-studio/src/ui.rs @@ -12,8 +12,8 @@ use egui::{Color32, Ui}; use moq_scene::browser::Paint; use moq_scene::source::{Feed, Status}; use moq_scene::{ - Background, Blend, Canvas, Clip, Collection, Crop, Direction, Draft, Ease, Filter, Item, Keyframe, Layer, MaskMode, - Rect, Scene, Source, Thumbnail, Transition, Video, + Background, Blend, Canvas, Clip, Collection, Crop, Direction, Draft, Ease, Filter, Format, Item, Keyframe, Layer, + MaskMode, Placement, Rect, Scene, Source, Thumbnail, Transition, Video, }; use winit::keyboard::{KeyCode, PhysicalKey}; @@ -33,6 +33,7 @@ mod build; mod cue; mod dark; mod drop; +mod formats; mod graph; mod inspector; mod keep; @@ -285,6 +286,10 @@ impl Studio { program, program_drawn: Canvas::default(), preview, + portrait: None, + portrait_drawn: None, + portrait_preview: None, + format: Format::Landscape, editing: None, framed, page: Page::default(), @@ -387,6 +392,8 @@ impl Studio { let mut renderer = render.renderer.write(); for registered in [&panels.program, &panels.preview] .into_iter() + .chain(&panels.portrait) + .chain(&panels.portrait_preview) .chain(panels.thumbnails.values()) { renderer.free_texture(®istered.id); @@ -401,6 +408,11 @@ impl Studio { self.panels.page = page; } + /// Edit `format`'s picture on BUILD, as its tab under the picture would. + pub(crate) fn edit_format(&mut self, format: Format) { + self.panels.format = format; + } + /// Turn to MIX in `view`, with the strip of the sound called `opened` /// open, as clicking its name would. pub(crate) fn mix(&mut self, view: mixer::View, opened: Option<&str>) { @@ -636,6 +648,26 @@ impl Registered { } } + /// Show `texture` in `registered` from now on, while there is one, + /// registering it when it appears and giving it back when it goes, as the + /// portrait picture does when the show stops going out in portrait. + pub(crate) fn follow_while( + registered: &mut Option, + render: &eframe::egui_wgpu::RenderState, + texture: Option<&wgpu::Texture>, + ) { + match (registered.as_mut(), texture) { + (Some(shown), Some(texture)) => shown.follow(render, texture), + (None, Some(texture)) => *registered = Some(Self::new(render, texture)), + (Some(_), None) => { + if let Some(gone) = registered.take() { + render.renderer.write().free_texture(&gone.id); + } + } + (None, None) => {} + } + } + /// Show `texture` from now on, if a resize replaced the one shown. pub(crate) fn follow(&mut self, render: &eframe::egui_wgpu::RenderState, texture: &wgpu::Texture) { if self.texture == *texture { @@ -674,6 +706,15 @@ struct Panels { /// The cued scene, or the one open off air, which the compositor draws /// while there is one. preview: Registered, + /// The portrait picture as viewers see it, while the show goes out in + /// portrait. + portrait: Option, + /// Its size this frame, as its texture was drawn, while there is one. + portrait_drawn: Option, + /// The preview in portrait, while the show goes out in portrait. + portrait_preview: Option, + /// The picture BUILD edits a scene drawn in both in. + format: Format, /// The scene open for editing off air on BUILD, while it is the one cued. editing: Option, /// Whether the title bar is the window's frame, which moves, resizes, @@ -811,6 +852,31 @@ impl Picked { collection.items(&self.layer)?.get(self.index) } + /// The layer at rest in `format`'s picture, at its place there, if it is + /// still there: in the landscape one for a layer drawn in one. + fn rest(&self, collection: &Collection, format: Format) -> Option { + let format = collection.format_of(&self.layer, format); + Some(collection.portrayed(self.item(collection)?, format)) + } + + /// The layer placed at `rect` with `crop` in `format`'s picture, as it + /// is in the other, if it is still there and shown in that picture. + fn placed(&self, collection: &Collection, format: Format, rect: Rect, crop: Crop) -> Option { + let format = collection.format_of(&self.layer, format); + let mut item = self.item(collection)?.clone(); + // Not shown there, it has no place there. + if !item.shown_in(format) { + return None; + } + // Set free and drawn in both without a portrait place, it is fitted + // there from its landscape one, so it keeps where it was fitted. + let follows = format == Format::Landscape && collection.dual(&self.layer) && item.slot.is_none(); + if follows && item.portrait.is_none() && item.shown_in(Format::Portrait) { + item.portrait = Some(collection.placement(&item, Format::Portrait)); + } + Some(item.placed(format, Placement { rect, crop })) + } + /// What its look edits. fn look(&self) -> Selection { Selection::Item(self.layer.clone(), self.index) @@ -2204,7 +2270,7 @@ const REACH: f32 = 8.0; /// than `dragged`: what a dragged layer snaps to. A scene open off air still /// snaps to the always-on-top ones: its picture draws them, as the program /// will, though BUILD leaves them to the program to edit. -fn others(state: &State, scene: Option<&str>, dragged: &Picked, since: Duration) -> Vec { +fn others(state: &State, scene: Option<&str>, dragged: &Picked, since: Duration, format: Format) -> Vec { let collection = state.collection(); // The top layer goes over a scene of the program's size, as the // compositor draws it. @@ -2214,10 +2280,10 @@ fn others(state: &State, scene: Option<&str>, dragged: &Picked, since: Duration) let mut rects = Vec::new(); let layers = scene.map(Layer::scene).into_iter().chain(top.then(Layer::top)); for shown in layers { - let items = slots::placed(state, &shown); + let items = slots::placed(state, &shown, format); for (at, item) in items.iter().enumerate() { if item.visible && !(shown == dragged.layer && at == dragged.index) { - rects.push(keep::placed(state, &shown, at, item).posed(since).rect); + rects.push(item.posed(since).rect); } } } @@ -2903,6 +2969,51 @@ mod tests { ); } + /// While the show goes out in both, placing a layer in the landscape + /// picture leaves where the portrait picture draws it, even fitted there + /// without a place of its own; a layer shown only in landscape takes no + /// portrait place. + #[test] + fn a_landscape_edit_leaves_the_portrait_picture_alone() { + let mut collection = Collection::new(); + collection.portrait = true; + let color = moq_scene::Rgba([255, 255, 255, 255]); + collection.sources.insert("logo".into(), Source::Color { color }); + let rect = |x| Rect { + x, + y: 40.0, + width: 320.0, + height: 180.0, + }; + collection.scenes[0].items = vec![ + Item::new("logo", rect(40.0)), + Item { + format: Some(Format::Landscape), + ..Item::new("logo", rect(40.0)) + }, + ]; + collection.validate().unwrap(); + let picked = |index| Picked { + layer: Layer::scene("Scene"), + index, + }; + let fitted = collection.placement(&collection.scenes[0].items[0], Format::Portrait); + let moved = picked(0).placed(&collection, Format::Landscape, rect(400.0), Crop::default()); + let moved = moved.unwrap(); + assert_eq!(moved.rect, rect(400.0)); + assert_eq!(moved.portrait, Some(fitted), "kept where it was fitted"); + let only = picked(1).placed(&collection, Format::Landscape, rect(400.0), Crop::default()); + assert_eq!(only.unwrap().portrait, None); + + // Placed in portrait, its landscape place stays. + let tall = picked(0).placed(&collection, Format::Portrait, rect(0.0), Crop::default()); + let tall = tall.unwrap(); + assert_eq!( + (tall.rect, tall.portrait.map(|placed| placed.rect)), + (rect(40.0), Some(rect(0.0))) + ); + } + /// A scene duplicated with its own copies keeps an unfilled role as the /// role, as it does a scene drawn in it, and copies a filled one. #[test] @@ -2920,6 +3031,25 @@ mod tests { assert!(build::own_copy(&collection, "Main"), "the source filling it"); } + /// A duplicated scene is laid out as the original in both formats, but is + /// none of the scenes there is one of at most. + #[test] + fn a_duplicated_scene_keeps_its_portrait_arrangement() { + let scene = moq_scene::Scene { + recorded: true, + safe: true, + arrangement: moq_scene::Arrangement::Grid, + portrait: moq_scene::Portrait { + arrangement: moq_scene::Arrangement::Inset, + }, + ..moq_scene::Scene::new("Interview") + }; + let copy = build::scene_copy(&scene, "Interview copy".into()); + assert_eq!(copy.name, "Interview copy"); + assert_eq!((copy.arrangement, copy.portrait), (scene.arrangement, scene.portrait)); + assert!(!copy.recorded && !copy.safe); + } + /// A layer the UI picked is let go of once an edit of the engine's own /// inserts or replaces a layer of its scene, and kept for one elsewhere. #[test] diff --git a/rs/moq-studio/src/ui/build.rs b/rs/moq-studio/src/ui/build.rs index 4c9ad407..c2a6eff4 100644 --- a/rs/moq-studio/src/ui/build.rs +++ b/rs/moq-studio/src/ui/build.rs @@ -44,10 +44,30 @@ impl Panels { kit::wash(ui.painter(), middle); let mut room = middle.shrink(28.0); let mut shape = snapshot.shape; - if let Some(cued) = snapshot.cued.as_ref().filter(|cued| cued.off_air) { + let off_air = snapshot.cued.as_ref().filter(|cued| cued.off_air); + if let Some(cued) = off_air { room.min.y = self.off_air_header(ui, cued, room, snapshot.shape); shape = cued.shape(); } + // A scene drawn in both: the picture edited large, the other small + // beside it, and under them which is edited and where the selected + // layer is shown. + let dual = { + let state = self.engine.state.lock().unwrap(); + self.edits_both(&state) + }; + let pictures = match (dual, off_air) { + (Some(portrait), Some(cued)) => self.portrait_preview.as_ref().map(|portrait_preview| { + let landscape = (self.preview.id, cued.canvas); + (landscape, (portrait_preview.id, portrait)) + }), + (Some(_), None) => self + .portrait + .as_ref() + .zip(self.portrait_drawn) + .map(|(registered, portrait)| ((self.program.id, self.program_drawn), (registered.id, portrait))), + (None, _) => None, + }; // A drag that selects a layer keeping clear leaves the picture where // it was until let go, so the pointer stays over what it moves. if self.drag.is_none() { @@ -57,14 +77,57 @@ impl Panels { if keeping { room.max.y -= keep::LEGEND; } - let at = super::layout::fit(room, shape); - self.stage(ui, fx, snapshot, at); + let Some((landscape, portrait)) = pictures else { + self.format = Format::Landscape; + let at = super::layout::fit(room, shape); + self.stage(ui, fx, snapshot, at); + if keeping { + keep::legend(ui, at); + } + self.build_keys(ui); + return; + }; + let (edited, other) = match self.format { + Format::Landscape => (landscape, portrait), + Format::Portrait => (portrait, landscape), + }; + let gap = super::formats::GAP; + room.max.y -= super::formats::UNDER; + // The other picture a third as tall as the edited one. + let shapes = [super::shell::shape(edited.1), super::shell::shape(other.1) / 3.0]; + let [main, beside] = super::layout::pair(room, shapes, gap); + let small = egui::Rect::from_min_size( + beside.min, + egui::vec2(beside.width(), beside.width() / super::shell::shape(other.1)), + ); + self.stage(ui, fx, snapshot, main); + self.other_picture(ui, small, other.0, other.1); + // Under both pictures: the legend of what a layer keeping clear saw, + // then the tabs at the left and the chip at the right. + let mut top = main.bottom(); if keeping { - keep::legend(ui, at); + keep::legend(ui, main); + top += keep::LEGEND; } + let under = egui::Rect::from_min_max( + egui::pos2(main.left(), top + 18.0), + egui::pos2(main.right().max(small.right()), top + 48.0), + ); + let tabs = self.format_tabs(ui, under); + self.shown_in(ui, under.with_min_x(tabs.right() + 12.0), None); self.build_keys(ui); } + /// The portrait picture's canvas while BUILD edits a scene drawn in both, + /// the one open off air or else the one on program. + pub(super) fn edits_both(&self, state: &State) -> Option { + let collection = state.collection(); + let scene = self.edited(state)?; + collection + .portrait_canvas() + .filter(|_| collection.dual(&Layer::scene(scene))) + } + /// The page's keys, unless a field or a page has the keyboard: A adds a /// layer, and Escape leaves cropping, then the selection. fn build_keys(&mut self, ui: &Ui) { @@ -111,10 +174,26 @@ impl Panels { }; (canvas, tag) }); - let (canvas, id, frame) = match off_air { - Some((canvas, tag)) => (canvas, self.preview.id, TallyFrame::new(Tally::Cue).tag(tag)), + // The portrait picture, while BUILD edits it: on program, at the size + // it was drawn at, as the landscape one is. + let portrait = match self.format { + Format::Portrait => self.edits_both(&state).and_then(|canvas| match off_air { + Some(_) => Some(canvas).zip(self.portrait_preview.as_ref().map(|registered| registered.id)), + None => self + .portrait_drawn + .zip(self.portrait.as_ref().map(|registered| registered.id)), + }), + Format::Landscape => None, + }; + if portrait.is_none() { + self.format = Format::Landscape; + } + let (canvas, id, frame) = match (off_air, portrait) { + (Some((_, tag)), Some((canvas, id))) => (canvas, id, TallyFrame::new(Tally::Cue).tag(tag)), + (Some((canvas, tag)), None) => (canvas, self.preview.id, TallyFrame::new(Tally::Cue).tag(tag)), // It is the live picture, so its frame says whether it is on air. - None => (self.program_drawn, self.program.id, self.program_frame()), + (None, Some((canvas, id))) => (canvas, id, self.program_frame()), + (None, None) => (self.program_drawn, self.program.id, self.program_frame()), }; let builder = egui::UiBuilder::new().max_rect(at.expand(10.0)); if self.interacting { @@ -133,9 +212,9 @@ impl Panels { // `interacted` found the page through the selection, so one is. let picked = self.selection.clone().expect("interacting with the selected layer"); let only = Some((&picked.layer, picked.index)); - let staged = super::slots::staged(&state, scene.map(|(scene, _)| scene)); + let staged = super::slots::staged(&state, scene.map(|(scene, _)| scene), self.format); // A layer keeping clear takes the pointer where it is drawn. - let staged = keep::drawn(&state, staged); + let staged = keep::drawn(&state, staged, self.format); let delivered = self.interact.update(ui, image, &staged, scene, only); drop(staged); deliver(&state.sources, delivered); @@ -203,15 +282,24 @@ impl Panels { }; let state = self.engine.state.lock().unwrap(); let item = picked.item(state.collection()); - item.is_some_and(|item| item.visible && keep::placement(&state, &picked.layer, picked.index, item).is_some()) + item.is_some_and(|item| { + self.drawn(&state, picked, item) + && keep::placement(&state, &picked.layer, picked.index, item, self.format).is_some() + }) } - /// The web page the selected layer shows, while it is a visible layer of - /// the picture that shows one. + /// Whether `item`, `picked`, is drawn in the picture BUILD edits. + fn drawn(&self, state: &State, picked: &Picked, item: &Item) -> bool { + let format = state.collection().format_of(&picked.layer, self.format); + item.visible && item.shown_in(format) + } + + /// The web page the selected layer shows, while it is a layer drawn in + /// the picture edited that shows one. fn interacted(&self, state: &State) -> Option { let picked = self.selection.as_ref()?; let collection = state.collection(); - let item = picked.item(collection).filter(|item| item.visible)?; + let item = picked.item(collection).filter(|item| self.drawn(state, picked, item))?; let shown = match &picked.layer.scene { Some(scene) => self.edited(state) == Some(scene.as_str()), None => self.off_air(state).is_none(), @@ -239,7 +327,7 @@ impl Panels { let layers = self.scene(state).ok().into_iter().chain(top); layers .flat_map(|layer| { - let items = super::slots::placed(state, &layer); + let items = super::slots::placed(state, &layer, self.format); let posed = items.into_iter().map(move |item| item.posed(since).into_owned()); posed.enumerate().map(move |(index, item)| { let picked = Picked { @@ -312,11 +400,11 @@ impl Panels { let collection = state.collection(); // A layer keeping clear moves from where it was placed. let resting = picked - .item(collection) - .map(|item| keep::placed(state, &picked.layer, picked.index, item)); + .rest(collection, self.format) + .map(|item| keep::placed(state, &picked.layer, picked.index, &item, self.format)); if let Some(resting) = resting { let drawn = match resting.slot { - Some(_) => super::slots::drawn(state, &picked.layer, picked.index), + Some(_) => super::slots::drawn(state, &picked.layer, picked.index, self.format), None => None, }; let drawn = drawn.unwrap_or((resting.rect, resting.crop)); @@ -390,7 +478,7 @@ impl Panels { let point = view.point(pos); // Whole canvas pixels, so an unsnapped drag saves round numbers. let delta = [(point[0] - drag.origin[0]).round(), (point[1] - drag.origin[1]).round()]; - let item = drag.picked.item(state.collection()).cloned(); + let item = drag.picked.rest(state.collection(), self.format); let held = ui.input(|input| input.modifiers); let (rect, crop) = match self.cropping { true => { @@ -416,7 +504,7 @@ impl Panels { height: shown.height - rest.height, }, }; - let others = others(state, scene.as_deref(), &drag.picked, since); + let others = others(state, scene.as_deref(), &drag.picked, since, self.format); snap.apply(drag.grab, rect, &others, &canvas) }); let rect = drag.hints.as_ref().map_or(rect, |snapped| snapped.rect); @@ -443,14 +531,16 @@ impl Panels { // a resize or a crop leaves it where it rests. let rect = match verb { "Move" => { - keep::hold(state, &picked.layer, picked.index, &item); + keep::hold(state, &picked.layer, picked.index, &item, self.format); rect } - _ => keep::resized(state, &picked.layer, picked.index, &item, rect), + _ => keep::resized(state, &picked.layer, picked.index, &item, rect, self.format), }; - result = edit(state, label, drag.gesture, |draft| { - draft.set_item(&picked.layer, picked.index, Item { rect, crop, ..item }) - }); + if let Some(placed) = picked.placed(state.collection(), self.format, rect, crop) { + result = edit(state, label, drag.gesture, |draft| { + draft.set_item(&picked.layer, picked.index, placed) + }); + } } } if !response.dragged() { @@ -458,7 +548,7 @@ impl Panels { } // A layer keeping clear stays put while in hand, which each frame // says again, at least four times a lease, whatever the canvas rate. - keep::grab(state, self.drag.as_ref().map(|drag| &drag.picked)); + keep::grab(state, self.drag.as_ref().map(|drag| &drag.picked), self.format); if self.drag.is_some() { ui.ctx().request_repaint_after(moq_scene::place::GRAB / 4); } @@ -477,7 +567,7 @@ impl Panels { // being dragged, which holds it where it is put. if let Some((picked, item)) = selected.filter(|(_, item)| item.visible && self.drag.is_none()) && let Some(resting) = picked.item(state.collection()) - && let Some(placement) = keep::placement(state, &picked.layer, picked.index, resting) + && let Some(placement) = keep::placement(state, &picked.layer, picked.index, resting, self.format) && let Some(keep) = resting.keep_clear { keep::overlay(&painter, &view, placement, &item.shape, keep.glide); @@ -487,7 +577,7 @@ impl Panels { if let Some((picked, item)) = selected.filter(|(_, item)| item.visible) { if self.cropping { let collection = state.collection(); - let resting = picked.item(collection).map_or(item.crop, |item| item.crop); + let resting = picked.rest(collection, self.format).map_or(item.crop, |item| item.crop); view.whole(&painter, &item.rect, &resting); } // A layer in a slot has no size of its own to drag. @@ -576,14 +666,13 @@ impl Panels { } }); let Some(item) = picked - .item(state.collection()) + .rest(state.collection(), self.format) .filter(|item| !item.locked && item.slot.is_none()) - .cloned() else { return (None, Ok(())); }; // A layer keeping clear moves from where it was placed. - let item = keep::placed(state, &picked.layer, picked.index, &item); + let item = keep::placed(state, &picked.layer, picked.index, &item, self.format); if delta == [0.0; 2] { return (None, Ok(())); } @@ -598,9 +687,12 @@ impl Panels { }; let label = format!("Move `{}`", item.source); let spot = Spot::Layer(picked.layer.clone(), item.source.clone()); - keep::hold(state, &picked.layer, picked.index, &item); + let Some(moved) = picked.placed(state.collection(), self.format, rect, item.crop) else { + return (None, Ok(())); + }; + keep::hold(state, &picked.layer, picked.index, &item, self.format); let result = edit(state, label, Some(gesture), |draft| { - draft.set_item(&picked.layer, picked.index, Item { rect, ..item }) + draft.set_item(&picked.layer, picked.index, moved) }); (Some(spot), result) } @@ -733,22 +825,38 @@ impl Panels { "`{name}` is in a slot, which places it; set it free to place it yourself" )), LayerAction::Slot(slot) => super::slots::set_slot(state, picked, slot), - LayerAction::Fit => { - let canvas = collection.canvas(layer.scene.as_deref()).unwrap_or(collection.canvas); - let rect = match aspect(state, &item, true) { - Some(aspect) => snap::fit(aspect, &canvas), - None => Rect::fill(&canvas), + LayerAction::Fit | LayerAction::Fill => { + // In the picture BUILD edits. + let format = collection.format_of(layer, self.format); + let canvas = match collection.dual(layer) { + true => collection.canvas_of(format), + false => collection.canvas(layer.scene.as_deref()).unwrap_or(collection.canvas), }; - set(state, format!("Fit `{name}`"), Item { rect, ..item }) - } - LayerAction::Fill => { - let canvas = collection.canvas(layer.scene.as_deref()).unwrap_or(collection.canvas); - let crop = match aspect(state, &item, false) { - Some(aspect) => snap::fill(aspect, &canvas), - None => Crop::default(), + if !item.shown_in(format) { + let why = format!("`{name}` is not shown in the {} picture", format.name().to_lowercase()); + return (spot, Err(anyhow::anyhow!(why))); + } + let rest = collection.portrayed(&item, format); + let (label, rect, crop) = match action { + LayerAction::Fit => { + let rect = match aspect(state, &rest, true, format) { + Some(aspect) => snap::fit(aspect, &canvas), + None => Rect::fill(&canvas), + }; + (format!("Fit `{name}`"), rect, rest.crop) + } + _ => { + let crop = match aspect(state, &rest, false, format) { + Some(aspect) => snap::fill(aspect, &canvas), + None => Crop::default(), + }; + (format!("Fill with `{name}`"), Rect::fill(&canvas), crop) + } }; - let rect = Rect::fill(&canvas); - set(state, format!("Fill with `{name}`"), Item { rect, crop, ..item }) + match picked.placed(collection, format, rect, crop) { + Some(placed) => set(state, label, placed), + None => Ok(()), + } } LayerAction::Shape(corners) => { let shape = moq_scene::Shape { corners, ..item.shape }; @@ -795,16 +903,31 @@ impl Panels { let count = collection.items(&to).map_or(0, <[Item]>::len); // The always-on-top layers never go on, so they have no // keyframes to play, nor slots, nor a picture under them to - // keep clear of; it stays where it was drawn. - let placed = keep::placed(state, layer, index, &item).rect; - let (rect, crop) = super::slots::drawn(state, layer, index).unwrap_or((item.rect, item.crop)); - let rect = if item.slot.is_none() { placed } else { rect }; + // keep clear of; it stays where it was drawn, in each picture + // it goes to. + let drawn = |format| { + let placed = collection.portrayed(&keep::placed(state, layer, index, &item, format), format); + let (rect, crop) = + super::slots::drawn(state, layer, index, format).unwrap_or((item.rect, item.crop)); + match item.slot { + None => (placed.rect, placed.crop), + Some(_) => (rect, crop), + } + }; + let (rect, crop) = drawn(Format::Landscape); + let dual = collection.dual(&to) && collection.dual(layer); + let portrait = dual.then(|| { + let (rect, crop) = drawn(Format::Portrait); + Placement { rect, crop } + }); let moved = Item { keyframes: if top { Vec::new() } else { item.keyframes.clone() }, slot: None, keep_clear: if top { None } else { item.keep_clear }, rect, crop, + format: item.format.filter(|_| dual), + portrait: portrait.filter(|_| item.shown_in(Format::Portrait)), ..item.clone() }; let label = match top { @@ -1351,6 +1474,28 @@ pub(super) fn own_copy(collection: &Collection, source: &str) -> bool { ) } +/// `scene` named `name`, laid out the same in each format, but neither +/// published, recorded, nor safe, and carrying the audio scene on program. +pub(super) fn scene_copy(scene: &Scene, name: String) -> Scene { + Scene { + name, + published: false, + recorded: false, + audio: None, + safe: false, + // A copy is the host's, never the studio's to give away. + items: scene + .items + .iter() + .map(|item| Item { + auto: false, + ..item.clone() + }) + .collect(), + ..scene.clone() + } +} + /// Add a scene after the one at `index`, as large and showing the same /// layers, named for it, and return its name. Its layers show the same /// sources, or with `independent` each a copy of its own, keeping the look, @@ -1362,21 +1507,7 @@ pub(super) fn duplicate_scene(state: &mut State, index: usize, independent: bool .chain((2..).map(|n| format!("{} copy {n}", scene.name))) .find(|name| collection.scene(name).is_none()) .expect("an unused name"); - let mut copy = Scene { - name: name.clone(), - size: scene.size, - arrangement: scene.arrangement, - // A copy is the host's, never the studio's to give away. - items: scene - .items - .iter() - .map(|item| Item { - auto: false, - ..item.clone() - }) - .collect(), - ..Scene::new("") - }; + let mut copy = scene_copy(scene, name.clone()); // Each source shown, and the copy of its own each of its layers shows. let mut copies: BTreeMap)> = BTreeMap::new(); if independent { @@ -1415,15 +1546,21 @@ pub(super) fn duplicate_scene(state: &mut State, index: usize, independent: bool /// The width over height of what `item` shows: after its crop if `cropped`, /// else after its source's alone. `None` for a source that fills any shape, /// such as a color. -fn aspect(state: &State, item: &Item, cropped: bool) -> Option { +fn aspect(state: &State, item: &Item, cropped: bool, format: Format) -> Option { let collection = state.collection(); let capture = collection.capture(&item.source); let (width, height) = match collection.sources.get(capture)? { Source::Browser { width, height, .. } | Source::Overlay { width, height, .. } => (*width, *height), + // The program, or a scene, as `format`'s picture draws it: turned + // in portrait while drawn in both. Source::Scene { scene } => collection + .formatted(format) .canvas(Some(scene)) .map(|canvas| (canvas.width, canvas.height))?, - Source::Program => (collection.canvas.width, collection.canvas.height), + Source::Program => { + let canvas = collection.canvas_of(format); + (canvas.width, canvas.height) + } Source::Color { .. } => return None, _ => state.compositor.picture_size(capture)?, }; diff --git a/rs/moq-studio/src/ui/cue.rs b/rs/moq-studio/src/ui/cue.rs index 1aca9a96..f68183d2 100644 --- a/rs/moq-studio/src/ui/cue.rs +++ b/rs/moq-studio/src/ui/cue.rs @@ -13,6 +13,8 @@ pub(super) struct Cued { pub(super) refused: Option, /// Its size, which its picture is drawn at. pub(super) canvas: Canvas, + /// Its portrait picture's size, while it goes out in both. + pub(super) portrait: Option, /// Whether it is open for editing off air, rather than only cued. pub(super) off_air: bool, /// Why viewers see it anyway, if they do, such as being published. @@ -24,10 +26,22 @@ pub(super) struct Cued { impl Cued { /// Its picture's shape, width over height. pub(super) fn shape(&self) -> f32 { - self.canvas.width as f32 / self.canvas.height.max(1) as f32 + super::shell::shape(self.canvas) + } + + /// The shape of its pictures side by side, while it goes out in both, + /// with room between them, or else of its picture. + fn card_shape(&self) -> f32 { + match self.portrait { + Some(portrait) => super::layout::pair_shape([self.shape(), super::shell::shape(portrait)], CARD_GAP), + None => self.shape(), + } } } +/// The gap between the NEXT card's pictures, as a share of their height. +const CARD_GAP: f32 = 0.06; + /// A corner of the live picture, where the NEXT card docks. #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub(super) enum Corner { @@ -76,7 +90,7 @@ impl Panels { let inner = width - 2.0 * PADDING; // The rest of the card is about 120 points tall; the picture gets what // the live picture leaves. - let tall = (inner / cued.shape()) + let tall = (inner / cued.card_shape()) .min(picture.height() - 2.0 * INSET - 120.0) .max(24.0); let (header, buttons, hint) = (24.0, 40.0, 14.0); @@ -140,11 +154,20 @@ impl Panels { // The cued scene as it will go on air, framed in amber. let frame = Rect::from_min_size(pos2(card.left() + PADDING, y), vec2(inner, tall)); painter.rect_filled(frame, 8, kit::BG); - let shown = vec2(cued.canvas.width as f32, cued.canvas.height as f32); - let fit = shown * (frame.shrink(3.0).size() / shown).min_elem(); - egui::Image::new((self.preview.id, fit)) - .corner_radius(6) - .paint_at(ui, Rect::from_center_size(frame.center(), fit)); + // Both of its pictures, side by side, while it goes out in both. + let room = frame.shrink(3.0); + let pictures = match (cued.portrait, &self.portrait_preview) { + (Some(portrait), Some(registered)) => { + let shapes = [cued.shape(), super::shell::shape(portrait)]; + let gap = CARD_GAP * room.height(); + let [wide, tall] = super::layout::pair(room, shapes, gap); + vec![(self.preview.id, wide), (registered.id, tall)] + } + _ => vec![(self.preview.id, super::layout::fit(room, cued.shape()))], + }; + for (id, at) in pictures { + egui::Image::new((id, at.size())).corner_radius(6).paint_at(ui, at); + } ui.painter() .rect_stroke(frame, 8, Stroke::new(2.0, kit::CUE), egui::StrokeKind::Inside); y = frame.bottom() + 8.0; diff --git a/rs/moq-studio/src/ui/dark.rs b/rs/moq-studio/src/ui/dark.rs index 7878b6ad..fa44badd 100644 --- a/rs/moq-studio/src/ui/dark.rs +++ b/rs/moq-studio/src/ui/dark.rs @@ -197,9 +197,11 @@ pub(super) fn counted(dark: usize, stalled: usize) -> Option { /// Whether the scene called `scene` draws a layer that `lost` its picture, /// as `lost` says of a source: through the scenes it nests, and the masks -/// its layers read, and not one it hides. +/// its layers read, in either picture, and not one it hides. pub(super) fn draws_lost(collection: &Collection, scene: &str, lost: impl Fn(&str) -> bool) -> bool { - collection.drawn([Layer::scene(scene)]).iter().any(|item| { + let formats = moq_scene::Format::ALL.into_iter(); + let mut drawn = formats.flat_map(|format| collection.drawn([Layer::scene(scene)], format)); + drawn.any(|item| { let masks = collection.filters(item).filter_map(|filter| match filter { Filter::Mask { source, .. } => Some(source.as_str()), _ => None, diff --git a/rs/moq-studio/src/ui/drop.rs b/rs/moq-studio/src/ui/drop.rs index 2c498927..212f9feb 100644 --- a/rs/moq-studio/src/ui/drop.rs +++ b/rs/moq-studio/src/ui/drop.rs @@ -53,6 +53,9 @@ impl Arrival { struct Landing { /// The layers it joins. layer: Layer, + /// The picture it lands in, BUILD's while it edits a scene drawn in both, + /// and that picture's canvas. + format: Format, canvas: Canvas, /// Its center, in canvas pixels. at: [f32; 2], @@ -538,7 +541,11 @@ impl Panels { fn landing(&self, state: &State, at: Option<[f32; 2]>) -> Result { let layer = self.scene(state)?; let collection = state.collection(); - let canvas = collection.canvas(layer.scene.as_deref()).unwrap_or(collection.canvas); + let format = collection.format_of(&layer, self.format); + let canvas = match collection.dual(&layer) { + true => collection.canvas_of(format), + false => collection.canvas(layer.scene.as_deref()).unwrap_or(collection.canvas), + }; let center = [canvas.width as f32 / 2.0, canvas.height as f32 / 2.0]; let onto = at.and_then(|at| { let shown = self.shown(state); @@ -548,6 +555,7 @@ impl Panels { }); Ok(Landing { layer, + format, canvas, at: at.unwrap_or(center), onto, @@ -738,13 +746,36 @@ impl Panels { let layer = &landing.layer; let heard = source.has_audio(); let index = state.collection().items(layer).map_or(0, <[Item]>::len); + // Landed in the portrait picture, it sits there; in the landscape + // one, it lands centered at the same size until moved there. One that + // landed in portrait before its layers stopped being drawn in both, + // such as while its file was read, lands centered in their one. + let collection = state.collection(); + let centered = |canvas: Canvas, rect: Rect| { + let center = [canvas.width as f32 / 2.0, canvas.height as f32 / 2.0]; + land(canvas, [rect.width, rect.height], center) + }; + let item = rect.map(|rect| match (landing.format, collection.dual(layer)) { + (Format::Landscape, _) => Item::new(name.clone(), rect), + (Format::Portrait, true) => Item { + portrait: Some(Placement { + rect, + crop: Crop::default(), + }), + ..Item::new(name.clone(), centered(collection.canvas, rect)) + }, + (Format::Portrait, false) => { + let canvas = collection.canvas(layer.scene.as_deref()).unwrap_or(collection.canvas); + Item::new(name.clone(), centered(canvas, rect)) + } + }); edit(state, format!("Drop `{name}` {}", place(layer)), None, |draft| { draft.set_source(&name, Some(source))?; if heard { draft.hear(&name)?; } - if let Some(rect) = rect { - draft.insert_item(layer, index, Item::new(name.clone(), rect))?; + if let Some(item) = item { + draft.insert_item(layer, index, item)?; } Ok(()) })?; diff --git a/rs/moq-studio/src/ui/formats.rs b/rs/moq-studio/src/ui/formats.rs new file mode 100644 index 00000000..6beddaae --- /dev/null +++ b/rs/moq-studio/src/ui/formats.rs @@ -0,0 +1,235 @@ +//! A show going out in landscape and portrait at once: what is said under +//! its two pictures, where a layer is shown, and which picture BUILD edits. + +use egui::{Align2, Sense, Stroke, StrokeKind, pos2, vec2}; +use moq_net::bandwidth::Rate; + +use super::*; + +/// Room between the two pictures on LIVE and BUILD. +pub(super) const GAP: f32 = 32.0; +/// Room under them for what is said about them. +pub(super) const UNDER: f32 = 48.0; + +/// Where a layer may be shown, as the chip and the inspector offer it: +/// `None` for both pictures. +pub(super) fn shown_in_options() -> [(Option, &'static str); 3] { + [ + (None, "Both"), + (Some(Format::Landscape), "Landscape"), + (Some(Format::Portrait), "Portrait"), + ] +} + +/// `item` shown only in `format`, or in both for `None`. Shown only in +/// landscape, it keeps no place in portrait. +pub(super) fn shown(item: &Item, format: Option) -> Item { + let portrait = match format { + Some(Format::Landscape) => None, + _ => item.portrait, + }; + Item { + format, + portrait, + ..item.clone() + } +} + +/// The edit's label for showing `name` as [`shown`] does. +pub(super) fn shown_label(name: &str, format: Option) -> String { + match format { + None => format!("Show `{name}` in both pictures"), + Some(format) => format!("Show `{name}` only in {}", format.name().to_lowercase()), + } +} + +/// Under a picture, in `rect`: its format, its size, and what its video sent +/// over the last interval, while on air, as many of those as fit, right to +/// left from the rate, with all of them on hover where some did not. The part +/// of `rect` left free between them. +pub(super) fn format_label( + ui: &Ui, + rect: egui::Rect, + format: Format, + canvas: Canvas, + sent: Option, +) -> egui::Rect { + let painter = ui.painter(); + let mid = rect.center().y; + mark(painter, pos2(rect.left() + 7.0, mid), format, kit::DIM); + let dims = format!("{} × {}", canvas.width, canvas.height); + let rate = sent.map(crate::health::rate); + let name = kit::caps_galley(painter, format.name(), kit::display_bold(11.5), kit::TEXT, None); + let size = painter.layout_no_wrap(dims.clone(), kit::ui(12.0), kit::DIM); + let said = rate + .clone() + .map(|rate| painter.layout_no_wrap(rate, kit::ui(12.0), kit::DIM)); + // Past the mark, and the rate with its dot. + let x = rect.left() + 20.0; + let mut room = rect.right() - x; + let said = said.filter(|said| said.size().x + 24.0 <= room); + if let Some(said) = &said { + room -= said.size().x + 24.0; + } + let named = name.size().x <= room; + let sized = named && name.size().x + 10.0 + size.size().x <= room; + let mut free = rect.with_min_x(x); + if named { + let width = name.size().x; + painter.galley(pos2(x, mid - name.size().y / 2.0), name, kit::TEXT); + free.min.x = x + width + 16.0; + } + if sized { + let at = pos2(free.min.x - 6.0, mid - size.size().y / 2.0); + free.min.x = at.x + size.size().x + 16.0; + painter.galley(at, size, kit::DIM); + } + free.max.x = rect.right(); + let all = said.is_some() == rate.is_some(); + if let Some(said) = said { + let at = pos2(rect.right() - said.size().x, mid - said.size().y / 2.0); + painter.circle_filled(pos2(at.x - 9.0, mid), 3.5, kit::OK); + free.max.x = at.x - 24.0; + painter.galley(at, said, kit::DIM); + } + if !(named && sized && all) { + let words = [Some(format.name().to_string()), Some(dims), rate]; + let words: Vec = words.into_iter().flatten().collect(); + let id = ui.id().with(("format label", format)); + ui.interact(rect, id, Sense::hover()).on_hover_text(words.join(" · ")); + } + free.max.x = free.max.x.max(free.min.x); + free +} + +/// A small outline of `format`'s shape, centered at `at`, in `ink`. +pub(super) fn mark(painter: &egui::Painter, at: egui::Pos2, format: Format, ink: Color32) { + let size = match format { + Format::Landscape => vec2(13.0, 9.0), + Format::Portrait => vec2(8.0, 12.0), + }; + let rect = egui::Rect::from_center_size(at, size); + painter.rect_stroke(rect, 2, Stroke::new(1.3, ink), StrokeKind::Inside); +} + +impl Panels { + /// In `rect`, under the pictures, centered on `center` as near as it fits, + /// or at its right for `None`: where the selected layer is shown, both + /// pictures, landscape, or portrait, for a layer drawn in both. Picking + /// another shows it there. Without its words where `rect` is too narrow + /// for them, and not at all where it is too narrow for the choices, which + /// the inspector offers too. + pub(super) fn shown_in(&mut self, ui: &mut Ui, rect: egui::Rect, center: Option) { + let engine = self.engine.clone(); + let mut state = engine.state.lock().unwrap(); + self.follow_reloads(&state); + let Some(picked) = self.selection.clone() else { return }; + let collection = state.collection(); + if !collection.dual(&picked.layer) { + return; + } + let Some(item) = picked.item(collection).cloned() else { + return; + }; + let options = shown_in_options(); + let font = kit::ui_strong(12.5); + let words = format!("{} is shown in", item.source); + let words = ui.painter().layout_no_wrap(words, kit::ui(12.0), kit::DIM); + let choices: f32 = options + .iter() + .map(|(_, name)| { + ui.painter() + .layout_no_wrap(name.to_string(), font.clone(), kit::TEXT) + .size() + .x + 28.0 + }) + .sum::() + + 4.0; + let full = 12.0 + words.size().x + 10.0 + choices + 4.0; + let (width, words) = match full <= rect.width() { + true => (full, Some(words)), + false => (choices + 8.0, None), + }; + if width > rect.width() { + return; + } + let size = vec2(width, 42.0); + let x = match center { + Some(center) => (center - size.x / 2.0).clamp(rect.left(), rect.right() - size.x), + None => rect.right() - size.x, + }; + let chip = egui::Rect::from_min_size(pos2(x, rect.center().y - 21.0), size); + ui.painter() + .rect(chip, 10, kit::P1, Stroke::new(1.0, kit::LINE), StrokeKind::Inside); + if let Some(words) = words { + let at = pos2(chip.left() + 12.0, chip.center().y - words.size().y / 2.0); + ui.painter().galley(at, words, kit::DIM); + } + let segmented = egui::Rect::from_min_size( + pos2(chip.right() - 4.0 - choices, chip.top() + 4.0), + vec2(choices, chip.height() - 8.0), + ); + let mut format = item.format; + ui.put(segmented, kit::Segmented::new(&mut format, &options)); + if format == item.format { + return; + } + let name = item.source.clone(); + let shown = shown(&item, format); + let spot = Spot::Layer(picked.layer.clone(), name.clone()); + let result = edit(&mut state, shown_label(&name, format), None, |draft| { + draft.set_item(&picked.layer, picked.index, shown) + }); + drop(state); + self.fail(spot, result); + } + + /// BUILD's choice of the picture it edits, a scene drawn in both: the + /// landscape one or the portrait one, at the left of `rect`; where it is. + pub(super) fn format_tabs(&mut self, ui: &mut Ui, rect: egui::Rect) -> egui::Rect { + let options = [(Format::Landscape, "Landscape"), (Format::Portrait, "Portrait")]; + let mut format = self.format; + let at = egui::Rect::from_min_size(rect.left_center() - vec2(0.0, 17.0), vec2(rect.width(), 34.0)); + let tabs = ui.scope_builder(egui::UiBuilder::new().max_rect(at), |ui| { + ui.add(kit::Segmented::new(&mut format, &options)) + .on_hover_text("The picture to edit: each places the layers on its own") + .rect + }); + self.format = format; + tabs.inner + } + + /// The picture BUILD is not editing, small in `rect`, `id` drawn on + /// `canvas`, with its format under it. A click edits it instead. + pub(super) fn other_picture(&mut self, ui: &mut Ui, rect: egui::Rect, id: egui::TextureId, canvas: Canvas) { + let other = self.format.other(); + let response = ui + .put( + rect, + egui::Image::new((id, rect.size())) + .corner_radius(6) + .sense(Sense::click()), + ) + .on_hover_text(format!("Edit the {} picture", other.name().to_lowercase())) + .on_hover_cursor(egui::CursorIcon::PointingHand); + let ink = match response.hovered() { + true => kit::FAINT, + false => kit::LINE, + }; + ui.painter() + .rect_stroke(rect, 6, Stroke::new(1.5, ink), StrokeKind::Outside); + let under = egui::Rect::from_min_size(rect.left_bottom() + vec2(0.0, 8.0), vec2(rect.width(), 18.0)); + let painter = ui.painter(); + mark(painter, pos2(under.left() + 7.0, under.center().y), other, kit::DIM); + painter.text( + pos2(under.left() + 18.0, under.center().y), + Align2::LEFT_CENTER, + format!("{} × {}", canvas.width, canvas.height), + kit::ui(11.0), + kit::DIM, + ); + if response.clicked() { + self.format = other; + } + } +} diff --git a/rs/moq-studio/src/ui/inspector.rs b/rs/moq-studio/src/ui/inspector.rs index 934d90ae..630c16a8 100644 --- a/rs/moq-studio/src/ui/inspector.rs +++ b/rs/moq-studio/src/ui/inspector.rs @@ -97,7 +97,8 @@ impl Panels { let mut asked = None; // Whether the edit asked for is a move by hand, or switches keeping // clear off, which leaves it where it is drawn as a move would. - let mut moved = false; + // The picture a layer was moved by hand in, if it was. + let mut moved = None; let kept = item.keep_clear; section(ui, "Place"); @@ -128,10 +129,56 @@ impl Panels { asked = Some(Asked::Act(LayerAction::Slot(slot))); } } + // Where it is shown, and its place in the picture BUILD edits, for a + // layer drawn in both. + let format = self.format; + if collection.dual(&picked.layer) { + let mut shown = item.format; + ui.horizontal(|ui| { + ui.label("Format"); + let options = super::formats::shown_in_options(); + ui.add(kit::Segmented::new(&mut shown, &options)) + .on_hover_text("Which pictures show it: both, or only the landscape or the portrait one"); + }); + if shown != item.format { + asked = Some(Asked::Item( + super::formats::shown(&item, shown), + super::formats::shown_label(&name, shown), + None, + )); + } + if item.slot.is_none() && item.shown_in(format) { + ui.label( + egui::RichText::new(format!("Its place in the {} picture", format.name().to_lowercase())) + .color(kit::DIM) + .font(kit::ui(11.0)), + ); + } + } + let rest = picked.rest(collection, format).unwrap_or_else(|| item.clone()); + let placed = |rect, crop| { + picked + .placed(collection, format, rect, crop) + .unwrap_or_else(|| item.clone()) + }; // Where the picture draws it: a layer keeping clear where it was - // placed, which a move by hand starts from. - let placement = keep::placement(&state, &picked.layer, picked.index, &item); - let mut rect = placement.map_or(item.rect, |placement| placement.drawn(item.rect)); + // placed, which a move by hand starts from. Keeping clear places only + // the landscape picture, so its switch and what it saw speak of that. + let placement = keep::placement(&state, &picked.layer, picked.index, &item, Format::Landscape); + let drawn = keep::placement(&state, &picked.layer, picked.index, &item, format); + let mut rect = drawn.map_or(rest.rect, |placement| placement.drawn(rest.rect)); + // A layer the picture BUILD edits never shows has no place there. + let placeable = item.shown_in(collection.format_of(&picked.layer, format)); + if !placeable { + ui.label( + egui::RichText::new(format!( + "Not shown in the {} picture; place it in the other", + format.name().to_lowercase() + )) + .color(kit::DIM) + .font(kit::ui(11.0)), + ); + } if item.slot.is_some() { ui.label( egui::RichText::new("Placed by the scene's arrangement; drag it onto another slot to swap them") @@ -139,7 +186,7 @@ impl Panels { .font(kit::ui(11.0)), ); } - if item.slot.is_none() { + if item.slot.is_none() && placeable { ui.horizontal(|ui| { ui.spacing_mut().item_spacing.x = 4.0; let width = ((ui.available_width() - 12.0) / 4.0).max(40.0); @@ -164,35 +211,33 @@ impl Panels { // stays free to move. let rect = match verb { "Resize" => Rect { - x: item.rect.x, - y: item.rect.y, + x: rest.rect.x, + y: rest.rect.y, ..rect }, _ => rect, }; - moved = verb == "Move"; - asked = Some(Asked::Item(Item { rect, ..item.clone() }, label, gesture)); + moved = (verb == "Move").then_some(format); + asked = Some(Asked::Item(placed(rect, rest.crop), label, gesture)); } } }); } - let mut crop = item.crop; - ui.horizontal(|ui| { - ui.label(egui::RichText::new("Crop").color(kit::DIM)); - if let Some(gesture) = crop_ui(ui, &mut crop, &mut self.gestures) { - asked = Some(Asked::Item( - Item { crop, ..item.clone() }, - format!("Crop `{name}`"), - gesture, - )); - } - }) - .response - .on_hover_text("Percent of the picture trimmed from the left, top, right, and bottom"); + let mut crop = rest.crop; + if placeable { + ui.horizontal(|ui| { + ui.label(egui::RichText::new("Crop").color(kit::DIM)); + if let Some(gesture) = crop_ui(ui, &mut crop, &mut self.gestures) { + asked = Some(Asked::Item(placed(rest.rect, crop), format!("Crop `{name}`"), gesture)); + } + }) + .response + .on_hover_text("Percent of the picture trimmed from the left, top, right, and bottom"); + } ui.horizontal(|ui| { - let fits = match item.slot { - Some(_) => &[][..], - None => &[ + let fits = match (item.slot, placeable) { + (Some(_), _) | (_, false) => &[][..], + (None, true) => &[ ( "Fit", LayerAction::Fit, @@ -223,11 +268,14 @@ impl Panels { } }); // Only a scene's layer has a picture under it to keep clear of, and - // viewers place a published source where the scene file does. + // viewers place a published source where the scene file does. Keeping + // clear places only the landscape picture, so not one shown only in + // portrait. let published = collection.published.contains(collection.capture(&item.source)); if picked.layer.scene.is_some() && item.slot.is_none() && !published + && item.shown_in(Format::Landscape) && let Some((keep_clear, label)) = keep::inspector(ui, item.keep_clear, placement) { let label = format!("{label}: `{name}`"); @@ -235,7 +283,7 @@ impl Panels { // on top does, rather than going back over what it moved off. let rect = match (keep_clear, placement) { (None, Some(placement)) => { - moved = true; + moved = Some(Format::Landscape); placement.drawn(item.rect) } _ => item.rect, @@ -552,13 +600,13 @@ impl Panels { let Some(asked) = asked else { return }; let result = match asked { Asked::Item(item, label, gesture) => { - if moved { + if let Some(format) = moved { // Held as it was before the edit, so an undo puts it back. let was = Item { keep_clear: kept, ..item.clone() }; - keep::hold(&mut state, &picked.layer, picked.index, &was); + keep::hold(&mut state, &picked.layer, picked.index, &was, format); } edit(&mut state, label, gesture, |draft| { draft.set_item(&picked.layer, picked.index, item) diff --git a/rs/moq-studio/src/ui/keep.rs b/rs/moq-studio/src/ui/keep.rs index 8c4c5e36..fec84426 100644 --- a/rs/moq-studio/src/ui/keep.rs +++ b/rs/moq-studio/src/ui/keep.rs @@ -25,18 +25,28 @@ const GLIDES: [Duration; 5] = [ /// The legend's height under the stage, with its gap. pub(super) const LEGEND: f32 = 30.0; -/// How the last render placed `item`, the `index`th of `layer`, while it -/// keeps clear. -pub(super) fn placement<'a>(state: &'a State, layer: &Layer, index: usize, item: &Item) -> Option<&'a Placement> { +/// How the last render placed `item`, the `index`th of `layer`, in +/// `format`'s picture while it keeps clear. Keeping clear places only the +/// landscape picture, so none in the portrait one. +pub(super) fn placement<'a>( + state: &'a State, + layer: &Layer, + index: usize, + item: &Item, + format: Format, +) -> Option<&'a Placement> { item.keep_clear?; let scene = layer.scene.as_deref()?; + if state.collection().format_of(layer, format) != Format::Landscape { + return None; + } state.compositor.placement(scene, index, &item.source) } -/// `item`, the `index`th of `layer`, resting where the picture draws it: a -/// layer keeping clear where it was placed. -pub(super) fn placed(state: &State, layer: &Layer, index: usize, item: &Item) -> Item { - match placement(state, layer, index, item) { +/// `item`, the `index`th of `layer`, resting where `format`'s picture draws +/// it: a layer keeping clear where it was placed. +pub(super) fn placed(state: &State, layer: &Layer, index: usize, item: &Item, format: Format) -> Item { + match placement(state, layer, index, item, format) { Some(placement) => Item { rect: placement.drawn(item.rect), ..item.clone() @@ -45,11 +55,11 @@ pub(super) fn placed(state: &State, layer: &Layer, index: usize, item: &Item) -> } } -/// `rect`, a new rect for `item` from a gesture other than a move, such as a -/// resize, to save: a layer placed off where it rests keeps resting where it -/// did, so the resize leaves it free to keep clear. -pub(super) fn resized(state: &State, layer: &Layer, index: usize, item: &Item, rect: Rect) -> Rect { - match placement(state, layer, index, item) { +/// `rect`, a new rect in `format`'s picture for `item` from a gesture other +/// than a move, such as a resize, to save: a layer placed off where it rests +/// keeps resting where it did, so the resize leaves it free to keep clear. +pub(super) fn resized(state: &State, layer: &Layer, index: usize, item: &Item, rect: Rect, format: Format) -> Rect { + match placement(state, layer, index, item, format) { Some(placement) if placement.drawn(item.rect) != item.rect => Rect { x: item.rect.x, y: item.rect.y, @@ -59,24 +69,29 @@ pub(super) fn resized(state: &State, layer: &Layer, index: usize, item: &Item, r } } -/// Note that `item`, the `index`th of `layer`, was moved by hand: a layer -/// keeping clear stays where it was put until the content under it changes. -pub(super) fn hold(state: &mut State, layer: &Layer, index: usize, item: &Item) { +/// Note that `item`, the `index`th of `layer`, was moved by hand in +/// `format`'s picture: a layer keeping clear stays where it was put until the +/// content under it changes. A move in the portrait picture, which keeping +/// clear does not place, holds nothing. +pub(super) fn hold(state: &mut State, layer: &Layer, index: usize, item: &Item, format: Format) { if item.keep_clear.is_some() && let Some(scene) = layer.scene.as_deref() + && state.collection().format_of(layer, format) == Format::Landscape { state.compositor.hold(scene, index, &item.source); } } -/// Note the layer a gesture on the picture has in hand, if any: a layer -/// keeping clear does not choose a place while in hand. -pub(super) fn grab(state: &mut State, picked: Option<&Picked>) { +/// Note the layer a gesture on `format`'s picture has in hand, if any: a +/// layer keeping clear does not choose a place while in hand in the +/// landscape picture, the one it places. +pub(super) fn grab(state: &mut State, picked: Option<&Picked>, format: Format) { let collection = state.collection(); let grabbed = picked.and_then(|picked| { let scene = picked.layer.scene.clone()?; let item = picked.item(collection)?; item.keep_clear?; + (collection.format_of(&picked.layer, format) == Format::Landscape).then_some(())?; Some((scene, picked.index, item.source.clone())) }); let grabbed = grabbed @@ -85,13 +100,16 @@ pub(super) fn grab(state: &mut State, picked: Option<&Picked>) { state.compositor.grab(grabbed, kio::clock::now()); } -/// `staged`, the collection as the picture draws it, with each layer -/// keeping clear where it was placed, so a pointer over the picture finds -/// what it sees. -pub(super) fn drawn<'a>(state: &State, staged: Cow<'a, Collection>) -> Cow<'a, Collection> { +/// `staged`, the collection as `format`'s picture draws it, with each +/// layer keeping clear where it was placed, so a pointer over the picture +/// finds what it sees. +pub(super) fn drawn<'a>(state: &State, staged: Cow<'a, Collection>, format: Format) -> Cow<'a, Collection> { let collection = state.collection(); let mut drawn = staged; for (number, scene) in collection.scenes.iter().enumerate() { + if collection.format_of(&Layer::scene(&scene.name), format) != Format::Landscape { + continue; + } for (index, item) in scene.items.iter().enumerate() { if item.keep_clear.is_none() { continue; diff --git a/rs/moq-studio/src/ui/layout.rs b/rs/moq-studio/src/ui/layout.rs index be810095..b0b2066a 100644 --- a/rs/moq-studio/src/ui/layout.rs +++ b/rs/moq-studio/src/ui/layout.rs @@ -259,6 +259,28 @@ pub fn fit(rect: Rect, shape: f32) -> Rect { Rect::from_center_size(rect.center(), size.max(Vec2::ZERO)) } +/// The landscape and portrait pictures of a show going out in both, of +/// `shapes` (width over height), side by side `gap` apart: as tall as fits +/// in `rect`, and centered in it. +pub fn pair(rect: Rect, shapes: [f32; 2], gap: f32) -> [Rect; 2] { + let [landscape, portrait] = shapes; + let tall = rect + .height() + .min((rect.width() - gap).max(0.0) / (landscape + portrait)) + .max(0.0); + let wide = tall * (landscape + portrait) + gap; + let min = pos2(rect.center().x - wide / 2.0, rect.center().y - tall / 2.0); + let first = Rect::from_min_size(min, vec2(tall * landscape, tall)); + let second = Rect::from_min_size(pos2(first.right() + gap, min.y), vec2(tall * portrait, tall)); + [first, second] +} + +/// The shape, width over height, of the pair [`pair`] places, with a gap +/// `gap` of their height between them. +pub fn pair_shape(shapes: [f32; 2], gap: f32) -> f32 { + shapes[0] + shapes[1] + gap +} + fn area(rect: Rect) -> f32 { rect.width().max(0.0) * rect.height().max(0.0) } @@ -286,6 +308,23 @@ mod tests { ); } + /// Both pictures of a show going out in both fit side by side, as tall + /// as the room allows, with the gap between them. + #[test] + fn a_pair_fits_side_by_side() { + let room = Rect::from_min_size(pos2(0.0, 0.0), vec2(1000.0, 400.0)); + let [landscape, portrait] = pair(room, [LANDSCAPE, PORTRAIT], 20.0); + assert!(room.contains_rect(landscape.union(portrait))); + assert_eq!(landscape.height(), portrait.height()); + assert!((portrait.left() - landscape.right() - 20.0).abs() < 1e-3); + assert!((landscape.width() / landscape.height() - LANDSCAPE).abs() < 1e-3); + assert!((portrait.width() / portrait.height() - PORTRAIT).abs() < 1e-3); + // Too narrow for the full height, the pair is as wide as the room. + let narrow = Rect::from_min_size(pos2(0.0, 0.0), vec2(600.0, 400.0)); + let [landscape, portrait] = pair(narrow, [LANDSCAPE, PORTRAIT], 20.0); + assert!((portrait.right() - landscape.left() - 600.0).abs() < 1e-3); + } + /// For every size from the smallest up, of every shape of picture and /// count of sounds: every part is inside the window, none overlaps /// another, the picture is inside its stage and larger than every part diff --git a/rs/moq-studio/src/ui/live.rs b/rs/moq-studio/src/ui/live.rs index 887361e7..b87f946c 100644 --- a/rs/moq-studio/src/ui/live.rs +++ b/rs/moq-studio/src/ui/live.rs @@ -124,6 +124,9 @@ impl Going { /// stream for the uplink. struct Measuring { task: Task>, + /// How many pictures its ceiling was sized for, so turning a second on + /// measures again, past what both send. + encodes: u32, /// Dropping it ends the measurement. _stop: Option>, } @@ -134,16 +137,22 @@ fn named(canvas: Canvas) -> String { } /// A smaller canvas to send, if there is one worth offering: 720p from -/// anything larger, then 30 frames a second rather than more. +/// anything larger, keeping it wider than tall, or taller, as it was, then +/// 30 frames a second rather than more. fn smaller(canvas: Canvas) -> Option { let short = canvas.width.min(canvas.height); if short > 720 { let scale = |side: u32| (u64::from(side) * 720 / u64::from(short)) as u32 & !1; - return Some(Canvas { - width: scale(canvas.width), - height: scale(canvas.height), - framerate: canvas.framerate, - }); + let (width, height) = (scale(canvas.width), scale(canvas.height)); + // Rounding a nearly square canvas can make it square, which a show + // going out in both formats cannot turn. + if width.cmp(&height) == canvas.width.cmp(&canvas.height) { + return Some(Canvas { + width, + height, + framerate: canvas.framerate, + }); + } } (canvas.framerate > 30).then_some(Canvas { framerate: 30, @@ -294,7 +303,8 @@ impl Panels { return; } ctx.request_repaint_after(Duration::from_millis(100)); - let (show, program, canvas) = { + let broadcast = self.controls.stream.lock().unwrap().broadcast().to_string(); + let (show, program, canvas, both) = { let state = self.engine.state.lock().unwrap(); let path = self.engine.autosave.path(); let show = path @@ -303,7 +313,15 @@ impl Panels { .to_string_lossy() .into_owned(); let program = state.compositor.program_scene().map(str::to_string); - (show, program, state.collection().canvas) + let collection = state.collection(); + // Each relay carries both pictures, each its own broadcast. + let both = collection.portrait.then(|| { + format!( + "Both formats go to each relay: landscape at {broadcast}, portrait at {broadcast}/{}.", + moq_scene::PORTRAIT + ) + }); + (show, program, collection.canvas, both) }; // Not while going live, which keeps the sheet waiting on it, and // whose encoders a new check would compete with: once it ends. @@ -334,6 +352,10 @@ impl Panels { ui.add_space(10.0); heading(ui, "Where it goes"); let count = self.destinations(ui, &mut settings); + if let Some(both) = &both { + ui.add_space(4.0); + ui.label(egui::RichText::new(both).font(kit::ui(12.0)).color(kit::DIM)); + } ui.add_space(10.0); heading(ui, "Checks"); for (index, check) in checks.iter().enumerate() { @@ -405,8 +427,15 @@ impl Panels { /// Collect the checks that finished, and start measuring the connection /// once the encoder says what it sends. fn advance_going(&mut self) { + let encodes = match self.engine.state.lock().unwrap().collection().portrait { + true => 2, + false => 1, + }; let Some(going) = &mut self.going else { return }; going.encoder.poll(); + if going.uplink.as_ref().is_some_and(|uplink| uplink.encodes < encodes) { + going.uplink = None; + } if going.uplink.is_none() && let Task::Done(encoder) = &going.encoder { @@ -414,11 +443,15 @@ impl Panels { Ok(encoding) => { // Up to past what it sends, so a connection that carries // it says so rather than stopping at its rate. - let ceiling = encoding.total().scaled(1.0 / setup::headroom()); - measure(&self.controls, &self.runtime, ceiling) + let ceiling = encoding.total_of(encodes).scaled(1.0 / setup::headroom()); + Measuring { + encodes, + ..measure(&self.controls, &self.runtime, ceiling) + } } Err(_) => Measuring { task: Task::Done(Err("nothing encodes, so nothing would be sent".into())), + encodes, _stop: None, }, }); @@ -621,7 +654,13 @@ impl Panels { }, }); - // The encoder keeps up with the canvas. + // The encoder keeps up with the canvas: twice over, a picture in each + // format, while the show goes out in both. + let encodes = if collection.portrait { 2 } else { 1 }; + let formats = match encodes { + 1 => String::new(), + _ => " in both formats".to_string(), + }; let canvas = going.canvas; let send = smaller(canvas).map(|smaller| (format!("Send {} instead", named(smaller)), Fix::Send(smaller))); let encoding = match &going.encoder { @@ -637,11 +676,11 @@ impl Panels { level: None, fixes: Vec::new(), }, - Task::Done(Ok(encoding)) if encoding.keeps_up(canvas) => Check { + Task::Done(Ok(encoding)) if encoding.keeps_up_all(canvas, encodes) => Check { mark: Outcome::Passed, icon: Icon::Encoder, name: "Encoder", - said: format!("Keeps up with {}", named(canvas)), + said: format!("Keeps up with {}{formats}", named(canvas)), level: None, fixes: Vec::new(), }, @@ -650,8 +689,8 @@ impl Panels { icon: Icon::Encoder, name: "Encoder", said: format!( - "The encoder takes {} ms a frame, too slow for {}. Viewers would see it stutter.", - encoding.frame.as_millis(), + "The encoder takes {} ms a frame, too slow for {}{formats}. Viewers would see it stutter.", + (encoding.frame * encodes).as_millis(), named(canvas) ), level: None, @@ -696,14 +735,18 @@ impl Panels { fixes: Vec::new(), }, (Some(Task::Done(Ok(carries))), Some(encoding)) => { - let needs = encoding.total(); + let needs = encoding.total_of(encodes); + let sending = match encodes { + 1 => "the program", + _ => "the program in both formats", + }; match carries.scaled(setup::headroom()) >= needs { true => Check { mark: Outcome::Passed, icon: Icon::Uplink, name: "Connection", said: format!( - "Carries {}, and the program needs {}", + "Carries {}, and {sending} needs {}", health::rate(*carries), health::rate(needs) ), @@ -715,7 +758,7 @@ impl Panels { icon: Icon::Uplink, name: "Connection", said: format!( - "The connection carries {} and the program needs {}. Viewers would see it stall.", + "The connection carries {} and {sending} needs {}. Viewers would see it stall.", health::rate(*carries), health::rate(needs) ), @@ -1107,6 +1150,7 @@ fn measure(controls: &Controls, runtime: &tokio::runtime::Handle, ceiling: Rate) }); Measuring { task: task.unwrap_or_else(|err| Task::Done(Err(format!("cannot start the measurement: {err}")))), + encodes: 1, _stop: Some(stop), } } @@ -1387,3 +1431,23 @@ pub(super) fn health_word(engine: &Engine) -> (String, Color32) { .flatten(); (counted.unwrap_or_else(|| word.to_string()), dot) } + +#[cfg(test)] +mod tests { + use super::*; + + /// 720p from anything larger, keeping its shape, unless rounding would + /// make a canvas wider than tall square; then a lower frame rate. + #[test] + fn a_smaller_canvas_keeps_its_orientation() { + let canvas = |width, height, framerate| Canvas { + width, + height, + framerate, + }; + assert_eq!(smaller(canvas(1920, 1080, 60)), Some(canvas(1280, 720, 60))); + assert_eq!(smaller(canvas(1080, 1920, 30)), Some(canvas(720, 1280, 30))); + assert_eq!(smaller(canvas(1082, 1080, 60)), Some(canvas(1082, 1080, 30))); + assert_eq!(smaller(canvas(1082, 1080, 30)), None); + } +} diff --git a/rs/moq-studio/src/ui/settings.rs b/rs/moq-studio/src/ui/settings.rs index 546e1f8f..5558dec5 100644 --- a/rs/moq-studio/src/ui/settings.rs +++ b/rs/moq-studio/src/ui/settings.rs @@ -257,9 +257,10 @@ impl Panels { /// Settings, Show: the format, the canvas size and frame rate by name or /// by number, and each scene's own size. fn show_pane(&mut self, ui: &mut Ui) { - let (current, scenes) = { + let (current, scenes, both) = { let state = self.engine.state.lock().unwrap(); - (state.collection().canvas, state.collection().scenes.clone()) + let collection = state.collection(); + (collection.canvas, collection.scenes.clone(), collection.portrait) }; let shown = self.canvas.unwrap_or(current); let mut wanted = None; @@ -268,7 +269,7 @@ impl Panels { row( ui, "Format", - "One format for the whole show. The layout of this window follows it.", + "The shape of the show's picture. The layout of this window follows it.", |ui| { ui.spacing_mut().item_spacing.x = CARD_GAP; // Right to left, so Landscape comes first. @@ -287,6 +288,26 @@ impl Panels { }, ); + let mut on = both; + row( + ui, + "Both at once", + "Also send a portrait picture, for phones, as its own broadcast beside the landscape one. Each \ + scene gets a portrait arrangement of its layers, and the machine encodes both.", + |ui| { + // As the scene file requires: only a canvas wider than tall turns. + let wide = current.width > current.height; + let toggle = ui + .add_enabled(wide, kit::Toggle::new(&mut on).label("Landscape and portrait")) + .on_disabled_hover_text( + "Only a canvas wider than tall turns: a portrait show turns to landscape first", + ); + if toggle.changed() { + self.set_both(on); + } + }, + ); + let named = named_size(current).filter(|_| self.canvas.is_none()); let mut size = named; let sizes: Vec<(Option, &str)> = SIZES @@ -294,7 +315,7 @@ impl Panels { .map(|(short, name)| (Some(*short), *name)) .chain([(None, "Custom")]) .collect(); - row(ui, "Canvas size", &upload(current), |ui| { + row(ui, "Canvas size", &upload(current, both), |ui| { // Custom lit, as for a size no name has, still opens its editor. if ui.add(kit::Segmented::new(&mut size, &sizes).reclick()).changed() { match size { @@ -529,20 +550,41 @@ impl Panels { "Set the canvas to {}x{} at {} fps", canvas.width, canvas.height, canvas.framerate ); - edit(&mut state, label, None, |draft| draft.set_canvas(canvas)) + // Only a landscape canvas goes out in portrait too. + let turning_off = canvas.height >= canvas.width && state.collection().portrait; + let result = edit(&mut state, label, None, |draft| { + if canvas.height >= canvas.width { + draft.set_portrait(false)?; + } + draft.set_canvas(canvas) + }); + // As turning Both off does, it removes the layers shown only in + // portrait, which may move the selected layer's index. + if result.is_ok() && turning_off { + self.deselect_item(); + } + result } Restart::Scene(name, size) => match state.collection().scenes.iter().position(|scene| scene.name == name) { Some(index) => { - let scene = state.collection().scenes[index].clone(); + // The program's size, while the show goes out in portrait + // too, follows the program's, so the scene keeps its + // portrait picture and can still go on program. + let program = state.collection().canvas; + let size = size.filter(|size| { + !state.collection().portrait || (size.width, size.height) != (program.width, program.height) + }); let label = match size { Some(size) => format!("Size scene `{name}` {}x{}", size.width, size.height), None => format!("Size scene `{name}` as the program"), }; - let result = edit(&mut state, label, None, |draft| { - draft.set_scene(index, Scene { size, ..scene }) - }); + let result = edit(&mut state, label, None, |draft| draft.set_size(index, size)); if result.is_ok() { self.sizes.remove(&name); + // Given its own size, its layers shown only in portrait go. + if size.is_some() { + self.deselect_item(); + } } result } @@ -553,6 +595,24 @@ impl Panels { self.check(result); } + /// Send a portrait picture too, or stop, as an edit undo takes back. + fn set_both(&mut self, on: bool) { + let engine = self.engine.clone(); + let mut state = engine.state.lock().unwrap(); + let label = match on { + true => "Go out in landscape and portrait", + false => "Go out in landscape only", + }; + let result = edit(&mut state, label.to_string(), None, |draft| draft.set_portrait(on)); + drop(state); + // Turned off, the layers shown only in portrait go, which may move + // the selected layer's index. + if result.is_ok() && !on { + self.deselect_item(); + } + self.check(result); + } + /// Every edit, newest first; pick one to undo or redo to just after it. pub(super) fn history(&mut self, ui: &mut Ui) { let engine = self.engine.clone(); @@ -1703,13 +1763,24 @@ fn sized(canvas: Canvas, short: u32) -> Canvas { } /// What `canvas` costs in upload: the most its renditions and the audio -/// send, as each rendition's ceiling allows. -fn upload(canvas: Canvas) -> String { - let video: u64 = crate::ladder::sizes(canvas.size()) - .into_iter() - .map(|size| crate::ladder::ceiling(size, canvas.framerate()).as_bps()) +/// send, as each rendition's ceiling allows, twice over, a picture in each +/// format, if `both`. +fn upload(canvas: Canvas, both: bool) -> String { + // The portrait picture is a broadcast of its own, audio included. + let pictures = std::iter::once(canvas).chain(both.then(|| canvas.turned())); + let total: u64 = pictures + .map(|canvas| { + let ladder = crate::ladder::sizes(canvas.size()).into_iter(); + let video: u64 = ladder + .map(|size| crate::ladder::ceiling(size, canvas.framerate()).as_bps()) + .sum(); + video + crate::engine::AUDIO_BITRATE.as_bps() + }) .sum(); - let total = video + crate::engine::AUDIO_BITRATE.as_bps(); + let both = match both { + true => " in both formats", + false => "", + }; let name = named_size(canvas) .and_then(|short| SIZES.iter().find(|(side, _)| *side == short)) .map_or_else( @@ -1717,7 +1788,7 @@ fn upload(canvas: Canvas) -> String { |(_, name)| name.to_string(), ); format!( - "Bigger needs a faster encoder and more upload: {name} at {} wants up to about {:.1} Mb/s.", + "Bigger needs a faster encoder and more upload: {name} at {}{both} wants up to about {:.1} Mb/s.", canvas.framerate, total as f64 / 1e6 ) @@ -1735,6 +1806,25 @@ mod tests { } } + /// The portrait picture is a broadcast of its own, as costly as the + /// landscape one. + #[test] + fn both_formats_want_twice_the_upload() { + let canvas = canvas(1920, 1080, 30); + let wanted = |both| { + let said = upload(canvas, both); + let number = said.rsplit("about ").next().unwrap().trim_end_matches(" Mb/s."); + number.parse::().unwrap() + }; + assert!(upload(canvas, true).contains("1080p at 30 in both formats")); + assert!( + (wanted(true) - 2.0 * wanted(false)).abs() <= 0.1, + "{} {}", + wanted(true), + wanted(false) + ); + } + #[test] fn a_named_size_keeps_the_format_and_rate() { let landscape = canvas(1920, 1080, 60); diff --git a/rs/moq-studio/src/ui/shell.rs b/rs/moq-studio/src/ui/shell.rs index 93f49681..ce4e279c 100644 --- a/rs/moq-studio/src/ui/shell.rs +++ b/rs/moq-studio/src/ui/shell.rs @@ -33,16 +33,40 @@ pub(super) struct Snapshot { pub(super) takes: u64, /// The program's shape, width over height. pub(super) shape: f32, + /// The portrait picture's size as drawn, while the show goes out in both. + pub(super) portrait: Option, /// The scene cued, or open off air. pub(super) cued: Option, /// Whether Studio mode is on: a scene's pad cues it. pub(super) studio: bool, } +impl Snapshot { + /// A pad's thumbnail's shape, width over height: the program's, or, while + /// the show goes out in both, the pair's. + pub(super) fn pad_shape(&self) -> f32 { + match self.portrait { + Some(portrait) => super::layout::pair_shape([self.shape, shape(portrait)], PAIR_GAP), + None => self.shape, + } + } +} + +/// The shape of a picture of `canvas`, width over height. +pub(super) fn shape(canvas: Canvas) -> f32 { + canvas.width as f32 / canvas.height.max(1) as f32 +} + +/// The gap between a pair's pictures in a thumbnail, as a share of their +/// height. +const PAIR_GAP: f32 = 0.04; + /// A scene as its pad shows it. struct ScenePad { name: String, canvas: Canvas, + /// Whether it goes out in portrait too, so its pad shows both pictures. + dual: bool, /// Why it cannot go on program, if it cannot. refused: Option, /// Whether a layer it shows went dark or stalled. @@ -119,8 +143,13 @@ impl Panels { if let Some(render) = frame.wgpu_render_state() { self.program.follow(render, state.compositor.layered()); self.preview.follow(render, state.compositor.preview()); + let portrait = state.compositor.portrait(); + Registered::follow_while(&mut self.portrait, render, portrait.map(|(texture, _)| texture)); + let preview = state.compositor.portrait_preview(); + Registered::follow_while(&mut self.portrait_preview, render, preview.map(|(texture, _)| texture)); self.follow_thumbnails(render, &state.compositor); } + self.portrait_drawn = state.compositor.portrait().map(|(_, canvas)| canvas); // Redraw at the canvas rate, so the picture and the pads are live. let framerate = state.compositor.canvas().framerate; ctx.request_repaint_after(Duration::from_secs(1) / framerate); @@ -186,6 +215,7 @@ impl Panels { .map(|scene| ScenePad { name: scene.name.clone(), canvas: scene.canvas(collection.canvas), + dual: collection.dual(&Layer::scene(&scene.name)), refused: collection.refuses(&scene.name), dark: super::dark::draws_lost(collection, &scene.name, |name| state.lost(name, now).is_some()), }) @@ -209,6 +239,9 @@ impl Panels { let cued = cued.map(|name| Cued { refused: collection.refuses(&name), canvas: collection.canvas(Some(name.as_str())).unwrap_or(collection.canvas), + portrait: collection + .portrait_canvas() + .filter(|_| collection.dual(&Layer::scene(&name))), off_air: self.editing.as_ref() == Some(&name), live: state.seen(&name), transition: transition_words(&collection.transition), @@ -230,7 +263,8 @@ impl Panels { sounds, stream, takes: state.takes, - shape: self.program_drawn.width as f32 / self.program_drawn.height.max(1) as f32, + shape: shape(self.program_drawn), + portrait: self.portrait_drawn.filter(|_| self.portrait.is_some()), cued, studio: self.controls.studio(), } @@ -241,7 +275,13 @@ impl Panels { /// messages over them. pub(super) fn window(&mut self, ui: &mut Ui, fx: &mut Fx, snapshot: &Snapshot) { let window = ui.max_rect(); - let (size, shape, sounds) = (window.size(), snapshot.shape, snapshot.sounds.len()); + let (size, mut shape, sounds) = (window.size(), snapshot.shape, snapshot.sounds.len()); + // LIVE shows both pictures of a show going out in both, and room for + // what is said under them, at about the height they are drawn. + if let (Page::Live, Some(portrait)) = (self.page, snapshot.portrait) { + let pair = super::layout::pair_shape([shape, self::shape(portrait)], super::formats::GAP / 600.0); + shape = pair * 600.0 / (600.0 + super::formats::UNDER); + } let layout = match self.page { // MIX puts a live picture in the sound's slot. Page::Mix => Layout::monitor(size, shape, sounds), @@ -273,7 +313,10 @@ impl Panels { match self.page { Page::Live => { kit::wash(ui.painter(), stage); - let picture = self.on_air(ui, Some(fx), layout.picture).rect; + let picture = match snapshot.portrait { + Some(portrait) => self.both_on_air(ui, fx, layout, portrait), + None => self.on_air(ui, Some(fx), layout.picture).rect, + }; if let Some(cued) = &snapshot.cued { self.next(ui, fx, cued, picture, Corner::TopRight); } @@ -288,8 +331,20 @@ impl Panels { /// The live picture in its tally frame, filling `picture`. pub(super) fn on_air(&mut self, ui: &mut Ui, fx: Option<&mut Fx>, picture: Rect) -> egui::Response { - let id = self.program.id; - let canvas = self.program_drawn; + let (id, canvas) = (self.program.id, self.program_drawn); + self.framed(ui, fx, picture, id, canvas) + } + + /// The picture `id` of `canvas` in the program's tally frame, filling + /// `picture`. + fn framed( + &self, + ui: &mut Ui, + fx: Option<&mut Fx>, + picture: Rect, + id: egui::TextureId, + canvas: Canvas, + ) -> egui::Response { let builder = UiBuilder::new().max_rect(picture.expand(10.0)); let frame = self.program_frame(); ui.scope_builder(builder, |ui| { @@ -298,6 +353,52 @@ impl Panels { .inner } + /// Both live pictures of a show going out in both, each in its tally + /// frame, side by side in the stage, with each one's format, size, and + /// what it sends under it, and where the selected layer is shown between + /// them. The landscape picture's rect, which the NEXT card docks in. + fn both_on_air(&mut self, ui: &mut Ui, fx: &mut Fx, layout: &Layout, portrait: Canvas) -> Rect { + let room = layout.stage.shrink(20.0); + let room = room.with_max_y(room.bottom() - super::formats::UNDER); + let [wide, tall] = super::layout::pair(room, [shape(self.program_drawn), shape(portrait)], super::formats::GAP); + self.on_air(ui, Some(fx), wide); + let Some(id) = self.portrait.as_ref().map(|registered| registered.id) else { + return wide; + }; + self.framed(ui, Some(fx), tall, id, portrait); + let sent = self.sent(); + let under = |picture: Rect| { + Rect::from_min_size( + pos2(picture.left(), picture.bottom() + 18.0), + vec2(picture.width(), 24.0), + ) + }; + let free = super::formats::format_label(ui, under(wide), Format::Landscape, self.program_drawn, sent[0]); + super::formats::format_label(ui, under(tall), Format::Portrait, portrait, sent[1]); + // Centered under the landscape picture, clear of what is said there. + self.shown_in(ui, free, Some(wide.center().x)); + wide + } + + /// What the landscape and portrait broadcasts' video sent over the last + /// interval, while on air. + fn sent(&self) -> [Option; 2] { + let report = self.engine.health.lock().unwrap(); + let video = |name: Option<&str>| { + let broadcast = report + .broadcasts + .iter() + .find(|broadcast| broadcast.name.as_deref() == name)?; + let tracks = broadcast.tracks.iter().filter(|track| track.label.starts_with("video")); + let sent: Vec = tracks + .filter_map(|track| track.sent) + .map(|sent| sent.as_bps()) + .collect(); + (!sent.is_empty()).then(|| moq_net::bandwidth::Rate::from_bps(sent.into_iter().sum())) + }; + [video(None), video(Some(moq_scene::PORTRAIT))] + } + /// The title bar: the show's menu, undo and redo, the pages, the health /// dot, record, the live control, Settings, and the window's buttons. Its /// empty room moves the window. @@ -666,10 +767,12 @@ impl Panels { return; } // As wide as fits the strip's height, at the show's shape. - let tall = kit::Pad::height(PAD, snapshot.shape); + let shape = snapshot.pad_shape(); + let pad = PAD * shape.max(1.0) / snapshot.shape.max(1.0); + let tall = kit::Pad::height(pad, shape); let width = match tall > room.height() { - true => ((room.height() - 33.0) * snapshot.shape + 8.0).floor().max(48.0), - false => PAD, + true => ((room.height() - 33.0) * shape + 8.0).floor().max(48.0), + false => pad, }; egui::ScrollArea::horizontal().id_salt("scenes").show(ui, |ui| { ui.horizontal_centered(|ui| { @@ -768,7 +871,7 @@ impl Panels { if index > snapshot.scenes.len() { return None; } - let size = vec2(width, kit::Pad::height(width, snapshot.shape)); + let size = vec2(width, kit::Pad::height(width, snapshot.pad_shape())); let (rect, response) = ui.allocate_exact_size(size, Sense::click()); let response = kit::tip(response, "Add a scene", None); let ink = if response.hovered() { kit::TEXT } else { kit::FAINT }; @@ -780,7 +883,7 @@ impl Panels { let cued = snapshot.cued.as_ref().is_some_and(|cued| cued.name == pad.name); let live = self.engine.on_air().live().is_some(); let (tally, tag) = tally(on_air, live, cued); - let response = self.pad(ui, pad, tally, tag, width, snapshot.shape); + let response = self.pad(ui, pad, tally, tag, width, snapshot); let click = match snapshot.studio { true => PadAction::Cue(pad.name.clone()), false => PadAction::Take(pad.name.clone()), @@ -859,7 +962,8 @@ impl Panels { } /// A scene's pad, `width` wide, with its thumbnail once the compositor - /// has drawn it, which it draws while the pad is on screen. + /// has drawn it, which it draws while the pad is on screen, and its + /// portrait one beside it while the show goes out in both. fn pad( &mut self, ui: &mut Ui, @@ -867,14 +971,25 @@ impl Panels { tally: Tally, tag: Option<&str>, width: f32, - shape: f32, + snapshot: &Snapshot, ) -> egui::Response { + let (shape, portrait) = (snapshot.pad_shape(), snapshot.portrait.filter(|_| pad.dual)); let thumbnail = Thumbnail::Scene(pad.name.clone()); let size = vec2(pad.canvas.width as f32, pad.canvas.height as f32); let picture = self .thumbnails .get(&thumbnail) .map(|registered| egui::load::SizedTexture::new(registered.id, size)); + // The portrait picture beside it, for a scene going out in both. + let second = portrait.map(|canvas| { + let thumbnail = Thumbnail::Portrait(pad.name.clone()); + let size = vec2(canvas.width as f32, canvas.height as f32); + let picture = self + .thumbnails + .get(&thumbnail) + .map(|registered| egui::load::SizedTexture::new(registered.id, size)); + (thumbnail, picture, self::shape(canvas)) + }); let key = self.key(&Action::Scene(pad.name.clone()), ui.ctx()); let fault = self.fault(&Spot::Pad(pad.name.clone())); let widget = kit::Pad::new(&pad.name) @@ -887,9 +1002,14 @@ impl Panels { .fault(fault) .width(width) .shape(shape); + let widget = match &second { + Some((_, picture, shape)) => widget.portrait(*picture, *shape), + None => widget, + }; let response = ui.add(widget); if ui.is_rect_visible(response.rect) { self.thumbnails_shown.insert(thumbnail); + self.thumbnails_shown.extend(second.map(|(thumbnail, ..)| thumbnail)); } response } @@ -1358,20 +1478,30 @@ impl Panels { // Every published scene, encoding only on air. let encoding = stats.published(); - let published: Vec<(String, String)> = { + // And the portrait picture, while the show goes out in both, which + // carries the program's audio. + let published: Vec<(String, String, bool)> = { let state = self.engine.state.lock().unwrap(); let collection = state.collection(); let scenes = collection.scenes.iter().filter(|scene| scene.published); + let portrait = collection.portrait.then(|| { + let audio = collection.program_audio().unwrap_or_default().to_string(); + (moq_scene::PORTRAIT.to_string(), audio, true) + }); scenes .map(|scene| { let audio = collection.published_audio(&scene.name).unwrap_or_default().to_string(); - (scene.name.clone(), audio) + (scene.name.clone(), audio, false) }) + .chain(portrait) .collect() }; - for (name, audio) in published { + for (name, audio, portrait) in published { ui.separator(); - ui.strong(format!("Scene `{name}`")); + match portrait { + true => ui.strong("The portrait picture"), + false => ui.strong(format!("Scene `{name}`")), + }; if let Some(live) = on_air.live() { ui.label(format!("Broadcast {}/{name}, with audio scene `{audio}`", live.path)); } diff --git a/rs/moq-studio/src/ui/slots.rs b/rs/moq-studio/src/ui/slots.rs index 69fad7d1..68c0c2a1 100644 --- a/rs/moq-studio/src/ui/slots.rs +++ b/rs/moq-studio/src/ui/slots.rs @@ -7,63 +7,67 @@ use moq_scene::{Arrangement, Crop, Slot}; use super::layers::section; use super::*; -/// `layer`'s layers where they are drawn, in order: those of a scene in -/// slots where the last render drew them, hidden while their slot is +/// `layer`'s layers where they are drawn in `format`'s picture, in order: +/// each at its place there, hidden unless shown there, and those of a scene +/// in slots where the last render drew them, hidden while their slot is /// closed. -pub(super) fn placed(state: &State, layer: &Layer) -> Vec { - let items = state.collection().items(layer).unwrap_or_default(); +pub(super) fn placed(state: &State, layer: &Layer, format: Format) -> Vec { + let collection = state.collection(); + let format = collection.format_of(layer, format); + let items = collection.items(layer).unwrap_or_default(); // A layer set free that keeps clear, where it was placed. - let free = |index, item| super::keep::placed(state, layer, index, item); - let arranged = layer.scene.as_deref().and_then(|scene| state.compositor.placed(scene)); + let items = (items.iter().enumerate()) + .map(|(index, item)| super::keep::placed(state, layer, index, &collection.portrayed(item, format), format)); + let arranged = (layer.scene.as_deref()).and_then(|scene| state.compositor.placed(format, scene)); // One frame behind an edit, until the next render arranges it. let Some(arranged) = arranged.filter(|arranged| arranged.len() == items.len()) else { - return items - .iter() - .enumerate() - .map(|(index, item)| free(index, item)) - .collect(); + return items.collect(); }; - let placed = items.iter().zip(arranged).enumerate(); - let placed = placed.map(|(index, (item, arranged))| match item.slot { + let placed = items.zip(arranged).map(|(item, arranged)| match item.slot { Some(_) => Item { rect: arranged.rect, visible: item.visible && arranged.visible, - ..item.clone() + ..item }, - None => free(index, item), + None => item, }); placed.collect() } -/// The collection with the scene called `scene` as the last render drew -/// it, cover crops and a glide under way included, so a page in a slot -/// takes input where it is drawn. -pub(super) fn staged<'a>(state: &'a State, scene: Option<&str>) -> std::borrow::Cow<'a, Collection> { +/// The collection as `format`'s picture draws it, with the scene called +/// `scene` as the last render drew it there, cover crops and a glide under +/// way included, so a page in a slot takes input where it is drawn. +pub(super) fn staged<'a>(state: &'a State, scene: Option<&str>, format: Format) -> std::borrow::Cow<'a, Collection> { let collection = state.collection(); + let format = scene.map_or(format, |scene| collection.format_of(&Layer::scene(scene), format)); let found = scene.and_then(|scene| { let index = collection.scenes.iter().position(|found| found.name == scene)?; - let arranged = state.compositor.placed(scene)?; + let arranged = state.compositor.placed(format, scene)?; (arranged.len() == collection.scenes[index].items.len()).then_some((index, arranged)) }); - let Some((index, arranged)) = found else { - return std::borrow::Cow::Borrowed(collection); - }; - let mut staged = collection.clone(); - staged.scenes[index].items = arranged.to_vec(); - std::borrow::Cow::Owned(staged) + let mut staged = collection.formatted(format); + if let Some((index, arranged)) = found { + staged.to_mut().scenes[index].items = arranged.to_vec(); + } + staged } -/// Where the layer at `index` of `layer` is drawn and what it shows of its -/// picture, cover crop included, so setting it free keeps how it looks; its -/// own place for a layer set free or not yet arranged. -pub(super) fn drawn(state: &State, layer: &Layer, index: usize) -> Option<(Rect, Crop)> { +/// Where the layer at `index` of `layer` is drawn in `format`'s picture and +/// what it shows of its picture, cover crop included, so setting it free +/// keeps how it looks; its own place there for a layer set free or not yet +/// arranged. +pub(super) fn drawn(state: &State, layer: &Layer, index: usize, format: Format) -> Option<(Rect, Crop)> { let collection = state.collection(); - let item = collection.items(layer)?.get(index)?; - let canvas = collection.canvas(layer.scene.as_deref()).unwrap_or(collection.canvas); + let format = collection.format_of(layer, format); + let item = &collection.portrayed(collection.items(layer)?.get(index)?, format); + let canvas = match collection.dual(layer) { + true => collection.canvas_of(format), + false => collection.canvas(layer.scene.as_deref()).unwrap_or(collection.canvas), + }; let arranged = layer .scene .as_deref() - .and_then(|scene| state.compositor.placed(scene)) + .and_then(|scene| state.compositor.placed(format, scene)) .filter(|arranged| arranged.len() == collection.items(layer).map_or(0, <[Item]>::len)) .and_then(|arranged| arranged.get(index)) .filter(|arranged| item.slot.is_some() && !arranged.rect.is_empty()); @@ -176,26 +180,39 @@ pub(super) fn assign(state: &mut State, dragged: &Picked, target: &Picked) -> an let taking = Item { slot: held.slot, rect: Rect::default(), + portrait: None, keep_clear: None, ..moved.clone() }; - // Set free where `moved` is drawn, as a layer keeping clear may be off - // where it is saved, cropped to its shape rather than stretched. - let (rect, crop) = match moved.slot { - Some(_) => (Rect::default(), held.crop), + // Set free where `moved` is drawn in each picture, as a layer keeping + // clear may be off where it is saved, cropped to its shape there rather + // than stretched. + let free = |format: Format| { + let resting = super::keep::placed(state, &dragged.layer, dragged.index, &moved, format); + let Placement { rect, .. } = state.collection().placement(&resting, format); + let (from, crop) = drawn(state, &target.layer, target.index, format).unwrap_or((rect, held.crop)); + let shape = |rect: &Rect| rect.width / rect.height; + let crop = crop.then(&Crop::covering(shape(&from), shape(&rect))); + Placement { rect, crop } + }; + let leaving = match moved.slot { + Some(_) => Item { + slot: moved.slot, + rect: Rect::default(), + ..held + }, None => { - let to = super::keep::placed(state, &dragged.layer, dragged.index, &moved).rect; - let (from, crop) = drawn(state, &target.layer, target.index).unwrap_or((to, held.crop)); - let shape = |rect: &Rect| rect.width / rect.height; - (to, crop.then(&Crop::covering(shape(&from), shape(&to)))) + let dual = state.collection().dual(&target.layer); + let landscape = free(Format::Landscape); + Item { + slot: None, + rect: landscape.rect, + crop: landscape.crop, + portrait: (dual && held.shown_in(Format::Portrait)).then(|| free(Format::Portrait)), + ..held + } } }; - let leaving = Item { - slot: moved.slot, - rect, - crop, - ..held - }; edit(state, label, None, |draft| { draft.set_item(&target.layer, target.index, taking)?; draft.set_item(&dragged.layer, dragged.index, leaving) @@ -217,9 +234,18 @@ pub(super) fn set_slot(state: &mut State, picked: &Picked, slot: Option) - return Ok(()); } let name = item.source.clone(); + // Set free where it is drawn in each picture. + let drawn = |format| drawn(state, &picked.layer, picked.index, format).unwrap_or((item.rect, item.crop)); let (rect, crop) = match slot { Some(_) => (Rect::default(), item.crop), - None => drawn(state, &picked.layer, picked.index).unwrap_or((item.rect, item.crop)), + None => drawn(Format::Landscape), + }; + let portrait = match slot { + None if collection.dual(&picked.layer) && item.shown_in(Format::Portrait) => { + let (rect, crop) = drawn(Format::Portrait); + Some(Placement { rect, crop }) + } + _ => None, }; let main = match slot { Some(Slot::Main) => collection @@ -250,6 +276,7 @@ pub(super) fn set_slot(state: &mut State, picked: &Picked, slot: Option) - slot, rect, crop, + portrait, // In a slot, the arrangement places it. keep_clear: item.keep_clear.filter(|_| slot.is_none()), ..item @@ -268,10 +295,11 @@ fn arrangement_name(arrangement: Arrangement) -> &'static str { } } -/// A button showing `arrangement`'s slots, a main item and three tiles, lit -/// while `on`. -fn arrangement_button(ui: &mut Ui, arrangement: Arrangement, on: bool) -> egui::Response { - let size = egui::vec2(58.0, 33.0); +/// A button showing `arrangement`'s slots on `canvas`, a main item and +/// three tiles, lit while `on`. +fn arrangement_button(ui: &mut Ui, arrangement: Arrangement, canvas: Canvas, on: bool) -> egui::Response { + // As tall as a landscape one, so a row of either lines up. + let size = egui::vec2(33.0 * canvas.width as f32 / canvas.height as f32, 33.0); let (rect, response) = ui.allocate_exact_size(size + egui::vec2(8.0, 8.0), egui::Sense::click()); response.widget_info(|| { egui::WidgetInfo::selected( @@ -293,7 +321,6 @@ fn arrangement_button(ui: &mut Ui, arrangement: Arrangement, on: bool) -> egui:: false => egui::Stroke::new(1.0, kit::LINE), }; painter.rect(rect, 6.0, fill, stroke, egui::StrokeKind::Inside); - let canvas = Canvas::default(); let picture = egui::Rect::from_center_size(rect.center(), size); let scale = size.x / canvas.width as f32; let (main, tiles) = arrangement.places(canvas, true, 3); @@ -324,12 +351,17 @@ enum Asked { impl Panels { /// The Slots section of the scene inspector for the scene at `index`: - /// how it arranges its slots, each slot and the role or source in it, - /// and a slot to add for the main picture or the camera; the result of - /// what it asked for, if it asked. + /// how it arranges its slots in the picture BUILD edits, each slot and + /// the role or source in it, and a slot to add for the main picture or + /// the camera; the result of what it asked for, if it asked. pub(super) fn slots(&mut self, ui: &mut Ui, state: &mut State, index: usize) -> Option> { let collection = state.collection(); let scene = collection.scenes[index].clone(); + let format = collection.format_of(&Layer::scene(&scene.name), self.format); + let (arranged, canvas) = match format { + Format::Landscape => (scene.arrangement, Canvas::default()), + Format::Portrait => (scene.portrait.arrangement, Canvas::default().turned()), + }; let room = self.engine.room(); let room = room.as_ref().map(|room| room.path()); let mut asked = None; @@ -341,10 +373,17 @@ impl Panels { .color(kit::DIM) .font(kit::ui(11.0)), ); + if collection.dual(&Layer::scene(&scene.name)) { + ui.label( + egui::RichText::new(format!("Arranged in {}", format.name().to_lowercase())) + .color(kit::DIM) + .font(kit::ui(11.0)), + ); + } ui.horizontal_wrapped(|ui| { ui.spacing_mut().item_spacing.x = 6.0; for arrangement in Arrangement::ALL { - if arrangement_button(ui, arrangement, scene.arrangement == arrangement).clicked() { + if arrangement_button(ui, arrangement, canvas, arranged == arrangement).clicked() { asked = Some(Asked::Arrange(arrangement)); } } @@ -421,16 +460,21 @@ impl Panels { let asked = asked?; Some(match asked { - Asked::Arrange(arrangement) => edit( - state, - format!( + Asked::Arrange(arrangement) => { + let label = format!( "Arrange `{}` as {}", scene.name, arrangement_name(arrangement).to_lowercase() - ), - None, - |draft| draft.set_scene(index, Scene { arrangement, ..scene }), - ), + ); + let scene = match format { + Format::Landscape => Scene { arrangement, ..scene }, + Format::Portrait => Scene { + portrait: moq_scene::Portrait { arrangement }, + ..scene + }, + }; + edit(state, label, None, |draft| draft.set_scene(index, scene)) + } Asked::Fill(role, source) => { let label = match &source { Some(source) => format!("Fill the role `{role}` with `{source}`"),