diff --git a/README.md b/README.md index e8f74353..9630eb8e 100644 --- a/README.md +++ b/README.md @@ -217,7 +217,7 @@ The **strip** holds: and cuts at once to the scene marked safe, if one is. Other remote feeds and media stay on. Mark one from its pad's menu, BUILD's inspector with no layer selected, or the scene graph. The scissors beside it save a replay clip (see - [Replay](#replay)). + [Replay](#replay)), and the flag drops a marker (see [Markers](#markers)). - A row for each sound the stream carries, with its meter and mute, and the **STREAM** meter of the audio scene on program, with **Listen** (the headphones) to hear it (see [Audio scenes](#audio-scenes)). @@ -227,7 +227,7 @@ The **strip** holds: | LIVE | Watch the program while on air, and TAKE or AUTO a cued scene. | | BUILD | Edit the scene on program, or the one open off air, shown in the middle, with its layers and what can be added on the left and the inspector on the right (see [BUILD](#build)). | | MIX | Ride each sound's fader and mute it, open its strip for its pan and delay, and send each sound and audio scene into audio scenes, from dots under the strips or as a matrix under **Routes**. Add, rename, reorder, and remove audio scenes, put one on the stream, and listen to one on the output you pick (see [Audio scenes](#audio-scenes)). The program shows small in the strip's place for the sound. | -| REVIEW | Pick a stretch of the last 30 seconds of the program or a source from a strip of its pictures, and replay it in a replay source, slowed down if you like, or save it as a clip (see [Replay](#replay)). Beside it, this show's recordings and clips as cards with their length and size, one repaired after a crash marked so, and **Open the folder**. The program's card is the live picture. | +| REVIEW | Pick a stretch of the last 30 seconds of the program or a source from a strip of its pictures, and replay it in a replay source, slowed down if you like, or save it as a clip (see [Replay](#replay)). Beside it, this session's markers, with a clip around each (see [Markers](#markers)), over this show's recordings and clips as cards with their length and size, one repaired after a crash marked so, and **Open the folder**. The program's card is the live picture. | The layout follows the program's shape and the window's size, giving the picture the most room it can. A landscape program sits above the strip; a @@ -475,7 +475,7 @@ setting costs. Escape or a click outside closes it. | Show | Pick the format, landscape or portrait, which swaps the canvas's sides; **Both at once**, which sends a portrait picture beside a landscape one (see [Both formats at once](#both-formats-at-once)); the canvas size as 720p, 1080p, 1440p, or 4K, with the upload each wants, for both pictures while Both at once is on, or **Custom** for exact numbers; the frame rate as 30, 60, or **Custom**; and, under **Scene sizes**, each scene's own size (see [Published scenes](#published-scenes)). Saved in the scene file, and undone like any edit. | | Stream | Set the relay, its token, the failover relay, the broadcast's name, and on demand (see [Stream settings](#stream-settings)). | | Recording | Change or open the recordings folder, and pick the picture and the audio scene a recording holds, and whether each source records a file of its own too (see [Recording](#recording)). | -| Hotkeys | Bind keys, by what they are for: going live, ending the stream, the panic key, and recording; Studio mode, TAKE, and AUTO; each scene; each sound's mute; and replay and clips (see [Hotkeys](#hotkeys)). | +| Hotkeys | Bind keys, by what they are for: going live, ending the stream, the panic key, and recording; Studio mode, TAKE, and AUTO; each scene; each sound's mute; and replay, clips, and markers (see [Hotkeys](#hotkeys)). | | Guests | Set the room guests join, its key, the join page, how long a link lasts, and whether guests appear on their own (see [Guests](#guests)). | | Advanced | Publish scenes as broadcasts of their own and sources independently, turn the virtual camera of the program or of each published scene on, pick how the window's [effects](#effects) move, and open the [scene graph](#scene-graph). | @@ -1718,12 +1718,12 @@ Quitting first lets the recording's encoders publish what they still hold, and the program's file keeps it. A file nothing reached before the recording stopped is removed. -While a file is written, a marker `.mp4.unfinished` sits beside it. A -crash, a power loss, or a write that failed leaves the marker, and the next +While a file is written, a sidecar `.mp4.unfinished` sits beside it. A +crash, a power loss, or a write that failed leaves the sidecar, and the next launch repairs that file without asking: it writes `.repaired.mp4` beside it, numbered if a file already has that name, a regular MP4 of everything the file holds up to its last complete fragment, leaves the -original untouched, and removes the marker, so the file is repaired +original untouched, and removes the sidecar, so the file is repaired only once. The repair runs in the background and appears at the top of the Health window and in the log, and REVIEW marks its card **Repaired after a crash**. A file another studio is still recording into the same directory is left alone. @@ -1748,7 +1748,7 @@ a WebGL page runs slower. Settings, Hotkeys binds a key to the **Panic** key, TAKE and AUTO for the cued scene, turning [Studio mode](#off-air-and-studio-mode) on or off, -**Replay**, **Save clip**, starting or stopping a recording, **Go live**, +**Replay**, **Save clip**, **Drop a marker**, starting or stopping a recording, **Go live**, **End stream**, each scene, which its key takes, or cues in Studio mode, as its pad does (the tab says which), and each sound, which its key mutes or unmutes, as its strip's button does. The [command palette](#command-palette) @@ -1862,6 +1862,29 @@ no past to replay: a page publishes no track of its own, so replay the program to show it. A replay source on program encodes each replay as its own track while it plays, like a camera. +## Markers + +Drop a marker on a moment as the show runs, on air or off, with the flag +beside the panic key or a key bound to **Drop a marker** (see +[Hotkeys](#hotkeys)). REVIEW lists this session's markers, the newest first, +each numbered in the order dropped, with how long ago it was. Type a name in +its row, or remove it. Each marker still within the last 30 seconds shows on +the strip as a numbered flag. + +The scissors in a marker's row save a clip around it, as **Save clip** does, +of what REVIEW has picked to replay: from 10 seconds before the marker to 5 +after, or up to now while those 5 have yet to pass. The same keyframe limits +apply. A clip reaches back only 30 seconds, so a marker older than that has +no clip; its scissors say so. + +The markers are kept beside the recordings, not in the scene file: +`show 2026-11-18T12-00-00Z.markers.toml` for `show.toml`, named for when the +studio opened the show, lists each marker's number, the UTC moment it was +dropped, and its name. It is written again on every change and removed once +no marker is left, in the background, so a slow disk never holds up a key. A +marker the folder refuses is kept for the session anyway, and REVIEW says why +until a write succeeds. A session holds at most 255 markers. + ## A capture gone dark A camera, display, or window can go black while its capture still runs: a diff --git a/quest/m2/markers.md b/quest/m2/markers.md index 73648af3..9af7a483 100644 --- a/quest/m2/markers.md +++ b/quest/m2/markers.md @@ -1,19 +1,34 @@ -# [M] Markers and clips +# [S] Recordings carry markers as chapters ## Goal -A key or a button drops a marker on the show as it runs. The REVIEW page -shows markers on its strip and in a list, a clip is cut around one in a -click, and a recording carries its markers as chapters. +A recording carries the session's markers that fall inside it as chapters, +each titled by the marker's name or number, which a player such as mpv or +VLC lists and seeks to. ## Plan Planned 2026-11-09 (maintainer, /quest-plan), and held for later: ✅ m2 -(rejected: m1). +(rejected: m1). Markers, their list and strip on REVIEW, and a clip around +one shipped in [moq-dev/studio#209](https://github.com/moq-dev/studio/pull/209); +the chapters wait on moq-mux, since the studio cannot place a moment in a +finished file: where the movie starts on the program clock is known only +inside moq-mux's export, and on moq's `main` its `Finish` is opaque. -- A marker is a time on the program clock and an optional name. Markers of - a session are kept beside the recordings, not in the scene file. -- A clip around a marker uses Save clip's copy without re-encoding, so the - same keyframe limits apply. -- Chapters are written when a recording becomes a regular MP4. -- Marker is a hotkey action. +- Chapters are written when a recording becomes a regular MP4: as it stops + or the studio quits (`finish` in `rs/moq-studio/src/record.rs`), and as a + launch repairs one a crash left (`rs/moq-studio/src/record/repair.rs`). +- A marker's moment is its `at` (`rs/moq-studio/src/markers.rs`) on the + program clock (`Program::clock`'s `capture` in `engine.rs`), which every + recording shares; moq-mux places it in the movie. +- Recommended: each marker's program-clock reading goes in the session's + markers file, and each recording's `.unfinished` sidecar names that file, + so a repair at the next launch finds the markers of the session that + recorded it. The recorder's own `Marker` type, that sidecar, becomes + `Sidecar` then, since the two meet in `record.rs`. +- Recommended: a chapter starts at each marker, with none before the first, + since a marker is a moment, not a span. + +## Required + +- [moq-mux's MP4 chapters pinned](/quest/m2/pin-mp4-chapters.md) - brings moq-mux's chapters on a finished MP4 into the `moq` submodule diff --git a/quest/m2/pin-mp4-chapters.md b/quest/m2/pin-mp4-chapters.md new file mode 100644 index 00000000..9bede7b1 --- /dev/null +++ b/quest/m2/pin-mp4-chapters.md @@ -0,0 +1,31 @@ +# [XS] moq-mux's MP4 chapters are pinned + +## Goal + +The `moq` submodule includes a way to give a recording that moq-mux's +`container::mp4` finishes, by `Export::finish` or `repair`, chapters: each a +moment on the recorded broadcast's timeline and a title, written into the +regular `moov`. + +## Plan + +moq-mux is where this belongs: it alone knows where the movie starts on the +broadcast's timeline, and on moq's `main` its `Finish` is opaque, so the +studio cannot edit the `moov` itself. mp4-atom 0.17, which moq-mux already +uses, encodes the Nero `chpl` box that ffmpeg, mpv, and VLC read. + +Once the wait clears, advance it with `just upgrade-moq` in its own PR, +fixing whatever the new API breaks here. + +## Required + +- [The studio catches up to moq's main](/quest/m0/pin-moq-tip.md) - absorbs the tip's other breaks, so this pin moves only by chapters + +## Waiting + +- moq-dev/moq: moq-mux's regular MP4 finish takes chapters. The change has + no quest in moq's tree yet; it was found by + [moq-dev/studio#209](https://github.com/moq-dev/studio/pull/209) and is + listed in its Follow-ups for `/quest-plan` there. Check: + `rs/moq-mux/src/container/mp4.rs` on moq's `main` writes chapters into the + finished `moov`. diff --git a/quest/m2/try-markers.md b/quest/m2/try-markers.md new file mode 100644 index 00000000..7aa7cb30 --- /dev/null +++ b/quest/m2/try-markers.md @@ -0,0 +1,30 @@ +# [XS] Try markers + +## Goal + +The maintainer tries markers on their own machine and either signs them off +or files what they find as quests. + +## Plan + +Markers are on `main` (`cargo run -p moq-studio --features capture`). What +to try, on a show going live to a relay: + +- **Feel:** dropping a marker with the flag beside the panic key and with a + bound key while a game has focus, then naming it, clipping around it, and + removing it on REVIEW. +- **Look:** the numbered flags on REVIEW's strip and the list over the + recordings, at a normal window size. +- **Fit:** whether 10 seconds before a marker and 5 after is the clip you + want, and whether a marker older than 30 seconds should clip at all before + [clips from a recording](/quest/m3/marker-clips.md) land. + +An agent then turns each finding into a quest, or an edit to the quest that +owns that part, and deletes this one. + +## Waiting + +- Maintainer: try markers as the Plan lists and report what to change, or + that nothing needs to. An agent cannot judge feel or taste; a kittest run + and `just shots` already cover the clicks and the layout. Check: a comment + or quest records the result. diff --git a/quest/m3/marker-clips.md b/quest/m3/marker-clips.md new file mode 100644 index 00000000..e7f7af5d --- /dev/null +++ b/quest/m3/marker-clips.md @@ -0,0 +1,23 @@ +# [M] A clip around an older marker, from the recording + +## Goal + +A marker older than the last 30 seconds still gets a clip around it, in one +click on REVIEW, cut from the recording that holds it without re-encoding, +as a clip from the recent past is. A marker no recording holds says so. + +## Plan + +A clip from the recent past copies the program's track, which keeps only 30 +seconds (`rs/moq-studio/src/replay.rs`), so today a marker past that has no +clip. A recording holds the rest: import it through moq-mux, from the +keyframe before the marker's span, and export the span as Save clip does, so +the same keyframe limits apply. A finished recording is a regular MP4, which +moq-mux's regular MP4 import reads; one still being written is a fragmented +MP4, which its fMP4 import already reads. + +Here in m3 because the import it needs is pinned in m3. + +## Required + +- [MP4 import pinned](/quest/m3/pin-mp4-import.md) - brings moq-mux's regular MP4 import into the `moq` submodule diff --git a/rs/moq-studio/assets/icons/flag.svg b/rs/moq-studio/assets/icons/flag.svg new file mode 100644 index 00000000..eed422df --- /dev/null +++ b/rs/moq-studio/assets/icons/flag.svg @@ -0,0 +1,14 @@ + + + + diff --git a/rs/moq-studio/src/controls.rs b/rs/moq-studio/src/controls.rs index b46e9d45..4a7c5d54 100644 --- a/rs/moq-studio/src/controls.rs +++ b/rs/moq-studio/src/controls.rs @@ -1,16 +1,19 @@ //! What the operator does from a button, a hotkey, or the command palette: -//! take a scene, mute a sound, panic, replay, save a clip, record, and go live -//! and end the stream, each registered once in the table here. Hotkeys fire -//! from outside the window's frames, so a key pressed while a game has focus -//! acts at once, even while the window is hidden and draws nothing. +//! take a scene, mute a sound, panic, replay, save a clip, drop a marker, +//! record, and go live and end the stream, each registered once in the table +//! here. Hotkeys fire from outside the window's frames, so a key pressed while +//! a game has focus acts at once, even while the window is hidden and draws +//! nothing. use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::{Arc, Mutex}; use anyhow::Context; +use kio::clock::Instant; use moq_scene::Source; use crate::engine::{Engine, OnAir, State}; +use crate::markers::Markers; use crate::replay; use crate::stream::{Stream, Switched}; @@ -31,6 +34,8 @@ pub enum Action { Replay, /// Save the stretch of the recent past REVIEW picks as a clip. SaveClip, + /// Mark this moment of the show. + Marker, /// Start recording, or stop the recording under way. Record, /// Cut to the safe scene and mute every microphone and guest. @@ -85,7 +90,7 @@ struct Fixed { /// The actions every show has, each registered once, in the order the /// Hotkeys pane and the palette list them within their group. Each scene and /// sound adds its own: see [`Action::listed`]. -const FIXED: [Fixed; 9] = [ +const FIXED: [Fixed; 10] = [ Fixed { action: Action::GoLive, group: Group::Live, @@ -158,6 +163,14 @@ const FIXED: [Fixed; 9] = [ detail: None, words: &["highlight"], }, + Fixed { + action: Action::Marker, + group: Group::Review, + id: "marker", + name: "Drop a marker", + detail: None, + words: &["mark", "flag", "bookmark", "moment"], + }, ]; impl Action { @@ -307,6 +320,8 @@ pub struct Controls { probes: Arc>, /// REVIEW's replay picks, which a replay or clip from a hotkey takes too. pub replay: Mutex, + /// The moments marked this session. + pub markers: Markers, /// Whether the next recording gives each source a file of its own too. pub record_sources: AtomicBool, /// Whether Studio mode is on: a scene's pad or key cues it rather than @@ -345,6 +360,7 @@ impl Default for Replay { impl Controls { pub fn new(engine: Arc, stream: Stream, runtime: tokio::runtime::Handle) -> Self { engine.set_guests_appear(stream.guests_appear()); + let markers = Markers::new(engine.recorder.show()); Self { engine, runtime, @@ -358,6 +374,7 @@ impl Controls { joined: Mutex::default(), probes: Arc::new(tokio::sync::watch::Sender::new(Probes::default())), replay: Mutex::default(), + markers, record_sources: AtomicBool::new(false), studio: AtomicBool::new(false), fired: Arc::default(), @@ -408,8 +425,12 @@ impl Controls { let rooms = self.engine.rooms(); self.engine.state.lock().unwrap().panic(&rooms) } - Action::Replay => self.replay(true, false), - Action::SaveClip => self.replay(false, false), + Action::Replay => self.replay(true, false, None), + Action::SaveClip => self.replay(false, false, None), + Action::Marker => { + let number = self.markers.mark(&self.engine.recorder.dir(), Instant::now())?; + Ok(format!("dropped marker {number}")) + } Action::Record => { let recorder = &self.engine.recorder; match recorder.status().filter(|status| !status.done) { @@ -447,6 +468,7 @@ impl Controls { Action::Mute(name) if !collection.sources.contains_key(name) => { Some(format!("there is no source `{name}`")) } + Action::Marker => self.markers.refused(), // Whichever is picked, any replay source plays it. Action::Replay => replay_source(collection, None).err().map(|err| err.to_string()), Action::Record => self @@ -724,21 +746,38 @@ impl Controls { /// Do `action` for a hotkey, keeping how it went for the status bar. pub fn press(self: &Arc, action: &Action) { match action { - Action::Replay => drop(self.replay(true, true)), - Action::SaveClip => drop(self.replay(false, true)), + Action::Replay => drop(self.replay(true, true, None)), + Action::SaveClip => drop(self.replay(false, true, None)), action => self.report(action, self.fire(action).map_err(|err| format!("{err:#}"))), } } - /// Play REVIEW's pick in its replay source, or save it as a - /// clip. Copied now, without waiting on anything, so the span ends here and - /// replays apply in the order they were asked for. How it went shows in - /// REVIEW, and in the status bar too when a hotkey `pressed` it, - /// since the panel may be hidden. - fn replay(&self, play: bool, pressed: bool) -> anyhow::Result { + /// Save a clip of what REVIEW picks around marker `number`, as Save clip + /// does, saying how it went in REVIEW. + pub fn clip_marker(&self, number: u32) -> anyhow::Result { + let marker = self + .markers + .get(number) + .with_context(|| format!("there is no marker {number}"))?; + let stretch = replay::Stretch::around(marker.at, Instant::now()); + let stretch = stretch.with_context(|| format!("cannot clip around {}", marker.title()))?; + self.replay(false, false, Some(stretch)) + } + + /// Play REVIEW's pick in its replay source, or save it as a clip, over + /// the stretch `around` a marker, if given. Copied now, without waiting + /// on anything, so the span ends here and replays apply in the order they + /// were asked for. How it went shows in REVIEW, and in the status bar too + /// when a hotkey `pressed` it, since the panel may be hidden. + fn replay(&self, play: bool, pressed: bool, around: Option) -> anyhow::Result { let replay = self.replay.lock().unwrap(); let action = if play { Action::Replay } else { Action::SaveClip }; - let taken = self.take(&replay, play, pressed); + let stretch = match (around, pressed) { + (Some(around), _) => around, + (None, true) => replay.stretch.last(), + (None, false) => replay.stretch, + }; + let taken = self.take(&replay, play, stretch); let message = taken .as_ref() .map(|(message, _)| message.clone()) @@ -762,23 +801,20 @@ impl Controls { Ok(message) } - /// What a replay or clip does at once, and the save a clip still runs. - /// One `pressed` from outside REVIEW's own buttons, by a hotkey or the - /// shell's clip button, takes the last stretch as long as the one picked. + /// What a replay or clip of `stretch` does at once, and the save a clip + /// still runs. One pressed from outside REVIEW's own buttons, by a hotkey + /// or the shell's clip button, takes the last stretch as long as the one + /// picked. fn take( &self, replay: &Replay, play: bool, - pressed: bool, + stretch: replay::Stretch, ) -> anyhow::Result<(String, Option> + use<>>)> { let into = match play { true => Some(self.replay_source(replay.into.as_deref())?), false => None, }; - let stretch = match pressed { - true => replay.stretch.last(), - false => replay.stretch, - }; let (ago, length) = stretch.span(kio::clock::Instant::now())?; let take = replay::take(&self.engine, &replay.of, ago, length, into.is_some())?; let of = replay.of.clone(); @@ -880,6 +916,7 @@ mod tests { | Action::StudioMode | Action::Replay | Action::SaveClip + | Action::Marker | Action::Record | Action::Panic | Action::GoLive diff --git a/rs/moq-studio/src/engine.rs b/rs/moq-studio/src/engine.rs index 6282b1b0..43d4f4c7 100644 --- a/rs/moq-studio/src/engine.rs +++ b/rs/moq-studio/src/engine.rs @@ -5725,8 +5725,9 @@ mod tests { .unwrap(); let held = view.held.expect("sampled").expect("the program's past"); assert!(held.video); - assert!(!view.pictures.is_empty(), "a keyframe of the past"); - for picture in &view.pictures { + let pictures = view.pictures.expect("the past's pictures"); + assert!(!pictures.is_empty(), "a keyframe of the past"); + for picture in &pictures { assert_eq!(picture.size[0], 480, "decoded small"); assert!(held.oldest <= picture.at && picture.at <= kio::clock::Instant::now()); let middle = (picture.size[1] / 2 * picture.size[0] + picture.size[0] / 2) * 4; @@ -5735,7 +5736,7 @@ mod tests { }; assert!(r > 200 && g < 60 && b < 60, "red, not {r} {g} {b}"); } - let times: Vec<_> = view.pictures.iter().map(|picture| picture.at).collect(); + let times: Vec<_> = pictures.iter().map(|picture| picture.at).collect(); assert!(times.is_sorted(), "oldest first"); // And its sound, a packet at a time: the program's mix, silent here. let sound = view.sound.expect("the program's sound"); diff --git a/rs/moq-studio/src/kit/icon.rs b/rs/moq-studio/src/kit/icon.rs index 307e2dae..6bc4792f 100644 --- a/rs/moq-studio/src/kit/icon.rs +++ b/rs/moq-studio/src/kit/icon.rs @@ -79,12 +79,14 @@ pub enum Icon { Info, /// Opens the command palette. Search, + /// A moment marked in the show. + Marker, } impl Icon { /// Every icon, for the gallery. #[cfg(test)] - pub const ALL: [Self; 66] = [ + pub const ALL: [Self; 67] = [ Self::Undo, Self::Redo, Self::Settings, @@ -151,6 +153,7 @@ impl Icon { Self::Detach, Self::Info, Self::Search, + Self::Marker, ]; fn source(self) -> ImageSource<'static> { @@ -221,6 +224,7 @@ impl Icon { Self::Detach => include_image!("../../assets/icons/external-link.svg"), Self::Info => include_image!("../../assets/icons/info.svg"), Self::Search => include_image!("../../assets/icons/search.svg"), + Self::Marker => include_image!("../../assets/icons/flag.svg"), } } diff --git a/rs/moq-studio/src/main.rs b/rs/moq-studio/src/main.rs index ded1df68..ca04196b 100644 --- a/rs/moq-studio/src/main.rs +++ b/rs/moq-studio/src/main.rs @@ -27,6 +27,7 @@ mod interact; mod kit; mod ladder; mod library; +mod markers; mod mix; mod past; mod pointer; diff --git a/rs/moq-studio/src/markers.rs b/rs/moq-studio/src/markers.rs new file mode 100644 index 00000000..6280d0d2 --- /dev/null +++ b/rs/moq-studio/src/markers.rs @@ -0,0 +1,440 @@ +//! The moments the operator marks as the show runs, by a key or the flag +//! beside the panic key: REVIEW shows them on its strip and in a list, and +//! cuts a clip around one. A session's markers are kept beside the +//! recordings, in ` .markers.toml`, named after when the studio +//! opened the show, not in the scene file. + +use std::io::Write as _; +use std::path::{Path, PathBuf}; +use std::sync::{Arc, Condvar, Mutex}; + +use anyhow::Context; +use kio::clock::{Instant, Wall}; + +/// The most markers a session holds, which bounds what a key held down, or a +/// remote firing actions, can make the studio keep. +pub const MAX: usize = 255; + +/// The longest name a marker takes, in characters. +pub const MAX_NAME: usize = 60; + +/// A moment marked in the show. +#[derive(Clone, Debug, PartialEq)] +pub struct Marker { + /// Counted from 1 in the order they were dropped and never reused, so a + /// marker keeps its number when another is removed. + pub number: u32, + /// When it was dropped, on the clock that stamps the program and every + /// source's past. + pub at: Instant, + /// When it was dropped, by the wall clock, as the file says it. + pub wall: Wall, + pub name: Option, +} + +impl Marker { + /// What the list and the strip call it: its name, or its number. + pub fn title(&self) -> String { + match &self.name { + Some(name) => name.clone(), + None => format!("Marker {}", self.number), + } + } +} + +/// This session's markers. +pub struct Markers { + /// The file's name in the recordings folder. + file: String, + list: Mutex, + writer: Writer, +} + +#[derive(Default)] +struct List { + markers: Vec, + /// The number the last marker took. + last: u32, +} + +/// The file's shape. +#[derive(serde::Serialize, serde::Deserialize)] +struct Saved { + #[serde(default)] + marker: Vec, +} + +#[derive(serde::Serialize, serde::Deserialize)] +struct SavedMarker { + number: u32, + /// UTC, to the millisecond, as RFC 3339. + at: String, + #[serde(skip_serializing_if = "Option::is_none")] + name: Option, +} + +impl Markers { + /// No markers yet, for the show `show` opened now. + pub fn new(show: &str) -> Self { + // UTC, as a recording's name is stamped, since the studio does not read + // the time zone; no colons, which Windows refuses in a file name. + let stamp = humantime::format_rfc3339_seconds(kio::clock::wall().into()) + .to_string() + .replace(':', "-"); + Self { + file: format!("{show} {stamp}.markers.toml"), + list: Mutex::default(), + writer: Writer::default(), + } + } + + /// Where they are kept in the recordings folder `dir`. + pub fn path(&self, dir: &Path) -> PathBuf { + dir.join(&self.file) + } + + /// Each marker, the oldest first. + pub fn list(&self) -> Vec { + self.list.lock().unwrap().markers.clone() + } + + /// Why another marker cannot be dropped, if it cannot. + pub fn refused(&self) -> Option { + (self.list.lock().unwrap().markers.len() >= MAX).then(full) + } + + /// Why the last write of the file failed, until one succeeds. The + /// markers are kept for this session whatever the folder does, so their + /// clips can still be cut. + pub fn failure(&self) -> Option { + self.writer.shared.0.lock().unwrap().failure.clone() + } + + /// Mark `at`, saving the markers into `dir` in the background, and return + /// its number. + pub fn mark(&self, dir: &Path, at: Instant) -> anyhow::Result { + let mut list = self.list.lock().unwrap(); + anyhow::ensure!(list.markers.len() < MAX, "{}", full()); + list.last += 1; + let number = list.last; + list.markers.push(Marker { + number, + at, + wall: kio::clock::wall(), + name: None, + }); + self.save(dir, &list); + Ok(number) + } + + /// Name marker `number` `name`, or no name when it is blank, saving the + /// markers into `dir` in the background. + pub fn rename(&self, dir: &Path, number: u32, name: &str) -> anyhow::Result<()> { + let name = name.trim(); + anyhow::ensure!( + name.chars().count() <= MAX_NAME, + "a marker's name holds at most {MAX_NAME} characters" + ); + let mut list = self.list.lock().unwrap(); + let marker = list + .markers + .iter_mut() + .find(|marker| marker.number == number) + .with_context(|| format!("there is no marker {number}"))?; + marker.name = (!name.is_empty()).then(|| name.to_string()); + self.save(dir, &list); + Ok(()) + } + + /// Remove marker `number`, saving the markers into `dir` in the + /// background. + pub fn remove(&self, dir: &Path, number: u32) -> anyhow::Result<()> { + let mut list = self.list.lock().unwrap(); + let before = list.markers.len(); + list.markers.retain(|marker| marker.number != number); + anyhow::ensure!(list.markers.len() < before, "there is no marker {number}"); + self.save(dir, &list); + Ok(()) + } + + /// Marker `number`, if it is still there. + pub fn get(&self, number: u32) -> Option { + let list = self.list.lock().unwrap(); + list.markers.iter().find(|marker| marker.number == number).cloned() + } + + /// Queue `list` to be written over the file in `dir`, or the file's + /// removal once the list is empty. Called under the list's lock, so the + /// writes queue in the order of the changes. + fn save(&self, dir: &Path, list: &List) { + let text = (!list.markers.is_empty()).then(|| { + let saved = Saved { + marker: list + .markers + .iter() + .map(|marker| SavedMarker { + number: marker.number, + at: humantime::format_rfc3339_millis(marker.wall.into()).to_string(), + name: marker.name.clone(), + }) + .collect(), + }; + let text = toml::to_string(&saved).expect("numbers and strings serialize"); + format!("# The moments marked while the show ran, beside its recordings.\n{text}") + }); + self.writer.queue(Write { + dir: dir.to_path_buf(), + path: self.path(dir), + text, + }); + } + + /// Wait until every write queued so far has been tried. + #[cfg(test)] + fn saved(&self) { + let (queue, changed) = &*self.writer.shared; + let queue = queue.lock().unwrap(); + drop(changed.wait_while(queue, |queue| queue.tried < queue.asked).unwrap()); + } +} + +/// Why a full session refuses another marker. +fn full() -> String { + format!("this session already holds {MAX} markers, the most it can; remove one on REVIEW") +} + +/// Writes the file on a thread of its own, so a slow disk never holds up the +/// window or a hotkey. Only the latest list queued is written; one that +/// changed again before its turn is skipped. The last write finishes before +/// the markers drop, as the studio closes. +#[derive(Default)] +struct Writer { + shared: Arc<(Mutex, Condvar)>, + thread: Mutex>>, +} + +#[derive(Default)] +struct Queue { + next: Option, + /// How many writes were queued, and how many of them were tried, the + /// skipped included. + asked: u64, + tried: u64, + failure: Option, + /// No more writes are coming. + closed: bool, +} + +/// One write of the file. +struct Write { + /// The recordings folder. + dir: PathBuf, + path: PathBuf, + /// What it holds, or `None` to remove it. + text: Option, +} + +impl Write { + /// Write the file whole, so a crash leaves this list or the one before. + fn run(&self) -> anyhow::Result<()> { + let Some(text) = &self.text else { + return match std::fs::remove_file(&self.path) { + Err(err) if err.kind() != std::io::ErrorKind::NotFound => { + Err(err).with_context(|| format!("cannot remove {}", self.path.display())) + } + _ => Ok(()), + }; + }; + let written = (|| { + // A default folder, such as the videos folder the platform names, may + // not exist yet. + std::fs::create_dir_all(&self.dir)?; + let mut file = tempfile::NamedTempFile::new_in(&self.dir)?; + file.write_all(text.as_bytes())?; + file.as_file().sync_all()?; + file.persist(&self.path)?; + std::io::Result::Ok(()) + })(); + written.with_context(|| format!("cannot write {}", self.path.display())) + } +} + +impl Writer { + /// Write `write` once the thread is free, in place of any still waiting. + fn queue(&self, write: Write) { + let mut thread = self.thread.lock().unwrap(); + { + let mut queue = self.shared.0.lock().unwrap(); + queue.next = Some(write); + queue.asked += 1; + } + self.shared.1.notify_all(); + if thread.is_none() { + let shared = self.shared.clone(); + let spawned = std::thread::Builder::new() + .name("markers".into()) + .spawn(move || write_queued(&shared)); + match spawned { + Ok(spawned) => *thread = Some(spawned), + Err(err) => { + let mut queue = self.shared.0.lock().unwrap(); + queue.failure = Some(format!("cannot start saving the markers: {err}")); + queue.tried = queue.asked; + drop(queue); + self.shared.1.notify_all(); + } + } + } + } +} + +impl Drop for Writer { + fn drop(&mut self) { + self.shared.0.lock().unwrap().closed = true; + self.shared.1.notify_all(); + if let Some(thread) = self.thread.get_mut().unwrap().take() { + let _ = thread.join(); + } + } +} + +/// Write each list queued in `shared`, the latest each time, until closed. +fn write_queued(shared: &(Mutex, Condvar)) { + let (queue, changed) = shared; + let mut guard = queue.lock().unwrap(); + loop { + match guard.next.take() { + Some(write) => { + let asked = guard.asked; + drop(guard); + let result = write.run(); + guard = queue.lock().unwrap(); + guard.tried = asked; + guard.failure = result.err().map(|err| { + tracing::error!(error = format!("{err:#}"), "cannot save the markers"); + format!("{err:#}") + }); + changed.notify_all(); + } + None if guard.closed => return, + None => guard = changed.wait(guard).unwrap(), + } + } +} + +#[cfg(test)] +mod tests { + use std::time::Duration; + + use super::*; + + fn read(path: &Path) -> Vec<(u32, Option)> { + let saved: Saved = toml::from_str(&std::fs::read_to_string(path).unwrap()).unwrap(); + saved + .marker + .into_iter() + .map(|marker| (marker.number, marker.name)) + .collect() + } + + /// Each change is saved whole beside the recordings, a removed marker's + /// number is never given again, and the file goes once the last marker + /// does. + #[test] + fn markers_are_kept_beside_the_recordings() { + let dir = tempfile::tempdir().unwrap(); + // The recordings folder may not exist until something is saved there. + let folder = dir.path().join("videos"); + let markers = Markers::new("show"); + let path = markers.path(&folder); + let name = path.file_name().unwrap().to_string_lossy().into_owned(); + assert!(name.starts_with("show ") && name.ends_with(".markers.toml"), "{name}"); + assert!(!name.contains(':'), "Windows refuses colons: {name}"); + + let now = Instant::now(); + assert_eq!(markers.mark(&folder, now).unwrap(), 1); + assert_eq!( + markers + .mark(&folder, now.saturating_add(Duration::from_secs(5))) + .unwrap(), + 2 + ); + markers.rename(&folder, 1, " The goal ").unwrap(); + markers.saved(); + assert_eq!(read(&path), [(1, Some("The goal".into())), (2, None)]); + assert_eq!(markers.get(1).unwrap().title(), "The goal"); + assert_eq!(markers.get(2).unwrap().title(), "Marker 2"); + + markers.rename(&folder, 1, " ").unwrap(); + assert_eq!(markers.get(1).unwrap().name, None, "a blank name is none"); + markers.remove(&folder, 1).unwrap(); + assert_eq!(markers.mark(&folder, now).unwrap(), 3, "numbers are not reused"); + markers.saved(); + assert_eq!(read(&path), [(2, None), (3, None)]); + + assert!(markers.remove(&folder, 1).is_err(), "already removed"); + assert!(markers.rename(&folder, 9, "x").is_err(), "never dropped"); + assert!(markers.rename(&folder, 2, &"x".repeat(MAX_NAME + 1)).is_err()); + markers.rename(&folder, 2, &"é".repeat(MAX_NAME)).unwrap(); + + markers.remove(&folder, 2).unwrap(); + markers.remove(&folder, 3).unwrap(); + markers.saved(); + assert!(!path.exists(), "nothing left to keep"); + assert_eq!(markers.failure(), None); + } + + /// A session holds at most [`MAX`] markers, refused past it until one is + /// removed. + #[test] + fn a_session_holds_a_bounded_number() { + let dir = tempfile::tempdir().unwrap(); + let markers = Markers::new("show"); + for _ in 0..MAX { + markers.mark(dir.path(), Instant::now()).unwrap(); + } + assert!(markers.refused().is_some()); + assert!(markers.mark(dir.path(), Instant::now()).is_err()); + assert_eq!(markers.list().len(), MAX); + markers.remove(dir.path(), 7).unwrap(); + assert!(markers.refused().is_none()); + assert_eq!(markers.mark(dir.path(), Instant::now()).unwrap(), MAX as u32 + 1); + } + + /// A folder that cannot take the file keeps the marker for this session, + /// and says why until a write succeeds. + #[test] + fn a_marker_the_folder_refuses_is_kept() { + let dir = tempfile::tempdir().unwrap(); + // A file where the folder should be. + let folder = dir.path().join("taken"); + std::fs::write(&folder, b"").unwrap(); + let markers = Markers::new("show"); + assert_eq!(markers.mark(&folder, Instant::now()).unwrap(), 1); + markers.saved(); + let failure = markers.failure().expect("the folder is a file"); + assert!(failure.contains("cannot write"), "{failure}"); + assert_eq!(markers.list().len(), 1, "kept anyway"); + markers.mark(dir.path(), Instant::now()).unwrap(); + markers.saved(); + assert_eq!(markers.failure(), None, "saved once the folder takes it"); + assert_eq!(read(&markers.path(dir.path())), [(1, None), (2, None)]); + } + + /// The last change queued is on disk once the markers drop, as the studio + /// closes. + #[test] + fn the_last_change_is_written_before_closing() { + let dir = tempfile::tempdir().unwrap(); + let markers = Markers::new("show"); + let path = markers.path(dir.path()); + for _ in 0..20 { + markers.mark(dir.path(), Instant::now()).unwrap(); + } + markers.rename(dir.path(), 20, "Last").unwrap(); + drop(markers); + let saved = read(&path); + assert_eq!(saved.len(), 20); + assert_eq!(saved[19], (20, Some("Last".into()))); + } +} diff --git a/rs/moq-studio/src/past.rs b/rs/moq-studio/src/past.rs index 930dffb1..0c28567e 100644 --- a/rs/moq-studio/src/past.rs +++ b/rs/moq-studio/src/past.rs @@ -58,12 +58,15 @@ struct Watch { pub struct View { /// What it is of, which lags a change of pick by a sample. pub of: Of, - /// The keyframe each group opens with, oldest first. - pub pictures: Vec>, + /// The keyframe each group opens with, oldest first, or why they could + /// not be read. + pub pictures: Result>, String>, /// The loudest sample of each packet of sound, from 0 to 1, oldest first, /// or why the sound could not be read. pub sound: Result, String>, - /// How far back it reaches, or why there is nothing to replay. + /// How far back it reaches, or why there is nothing to replay. Pictures + /// or sound that cannot be read for the strip leave it held, since a + /// replay or clip copies what is held without decoding it here. pub held: Option>, } @@ -71,7 +74,7 @@ impl Default for View { fn default() -> Self { Self { of: Of::Program, - pictures: Vec::new(), + pictures: Ok(Vec::new()), sound: Ok(Vec::new()), held: None, } @@ -196,18 +199,25 @@ struct Decoding { impl Decoding { fn sample(&mut self, engine: &Engine, of: &Of) -> View { - let mut view = View { - of: of.clone(), - ..View::default() - }; let now = Instant::now(); - let take = match replay::take(engine, of, Duration::ZERO, replay::WINDOW, true) { - Ok(take) => take, + match replay::take(engine, of, Duration::ZERO, replay::WINDOW, true) { + Ok(take) => self.read(of, &take, now), Err(err) => { self.reset(); - view.held = Some(Err(format!("{err:#}"))); - return view; + View { + of: of.clone(), + held: Some(Err(format!("{err:#}"))), + ..View::default() + } } + } + } + + /// What `take`, of `of` and ending `now`, holds. + fn read(&mut self, of: &Of, take: &moq_scene::replay::Take, now: Instant) -> View { + let mut view = View { + of: of.clone(), + ..View::default() }; let end = match Duration::try_from(take.end) { Ok(end) => end, @@ -235,14 +245,15 @@ impl Decoding { .map(|(at, _)| *at); } if let Some((name, config)) = &take.video { - match self.pictures(&broadcast, of, name, config, &at) { - Ok(pictures) => view.pictures = pictures, - Err(err) => { - view.held = Some(Err(format!("{err:#}"))); - return view; - } - } - let first = view.pictures.first().map(|picture| picture.at); + view.pictures = self + .pictures(&broadcast, of, name, config, &at) + .map_err(|err| format!("{err:#}")); + let first = view + .pictures + .as_ref() + .ok() + .and_then(|pictures| pictures.first()) + .map(|picture| picture.at); oldest = oldest.into_iter().chain(first).min(); } view.held = Some(Ok(Held { @@ -451,10 +462,8 @@ fn groups( mod tests { use super::*; - /// The strip holds the keyframe each group of a take opens with, at the - /// moment it was shown, and decodes each only once across samples. - #[tokio::test] - async fn the_strip_shows_each_groups_keyframe_once() { + /// Three seconds of gray, a keyframe a second, as a take holds them. + async fn gray() -> moq_scene::replay::Take { let mut broadcast = moq_net::broadcast::Info::new().produce(); let catalog = moq_mux::catalog::Producer::new(&mut broadcast, Default::default()).unwrap(); let mut config = moq_video::encode::Config::new(64, 64, moq_video::FrameRate::new(30, 1).unwrap()); @@ -477,14 +486,22 @@ mod tests { let unit = encoder.receive().unwrap().expect("an access unit per picture"); producer.publish(&unit).unwrap(); } - let (name, video) = catalog.snapshot().video.renditions.into_iter().next().unwrap(); + let video = catalog.snapshot().video.renditions.into_iter().next(); let target = moq_scene::replay::Target { broadcast: broadcast.consume(), - video: Some((name.clone(), video.clone())), + video, audio: None, end: at(89), }; - let take = moq_scene::replay::Take::new(&target, Duration::from_secs(3)).unwrap(); + moq_scene::replay::Take::new(&target, Duration::from_secs(3)).unwrap() + } + + /// The strip holds the keyframe each group of a take opens with, at the + /// moment it was shown, and decodes each only once across samples. + #[tokio::test] + async fn the_strip_shows_each_groups_keyframe_once() { + let take = gray().await; + let (name, video) = take.video.clone().unwrap(); let zero = Instant::now(); let shown = |timestamp: moq_net::Timestamp| -> Result { @@ -504,4 +521,26 @@ mod tests { let ids = |pictures: &[Arc]| pictures.iter().map(|picture| picture.id).collect::>(); assert_eq!(ids(&again), ids(&pictures), "decoded again"); } + + /// Pictures the strip cannot decode leave the past held: a replay or a + /// clip copies it without decoding it here, so neither is refused. + #[tokio::test] + async fn pictures_it_cannot_decode_leave_the_past_held() { + let mut take = gray().await; + let (_, video) = take.video.as_mut().unwrap(); + // avc1, whose parameter sets must come from a description, and this + // one is no avcC. + let hang::catalog::VideoCodec::H264(h264) = &mut video.codec else { + unreachable!("the software encoder makes H.264") + }; + h264.inline = false; + video.description = Some(bytes::Bytes::from_static(&[0xff])); + let view = Decoding::default().read(&Of::Program, &take, Instant::now()); + let held = view.held.expect("sampled").expect("the past is held"); + assert!(held.video); + let Err(why) = view.pictures else { + panic!("the strip cannot decode it") + }; + assert!(why.starts_with("cannot decode the pictures for the strip"), "{why}"); + } } diff --git a/rs/moq-studio/src/replay.rs b/rs/moq-studio/src/replay.rs index c0830e09..1b48d617 100644 --- a/rs/moq-studio/src/replay.rs +++ b/rs/moq-studio/src/replay.rs @@ -18,6 +18,11 @@ pub const WINDOW: Duration = moq_scene::replay::MAX_LENGTH; /// The shortest stretch REVIEW's handles pick. pub const MIN_STRETCH: Duration = Duration::from_millis(500); +/// How far before a marker its clip starts, and how far after it ends: a +/// marker is dropped once something has happened, so most of it is before. +pub const BEFORE: Duration = Duration::from_secs(10); +pub const AFTER: Duration = Duration::from_secs(5); + /// The speeds REVIEW offers: real time, and slow motion. pub const SPEEDS: [f32; 3] = [1.0, 0.5, 0.25]; @@ -55,6 +60,21 @@ impl Stretch { Ok((to, length)) } + /// The stretch a clip around a marker dropped `at` takes: from [`BEFORE`] + /// it to [`AFTER`] it, or to `now` when that is sooner. Refuses a marker + /// that has aged out of the [`WINDOW`]. + pub fn around(at: Instant, now: Instant) -> anyhow::Result { + anyhow::ensure!( + now.saturating_duration_since(at) < WINDOW, + "the marker is older than the last {} s, as far back as a clip reaches", + WINDOW.as_secs() + ); + Ok(Self::Between( + at.saturating_sub(BEFORE), + at.saturating_add(AFTER).min(now), + )) + } + /// The last stretch as long as this one, as a hotkey or the shell's clip /// button fires it: pressed away from REVIEW's strip, it means what just /// happened, not a moment picked there earlier, which would age out. @@ -261,6 +281,22 @@ mod stretch_tests { assert!(empty.span(now).is_err(), "empty"); } + /// A clip around a marker takes the seconds before it and after it, ends + /// now while those after have yet to happen, is cut where it leaves the + /// window, and is refused once the marker has left it. + #[test] + fn a_clip_around_a_marker_spans_both_sides() { + let secs = Duration::from_secs; + let now = Instant::now().saturating_add(secs(100)); + let ago = |seconds| now.saturating_sub(secs(seconds)); + let span = |at| Stretch::around(at, now).and_then(|stretch| stretch.span(now)); + assert_eq!(span(ago(12)).unwrap(), (secs(7), BEFORE + AFTER)); + assert_eq!(span(ago(2)).unwrap(), (Duration::ZERO, BEFORE + secs(2)), "ends now"); + assert_eq!(span(now).unwrap(), (Duration::ZERO, BEFORE), "dropped this moment"); + assert_eq!(span(ago(25)).unwrap(), (secs(20), secs(10)), "cut at the window"); + assert!(span(ago(30)).is_err(), "aged out"); + } + /// A hotkey takes the last stretch as long as the one picked, even once /// the picked one has aged out. #[test] diff --git a/rs/moq-studio/src/shots.rs b/rs/moq-studio/src/shots.rs index 5c3af1ea..d95e7662 100644 --- a/rs/moq-studio/src/shots.rs +++ b/rs/moq-studio/src/shots.rs @@ -101,10 +101,12 @@ enum View { OnAirHealth, /// REVIEW, with this show's recordings and clips, and if `picked`, the /// last 30 seconds held and a stretch picked. With `replays`, the show - /// has two replay sources to pick between. + /// has two replay sources to pick between, and with `markers`, three + /// markers were dropped, two still on the strip. Review { picked: bool, replays: bool, + markers: bool, }, /// A page in Studio mode, with `Interview` cued. Cued(Page), @@ -400,14 +402,18 @@ views! { mix_compact: Still, mixing(MixView::Strips, None, HALF); /// REVIEW off air: the program keeps no past until it goes live, which the /// strip and both actions say, beside this show's recordings and clips. - review: Still, studio(View::Review { picked: false, replays: false }); + review: Still, studio(View::Review { picked: false, replays: false, markers: false }); /// REVIEW with the last 30 seconds held: a stretch picked between the /// handles, the head inside it, the picture under the head large, and /// Replay saying why it cannot play without a replay source. - review_picked: Still, studio(View::Review { picked: true, replays: false }); + review_picked: Still, studio(View::Review { picked: true, replays: false, markers: false }); /// REVIEW on a show with two replay sources: which one Replay plays in is /// picked under the speed, in place of its note. - review_into: Still, studio(View::Review { picked: true, replays: true }); + review_into: Still, studio(View::Review { picked: true, replays: true, markers: false }); + /// REVIEW with markers: each still in the last 30 seconds a numbered + /// pennant on the strip, and the list over the recordings, the newest + /// first, one named, and the oldest too old to clip around. + review_markers: Still, studio(View::Review { picked: true, replays: false, markers: true }); /// The edit history, open under undo. history: Still, studio(View::History); /// The health dot's details off air: the encoders and their timings. @@ -693,6 +699,181 @@ fn a_click_takes_and_the_panic_key_cuts() { quit(&engine, &runtime); } +/// A name typed into a marker's row on REVIEW names that marker: when +/// another marker's name is clicked next, and when a key drops a new marker +/// above it while it is being typed. Off air, the program keeps no past, so +/// no marker can be clipped around. It draws no view. +#[test] +#[ignore = "draws on a GPU; run with `just shots`"] +fn a_marker_is_named_where_it_is_typed() { + let (_show, runtime, gpu) = lavapipe(); + let _entered = runtime.enter(); + let fixture = Path::new(env!("CARGO_MANIFEST_DIR")); + std::fs::copy(fixture.join("shots/show.toml"), "show.toml").unwrap(); + let app = start(launch(&gpu), stream(None), &runtime); + let (mut harness, started, _) = window(app, &gpu, &runtime, SIZE, library::Recent::default()); + harness.step(); + let engine = started.get().expect("the studio started").clone(); + runtime + .block_on(kio::clock::timeout(SETTLE, settle(&mut harness, &engine))) + .expect("the sources and pictures settle"); + let App::Studio(studio) = &mut harness.state_mut().app else { + panic!("the studio starts on its first frame") + }; + let controls = studio.controls(); + studio.turn(Page::Review); + let dir = engine.recorder.dir(); + for _ in 0..2 { + controls.markers.mark(&dir, kio::clock::Instant::now()).unwrap(); + } + harness.run_steps(2); + let name = |number| controls.markers.get(number).unwrap().name; + // The newest first: marker 2's name, then marker 1's. + let field = |harness: &Harness<'_, Window>, index: usize| { + harness.get_all_by_role(Role::TextInput).nth(index).unwrap().focus(); + }; + + field(&harness, 1); + harness.run_steps(1); + harness.event(egui::Event::Text("Opening".into())); + harness.run_steps(1); + field(&harness, 0); + harness.run_steps(2); + assert_eq!( + name(1).as_deref(), + Some("Opening"), + "kept when marker 2 took the typing" + ); + + harness.event(egui::Event::Text("Come".into())); + harness.run_steps(1); + // A key drops marker 3 over the row being typed in. + controls.press(&crate::controls::Action::Marker); + harness.run_steps(2); + harness.event(egui::Event::Text("back".into())); + harness.run_steps(1); + harness.key_press(egui::Key::Enter); + harness.run_steps(2); + assert_eq!(name(2).as_deref(), Some("Comeback"), "the typing stayed on marker 2"); + assert_eq!(name(3), None); + + // Until the strip's thread first samples the program, nothing refuses. + let refused = |harness: &Harness<'_, Window>| { + let scissors: Vec<_> = harness.get_all_by_label("Clip around it").collect(); + scissors.len() == 3 && scissors.iter().all(|clip| clip.accesskit_node().is_disabled()) + }; + runtime + .block_on(kio::clock::timeout(SETTLE, async { + while !refused(&harness) { + harness.step(); + kio::clock::sleep(Duration::from_millis(20)).await; + } + })) + .expect("no marker is clipped around off air"); + + drop(harness); + quit(&engine, &runtime); +} + +/// A marker dropped by the flag beside the panic key, on air, shows on +/// REVIEW, and a click on its row saves the seconds around it as a clip: +/// from [`replay::BEFORE`] it to [`replay::AFTER`] it, its video starting up +/// to a keyframe interval early. The window as a person uses it, on a real +/// engine going live to a relay. It draws no view. +#[test] +#[ignore = "draws on a GPU; run with `just shots`"] +fn a_marker_is_clipped_around() { + use crate::replay; + + let (_show, runtime, gpu) = lavapipe(); + let _entered = runtime.enter(); + let fixture = Path::new(env!("CARGO_MANIFEST_DIR")); + std::fs::copy(fixture.join("shots/show.toml"), "show.toml").unwrap(); + let relay = Relay::start(&runtime); + let app = start(launch(&gpu), stream(Some(&relay)), &runtime); + let (mut harness, started, _) = window(app, &gpu, &runtime, SIZE, library::Recent::default()); + harness.step(); + let engine = started.get().expect("the studio started").clone(); + runtime + .block_on(kio::clock::timeout(SETTLE, settle(&mut harness, &engine))) + .expect("the sources and pictures settle"); + let App::Studio(studio) = &mut harness.state_mut().app else { + panic!("the studio starts on its first frame") + }; + let controls = studio.controls(); + controls.go_live().expect("going live starts"); + runtime + .block_on(kio::clock::timeout(SETTLE, async { + while engine.on_air().live().is_none() { + kio::clock::sleep(Duration::from_millis(20)).await; + } + })) + .expect("the studio goes live"); + // On air long enough before the marker for all of its clip to be held. + let wait = |seconds: Duration| runtime.block_on(kio::clock::sleep(seconds)); + wait(replay::BEFORE + Duration::from_secs(1)); + + harness.get_by_label("Drop a marker").click(); + harness.run_steps(2); + assert_eq!(controls.markers.list().len(), 1, "the flag dropped a marker"); + // Past the seconds after it, so the clip is the whole span around it. + wait(replay::AFTER + Duration::from_secs(1)); + let App::Studio(studio) = &mut harness.state_mut().app else { + unreachable!("still the studio") + }; + studio.turn(Page::Review); + harness.run_steps(2); + harness.get_by_label("Clip around it").click(); + harness.run_steps(2); + let outcome = controls.replay.lock().unwrap().outcome.clone(); + let saved = runtime + .block_on(kio::clock::timeout(SETTLE, async { + loop { + match outcome.lock().unwrap().clone() { + Some(Ok(saved)) if saved.starts_with("saved") => return Ok(saved), + Some(Err(err)) => return Err(err), + _ => {} + } + kio::clock::sleep(Duration::from_millis(50)).await; + } + })) + .expect("the clip is written in real time"); + saved.expect("the clip saves"); + + let mut file = std::fs::File::open("show-clip-1.mp4").expect("the clip is in the recordings folder"); + let moov = std::iter::from_fn(|| mp4_atom::ReadFrom::read_from(&mut file).unwrap()) + .find_map(|atom| match atom { + mp4_atom::Any::Moov(moov) => Some(moov), + _ => None, + }) + .expect("a regular MP4"); + let length = Duration::from_secs_f64(moov.mvhd.duration as f64 / f64::from(moov.mvhd.timescale)); + let span = replay::BEFORE + replay::AFTER; + assert!( + length >= span - Duration::from_millis(200), + "{length:?} is short of {span:?}" + ); + assert!( + length <= span + Duration::from_secs(3), + "{length:?} is more than a keyframe past {span:?}" + ); + // Sound, a keyframe every packet, holds the span itself. + let sound = moov + .trak + .iter() + .find(|trak| trak.mdia.hdlr.handler == b"soun".into()) + .expect("the program's sound"); + let mdhd = &sound.mdia.mdhd; + let heard = Duration::from_secs_f64(mdhd.duration as f64 / f64::from(mdhd.timescale)); + assert!( + heard.abs_diff(span) < Duration::from_millis(100), + "{heard:?} heard of {span:?}" + ); + + drop(harness); + quit(&engine, &runtime); +} + /// The command palette as a person uses it: its key opens it from LIVE, /// what is typed finds a scene, a sound, and a setting, and Enter runs the /// one picked, taking the scene, muting the sound, and opening Settings at @@ -1729,14 +1910,21 @@ fn shoot(open: &Open, gpu: &engine::Gpu, runtime: &tokio::runtime::Runtime, show studio.bind(&Action::Scene("Main".into()), key(KeyCode::F1, "F1")); studio.bind(&Action::Cut, key(KeyCode::Numpad0, "0")); } - View::Review { picked, .. } => { + View::Review { picked, markers, .. } => { studio.turn(Page::Review); studio.seed_shelf(shelf()); + let secs = Duration::from_secs; + let now = kio::clock::Instant::now(); if *picked { - let secs = Duration::from_secs; - let now = kio::clock::Instant::now(); studio.seed_past(past(now), now, (secs(14), secs(6)), secs(9)); } + if *markers { + let dir = controls.engine.recorder.dir(); + for ago in [secs(3 * 60 + 5), secs(22), secs(4)] { + controls.markers.mark(&dir, now.saturating_sub(ago)).unwrap(); + } + controls.markers.rename(&dir, 2, "The comeback").unwrap(); + } } } let engine = started.get().expect("the studio started").clone(); @@ -2221,7 +2409,7 @@ fn past(now: kio::clock::Instant) -> crate::past::View { oldest: pictures[0].at, video: true, })), - pictures, + pictures: Ok(pictures), sound: Ok(sound), } } diff --git a/rs/moq-studio/src/ui/review.rs b/rs/moq-studio/src/ui/review.rs index c43fe858..e9a33b17 100644 --- a/rs/moq-studio/src/ui/review.rs +++ b/rs/moq-studio/src/ui/review.rs @@ -1,12 +1,13 @@ //! The REVIEW page: the last 30 seconds as a strip to pick a stretch from, //! what to replay and how fast, Replay and Save clip, and beside them this -//! show's recordings and clips. +//! session's markers and this show's recordings and clips. use egui::{Align2, CursorIcon, Rect, Sense, Stroke, UiBuilder, Vec2, pos2, vec2}; use kio::clock::Instant; use super::*; use crate::controls; +use crate::markers::{self, Marker}; use crate::past::{self, Past}; use crate::shelf::{self, Shelf}; @@ -25,6 +26,9 @@ const PIN: f32 = 8.0; /// How tall the sound under the strip's pictures is, and how wide each bar. const WAVE: f32 = 28.0; const BAR: f32 = 6.0; +/// How tall a marker's row is, and how many show before the list scrolls. +const MARKER_ROW: f32 = 32.0; +const MARKER_ROWS: usize = 4; /// The page's own state: the strip, the cards, and what a drag holds. pub(super) struct Review { @@ -38,6 +42,8 @@ pub(super) struct Review { grip: Option, /// The last outcome of a replay or clip shown, so each shows once. outcome: Option>, + /// Each marker's name as typed, by number, until its field lets go. + drafts: BTreeMap, /// The moment the page draws as now, held still for the window's views. frozen: Option, } @@ -62,6 +68,7 @@ impl Review { head: None, grip: None, outcome: None, + drafts: BTreeMap::new(), frozen: None, } } @@ -93,14 +100,15 @@ impl Review { /// The picture shown at or before `at`, or the first after it when none was. fn picture_at<'a>(&self, view: &'a past::View, at: Instant) -> Option<&'a past::Picture> { - let before = view.pictures.iter().rev().find(|picture| picture.at <= at); - before.or(view.pictures.first()).map(|picture| &**picture) + let pictures = view.pictures.as_deref().unwrap_or_default(); + let before = pictures.iter().rev().find(|picture| picture.at <= at); + before.or(pictures.first()).map(|picture| &**picture) } /// Upload each picture not on the GPU yet, and let go of those gone. fn upload(&mut self, ctx: &egui::Context, view: &past::View) { let mut kept = BTreeMap::new(); - for picture in &view.pictures { + for picture in view.pictures.as_deref().unwrap_or_default() { let texture = self.textures.remove(&picture.id).unwrap_or_else(|| { let image = egui::ColorImage::from_rgba_unmultiplied(picture.size, &picture.rgba); ctx.load_texture(format!("past-{}", picture.id), image, egui::TextureOptions::LINEAR) @@ -117,15 +125,24 @@ impl Review { } } +/// What a marker's row asks for, besides its name, which it takes itself. +enum Row { + Clip, + Remove, +} + /// Why REVIEW's picks cannot play or save now, if they cannot. struct Refusals { replay: Option, save: Option, + /// Why the pick holds no past, which a clip around a marker needs too. + held: Option, } impl Panels { - /// The page in `stage`: the strip and its actions, and the recordings - /// column on the right. `shape` is the program's, width over height. + /// The page in `stage`: the strip and its actions, and the markers over + /// the recordings in the column on the right. `shape` is the program's, + /// width over height. pub(super) fn review(&mut self, ui: &mut Ui, fx: &mut Fx, stage: Rect, shape: f32) { let side = SHELF.min(stage.width() * 0.34); let shelf = stage.with_min_x(stage.right() - side); @@ -134,13 +151,157 @@ impl Panels { shell::bar(ui, fx, shelf); ui.painter() .vline(shelf.left() + 0.5, shelf.y_range(), Stroke::new(1.0, kit::LINE)); - self.replays(ui, main.shrink2(vec2(24.0, 14.0)), shape); - self.shelf(ui, shelf.shrink2(vec2(16.0, 14.0))); + let held = self.replays(ui, main.shrink2(vec2(24.0, 14.0)), shape); + let column = shelf.shrink2(vec2(16.0, 14.0)); + let marked = self.controls.markers.list(); + let rows = match marked.len() { + // Room for the lines saying how to drop one. + 0 => 1.5, + count => count.min(MARKER_ROWS) as f32, + }; + let height = (34.0 + rows * MARKER_ROW).min(column.height() * 0.45); + let marks = column.with_max_y(column.top() + height); + ui.painter() + .hline(column.x_range(), marks.bottom() + 9.5, Stroke::new(1.0, kit::LINE)); + self.markers(ui, marks, &marked, held.as_deref()); + self.shelf(ui, column.with_min_y(marks.bottom() + 20.0)); + } + + /// This session's markers, the newest first, each with how long ago it + /// was dropped, its name to type, a clip around it, and its removal. + /// `held` says why REVIEW's pick has no past to clip, if it has none. + fn markers(&mut self, ui: &mut Ui, rect: Rect, marked: &[Marker], held: Option<&str>) { + let painter = ui.painter(); + let title = kit::caps_galley(painter, "Markers", kit::display_bold(22.0), kit::TEXT, None); + let title_height = title.size().y; + painter.galley(rect.left_top(), title, kit::TEXT); + painter.text( + pos2(rect.right(), rect.top() + title_height / 2.0), + Align2::RIGHT_CENTER, + "of this session", + kit::ui(11.5), + kit::DIM, + ); + let list = rect.with_min_y(rect.top() + title_height + 8.0); + let now = self.review.now(); + let failure = self.controls.markers.failure(); + self.review + .drafts + .retain(|number, _| marked.iter().any(|marker| marker.number == *number)); + let key = self.key(&Action::Marker, ui.ctx()); + // What a row asked for, done once the list is drawn. + let mut asked = None; + ui.scope_builder(UiBuilder::new().max_rect(list), |ui| { + if let Some(failure) = &failure { + ui.label(egui::RichText::new(capitalize(failure)).color(kit::ERR)); + } + if marked.is_empty() { + let how = match &key { + Some(key) => format!("Drop one with {key} or the flag beside the panic key as the show runs."), + None => { + "Drop one with the flag beside the panic key, or a key bound in Settings, as the show runs." + .into() + } + }; + ui.label(egui::RichText::new(how).color(kit::DIM)); + return; + } + egui::ScrollArea::vertical() + .id_salt("markers") + .auto_shrink(false) + .show(ui, |ui| { + ui.spacing_mut().item_spacing.y = 0.0; + for marker in marked.iter().rev() { + let size = vec2(ui.available_width(), MARKER_ROW); + ui.allocate_ui_with_layout(size, egui::Layout::left_to_right(egui::Align::Center), |ui| { + ui.spacing_mut().item_spacing.x = 6.0; + if let Some(row) = self.marker_row(ui, marker, now, held) { + asked = Some((marker.number, row)); + } + }); + } + }); + }); + let dir = self.engine.recorder.dir(); + match asked { + Some((number, Row::Clip)) => { + let result = self.controls.clip_marker(number); + self.report(result); + } + Some((number, Row::Remove)) => { + let result = self.controls.markers.remove(&dir, number); + self.check(result); + } + None => {} + } + } + + /// One marker's row, naming the marker once its name loses focus, and + /// returning what else it asked for. + fn marker_row(&mut self, ui: &mut Ui, marker: &Marker, now: Instant, held: Option<&str>) -> Option { + let badge = ui + .painter() + .layout_no_wrap(marker.number.to_string(), kit::mono(11.0), kit::TEXT); + let (badge_rect, _) = ui.allocate_exact_size(vec2(badge.size().x.max(14.0) + 10.0, 20.0), Sense::hover()); + ui.painter().rect_filled(badge_rect, 4, kit::P3); + ui.painter() + .galley(badge_rect.center() - badge.size() / 2.0, badge, kit::TEXT); + + let ago = now.saturating_duration_since(marker.at); + let when = ui.painter().layout_no_wrap(since(ago), kit::mono(11.0), kit::DIM); + // The name takes what the age and the two buttons leave. + let field = (ui.available_width() - when.size().x - 2.0 * (24.0 + 6.0) - 12.0).max(40.0); + let mut text = match self.review.drafts.get(&marker.number) { + Some(draft) => draft.clone(), + None => marker.name.clone().unwrap_or_default(), + }; + // Named for the marker, so focus stays on it when a newer one is + // dropped above it. + let edit = egui::TextEdit::singleline(&mut text) + .id(egui::Id::new(("marker name", marker.number))) + .hint_text(format!("Marker {}", marker.number)) + .char_limit(markers::MAX_NAME) + .desired_width(field); + let response = ui.add(edit); + let mut asked = None; + if response.gained_focus() || response.changed() { + self.review.drafts.insert(marker.number, text); + } + // Taken here, so a click on the row's buttons, or on another marker's + // name, that ends the typing keeps the name too. + if response.lost_focus() + && let Some(draft) = self.review.drafts.remove(&marker.number) + && draft.trim() != marker.name.as_deref().unwrap_or("") + { + let dir = self.engine.recorder.dir(); + let result = self.controls.markers.rename(&dir, marker.number, &draft); + self.check(result); + } + let (when_rect, _) = ui.allocate_exact_size(when.size(), Sense::hover()); + ui.painter().galley(when_rect.min, when, kit::DIM); + let refusal = clip_refusal(ago, held); + let note = refusal.clone().unwrap_or_else(|| { + format!( + "Save from {} s before it to {} s after it, as Save clip does", + replay::BEFORE.as_secs(), + replay::AFTER.as_secs() + ) + }); + let clip = IconButton::new(Icon::Clip, "Clip around it").small().note(Some(¬e)); + if ui.add_enabled(refusal.is_none(), clip).clicked() { + asked = Some(Row::Clip); + } + let remove = IconButton::new(Icon::Close, "Remove the marker").small(); + if ui.add(remove).clicked() { + asked = Some(Row::Remove); + } + asked } /// The large picture and what to replay, the strip, the speed, and the - /// two actions, top to bottom in `rect`. - fn replays(&mut self, ui: &mut Ui, rect: Rect, shape: f32) { + /// two actions, top to bottom in `rect`, returning why the pick holds no + /// past, if it holds none. + fn replays(&mut self, ui: &mut Ui, rect: Rect, shape: f32) -> Option { let replayable = self.replayable(); let controls = self.controls.clone(); let mut replay = controls.replay.lock().unwrap(); @@ -195,6 +356,7 @@ impl Panels { if let Some(action) = fired { let _ = controls.fire(&action); } + refusals.held } /// Each source with a past to replay, and whether it is sound alone. @@ -250,7 +412,10 @@ impl Panels { None => "Reading the last 30 seconds".to_string(), Some(Err(why)) => capitalize(why), Some(Ok(held)) if !held.video => "Sound only: nothing to show".to_string(), - Some(Ok(_)) => "No pictures held".to_string(), + Some(Ok(_)) => match &view.pictures { + Err(why) => capitalize(why), + Ok(_) => "No pictures held".to_string(), + }, }; let galley = ui.painter().layout(why, kit::ui(14.0), kit::DIM, rect.width() - 80.0); ui.painter() @@ -554,12 +719,35 @@ impl Panels { }, Color32::WHITE, ); + // Each marker still in the window: a pennant over the pictures with its + // number, and a line down through the sound. + for marker in self.controls.markers.list() { + let age = now.saturating_duration_since(marker.at); + if age >= replay::WINDOW { + continue; + } + let x = x_at(age); + let top = thumbs.top() - 12.0; + painter.vline(x, top..=wave.bottom(), Stroke::new(1.5, kit::TEXT.gamma_multiply(0.8))); + painter.add(egui::Shape::convex_polygon( + vec![pos2(x, top), pos2(x + 9.0, top + 3.5), pos2(x, top + 7.0)], + kit::TEXT, + Stroke::NONE, + )); + painter.text( + pos2(x + 11.0, top + 3.5), + Align2::LEFT_CENTER, + marker.number.to_string(), + kit::mono(10.0), + kit::TEXT, + ); + } // Why there are no pictures, over everything. let note = match &view.held { None => Some("Reading the last 30 seconds".to_string()), Some(Err(why)) => Some(capitalize(why)), Some(Ok(held)) if !held.video => Some("Sound only".to_string()), - Some(Ok(_)) => None, + Some(Ok(_)) => view.pictures.as_ref().err().map(|why| capitalize(why)), }; if let Some(note) = note { let galley = painter.layout(note, kit::ui(13.0), kit::TEXT, thumbs.width() - 80.0); @@ -666,6 +854,7 @@ impl Panels { let mut refusals = Refusals { replay: None, save: None, + held: None, }; let stretch = replay .stretch @@ -684,11 +873,12 @@ impl Panels { .iter() .any(|(_, source)| matches!(source, Source::Replay)); drop(state); - refusals.replay = held.or(stretch).or_else(|| match () { + refusals.replay = held.clone().or(stretch).or_else(|| match () { _ if !into => Some("Add a replay source to play replays in".into()), _ if sound_only && replay.speed < 1.0 => Some("Sound plays only at 100%".into()), _ => None, }); + refusals.held = held; refusals } @@ -1303,6 +1493,31 @@ fn ago(then: kio::clock::Wall) -> String { } } +/// Why a marker dropped `ago` cannot be clipped around, if it cannot: REVIEW's +/// pick holding no past, as `held` says, or the marker older than its window. +fn clip_refusal(ago: Duration, held: Option<&str>) -> Option { + match held { + Some(held) => Some(held.to_string()), + None if ago >= replay::WINDOW => Some(format!( + "It is older than the last {} s, as far back as a clip reaches", + replay::WINDOW.as_secs() + )), + None => None, + } +} + +/// How long ago a marker was dropped: "now", "12 s ago", "4 min ago", +/// "1 h 5 min ago". +fn since(ago: Duration) -> String { + let seconds = ago.as_secs(); + match seconds { + 0 => "now".into(), + 1..60 => format!("{seconds} s ago"), + 60..3600 => format!("{} min ago", seconds / 60), + _ => format!("{} h {} min ago", seconds / 3600, seconds / 60 % 60), + } +} + /// A length: "0:08", "52:10", "2:10:44". pub(super) fn clock(length: Duration) -> String { let seconds = length.as_secs(); @@ -1323,6 +1538,10 @@ mod tests { assert_eq!(clock(Duration::from_secs(8)), "0:08"); assert_eq!(clock(Duration::from_secs(52 * 60 + 10)), "52:10"); assert_eq!(clock(Duration::from_secs(2 * 3600 + 10 * 60 + 44)), "2:10:44"); + assert_eq!(since(Duration::from_millis(900)), "now"); + assert_eq!(since(Duration::from_secs(59)), "59 s ago"); + assert_eq!(since(Duration::from_secs(4 * 60 + 59)), "4 min ago"); + assert_eq!(since(Duration::from_secs(3600 + 5 * 60)), "1 h 5 min ago"); } /// Each handle is taken near it, and where both are in reach, the one on @@ -1339,6 +1558,17 @@ mod tests { assert_eq!(grip_at(104.0, 100.0, 106.0), Some(Grip::To)); } + /// A marker's scissors give the reason Save clip gives when the pick + /// holds no past, such as the program off air, before the marker's age. + #[test] + fn a_marker_is_clipped_only_while_its_past_is_held() { + let off_air = Some("The program keeps no past until it goes live"); + assert_eq!(clip_refusal(Duration::from_secs(4), None), None); + assert_eq!(clip_refusal(Duration::from_secs(4), off_air).as_deref(), off_air); + assert_eq!(clip_refusal(replay::WINDOW, off_air).as_deref(), off_air); + assert!(clip_refusal(replay::WINDOW, None).is_some_and(|why| why.starts_with("It is older"))); + } + /// A wide picture in a narrow room is cut at its sides, a tall one at its /// top and bottom, and one of the room's shape is whole. #[test] diff --git a/rs/moq-studio/src/ui/shell.rs b/rs/moq-studio/src/ui/shell.rs index ce4e279c..df26a05e 100644 --- a/rs/moq-studio/src/ui/shell.rs +++ b/rs/moq-studio/src/ui/shell.rs @@ -1021,7 +1021,8 @@ impl Panels { Some(bindings.get(action)?.shown(ctx)) } - /// The panic key, named for the scene it cuts to, and saving a clip. + /// The panic key, named for the scene it cuts to, saving a clip, and + /// dropping a marker. fn panic_key(&mut self, ui: &mut Ui, snapshot: &Snapshot, rect: Rect, tall: bool) { let room = rect.shrink(MARGIN); let note = match &snapshot.safe { @@ -1030,6 +1031,7 @@ impl Panels { }; let key = self.key(&Action::Panic, ui.ctx()); let clip_key = self.key(&Action::SaveClip, ui.ctx()); + let marker_key = self.key(&Action::Marker, ui.ctx()); let text = snapshot.safe.as_deref().unwrap_or("Panic"); let panic = kit::Quick::new(Icon::Panic, text, "Panic", room.width()) .look(kit::Look::Cue) @@ -1041,6 +1043,10 @@ impl Panels { .note(Some( "Save the last seconds, as many as REVIEW picks, as a clip into the recordings folder", )); + let marker = IconButton::new(Icon::Marker, "Drop a marker") + .look(kit::Look::Boxed) + .key(marker_key.as_deref()) + .note(Some("Mark this moment, which REVIEW lists and cuts a clip around")); let (panic_at, clip_at) = match tall { true => ( Rect::from_min_size(room.min, vec2(room.width(), 30.0)), @@ -1048,15 +1054,19 @@ impl Panels { ), false => ( Rect::from_center_size(room.center() - vec2(0.0, 19.0), vec2(room.width(), 30.0)), - Rect::from_center_size(room.center() + vec2(0.0, 19.0), Vec2::splat(30.0)), + Rect::from_center_size(room.center() + vec2(-19.0, 19.0), Vec2::splat(30.0)), ), }; + let marker_at = clip_at.translate(vec2(38.0, 0.0)); if ui.put(panic_at, panic).clicked() { self.controls.press(&Action::Panic); } if ui.put(clip_at, clip).clicked() { self.controls.press(&Action::SaveClip); } + if ui.put(marker_at, marker).clicked() { + self.controls.press(&Action::Marker); + } } /// A row for each sound the stream carries, with its meter and mute, and