diff --git a/quest/m1/mp4-chapters-apple.md b/quest/m1/mp4-chapters-apple.md new file mode 100644 index 0000000000..a0cff92f86 --- /dev/null +++ b/quest/m1/mp4-chapters-apple.md @@ -0,0 +1,20 @@ +# [XS] Apple's players list a finished MP4's chapters + +## Goal + +A regular MP4 from `container::mp4` with chapters lists them in QuickTime Player +and Safari, read with AVFoundation (`AVAsset.chapterMetadataGroups(bestMatchingPreferredLanguages:)`) +rather than assumed. The chapter text track writes a `tx3g` sample entry because +mp4-atom cannot encode QuickTime's `text` entry; if Apple's reader ignores it, +carry `text` upstream in mp4-atom or find what Apple accepts. + +## Plan + +ffprobe lists the chapters from either form alone, which a Linux host proves. +Only a Mac reads them the way Apple's players do: write a finished file with +chapters, load it with AVFoundation, and compare titles and start times with +what `chpl` holds. + +## Hosts + +- macos - AVFoundation and QuickTime Player are the reader under test diff --git a/quest/m1/mp4-chapters.md b/quest/m1/mp4-chapters.md deleted file mode 100644 index 39724f6aa0..0000000000 --- a/quest/m1/mp4-chapters.md +++ /dev/null @@ -1,26 +0,0 @@ -# [S] A finished MP4 carries chapters - -## Goal - -moq-mux's regular MP4 export (`container::mp4`, both `Export::finish` and -crash `repair`) takes a list of chapters, each a moment on the recorded -broadcast's timeline and a title, and writes them into the finished `moov`. -ffmpeg, mpv, VLC, and Apple's players (QuickTime Player, Safari, Photos) list -them. Reading chapters back on import is out of scope. - -## Plan - -Decided 2026-11-20 (maintainer, /quest-plan): ✅ write both forms, Nero's -`chpl` box in the `moov`'s `udta` and a QuickTime chapter text track linked -from the video track by `tref chap` (rejected: `chpl` only, which Apple's -players ignore; the QuickTime track only, which some editors ignore). - -moq-mux alone knows where the movie starts on the broadcast's timeline, so it -maps each chapter's moment onto the movie's time; a chapter before the first -sample or after the last is dropped, not clamped. mp4-atom already encodes -`chpl`. Prove it with ffprobe listing the chapters of a finished and of a -repaired file. - -Found by [moq-dev/studio#209](https://github.com/moq-dev/studio/pull/209), -whose markers become a recording's chapters through -[the studio's pin](https://github.com/moq-dev/studio/blob/main/quest/m2/pin-mp4-chapters.md). diff --git a/rs/moq-cli/src/repair.rs b/rs/moq-cli/src/repair.rs index e30dbec720..bc356122b9 100644 --- a/rs/moq-cli/src/repair.rs +++ b/rs/moq-cli/src/repair.rs @@ -58,7 +58,7 @@ impl Mp4 { })?; let repaired = tokio::task::spawn_blocking(move || { let mut file = file; - let Some(finish) = moq_mux::container::mp4::repair(&mut file)? else { + let Some(finish) = moq_mux::container::mp4::repair(&mut file, &[])? else { return anyhow::Ok(false); }; finish.write(&mut file)?; diff --git a/rs/moq-cli/src/subscribe.rs b/rs/moq-cli/src/subscribe.rs index 76a83751e0..78c7460f00 100644 --- a/rs/moq-cli/src/subscribe.rs +++ b/rs/moq-cli/src/subscribe.rs @@ -423,7 +423,7 @@ impl Subscribe { } }; - match mp4.finish()? { + match mp4.finish(&[])? { Some(trailer) => { // Converting drops the error of a write still in flight, which flushing returns. file.flush().await?; diff --git a/rs/moq-mux/CHANGELOG.md b/rs/moq-mux/CHANGELOG.md index af700416ab..4c4729d4f9 100644 --- a/rs/moq-mux/CHANGELOG.md +++ b/rs/moq-mux/CHANGELOG.md @@ -12,6 +12,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `sync::Consumer::poll_changed` (and `changed` with the `async` feature) waits until the playout schedule differs from a `sync::Schedule` read earlier with `Consumer::schedule`, and returns the new one. It resolves on a re-pinned reference, a moved delay, a lost anchor, or a producer reset onto a clock that schedules differently, and stays parked through anything that moves no due time. A presenter asleep on a picture's due time can wake and re-schedule instead of polling. `Schedule::due` and `Schedule::delay` read the snapshot it waits on. - MP4 and Matroska import fill a video rendition's `projection` and `stereo` from Spherical Video V2's `sv3d`/`st3d` boxes and Matroska's `Projection`/`StereoMode`, and Opus channel mapping family 2 (raw Opus, Matroska, and other OpusHead sources) imports with `layout: "ambisonic"`. What the catalog cannot describe is refused with `fmp4::Error::UnsupportedSpatial`, `mkv::Error::UnsupportedSpatial`, or `opus::Error::UnsupportedChannelCount`: a cubemap or mesh projection, a nonzero pose, uncentred or vertically cropped equirectangular bounds, a stereo packing other than mono, left-right, or top-bottom with the left eye first, and family 2 at zeroth order or with RFC 8486's non-diegetic stereo pair. - fMP4 carries surround Opus: a `dOps` channel mapping table (family 1, 2, or 255) imports into the OpusHead description, and an exported `dOps` writes the description's table back instead of refusing it with `opus::Error::UnsupportedMappingFamily`. Family 3, whose OpusHead carries a demixing matrix rather than a table, is still refused both ways. +- `container::mp4::Chapter` and a `chapters` argument to `container::mp4::Export::finish` and `container::mp4::repair` write chapters into a finished MP4, both as Nero's `chpl` box and as a QuickTime chapter text track that the first track references. A chapter is a `Timestamp` on the recorded broadcast's timeline and a title; one before the first sample or after the last is dropped, and more than 255 chapters or a title over 255 bytes is refused with `fmp4::Error::TooManyChapters` or `fmp4::Error::ChapterTitleTooLong`. Pass `&[]` for none. - `container::mp4::Export::stop` ends a recording before the broadcast does: `next` still returns the fragment each track was buffering, then `None`, so `finish` indexes it. - `container::fmp4::Muxer::picture_size` reads the picture size a group's keyframe declares in band, so a fetch-on-demand consumer can tell which side of an in-band resize a group falls on. - `container::Producer::fail` finishes a track with an error instead of aborting it: every group written stays readable, and a reader reaching the end gets the error instead of the end of stream. diff --git a/rs/moq-mux/src/container/fmp4/mod.rs b/rs/moq-mux/src/container/fmp4/mod.rs index 31aa90cd4c..fd51aa0fa6 100644 --- a/rs/moq-mux/src/container/fmp4/mod.rs +++ b/rs/moq-mux/src/container/fmp4/mod.rs @@ -389,6 +389,14 @@ pub enum Error { reason: &'static str, }, + /// A regular MP4's chapter list holds 255 chapters. + #[error("{0} chapters, more than the 255 a chapter list holds")] + TooManyChapters(usize), + + /// A regular MP4's chapter list holds titles of 255 bytes. + #[error("a chapter title of {0} bytes, more than the 255 a chapter list holds")] + ChapterTitleTooLong(usize), + /// A file to repair is not a recording that an MP4 export left unfinished. #[error("not an unfinished mp4 recording: {0}")] Unrepairable(&'static str), diff --git a/rs/moq-mux/src/container/mp4.rs b/rs/moq-mux/src/container/mp4.rs index 9de72d92dd..0d4ce47839 100644 --- a/rs/moq-mux/src/container/mp4.rs +++ b/rs/moq-mux/src/container/mp4.rs @@ -11,6 +11,10 @@ //! everything before that `moov`. The reserved box must stay `free` until then: an `mdat` //! header there would swallow the fragmented `moov` and leave a crashed file unreadable. //! +//! A finished file lists [`Chapter`]s two ways, so every player finds them: Nero's `chpl` box +//! in the `moov`, and a QuickTime chapter text track that the first track references with +//! `tref chap`. The text lies in an `mdat` between the media and the `moov`. +//! //! A rendition that joins after the fragmented `moov` is written into `free` boxes, which a //! reader of the unfinished file skips. Before a track's first fragment, and again whenever //! its description changes, a `free` box holds a `moov` declaring it, so the final `moov` @@ -22,6 +26,7 @@ use std::task::Poll; use std::time::Duration; use bytes::{BufMut, Bytes, BytesMut}; +use moq_net::Timestamp; use mp4_atom::{Decode, Encode}; use crate::Result; @@ -34,9 +39,26 @@ const RESERVED: u64 = 16; /// The reserved box as a recording writes it, an empty `free` box. const FREE: [u8; RESERVED as usize] = [0, 0, 0, RESERVED as u8, b'f', b'r', b'e', b'e', 0, 0, 0, 0, 0, 0, 0, 0]; +/// What a `chpl` box's one-byte counts allow: chapters in the list, and bytes in a title. +const CHPL_LIMIT: usize = 255; + +/// The chapter text track's timescale. +const TEXT_TIMESCALE: u32 = 1000; + /// The `sample_is_non_sync_sample` bit of ISO/IEC 14496-12 sample flags. const NON_SYNC: u32 = 0x0001_0000; +/// A moment in a recording that a player lists as a chapter. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Chapter { + /// When the chapter starts on the recorded broadcast's timeline, the clock its frames' + /// timestamps are on. One before the first sample or after the last is dropped. + pub time: Timestamp, + + /// The chapter's name, at most 255 bytes. + pub title: String, +} + /// Subscribe to a moq broadcast and record it as a crash-safe regular MP4. /// /// [`poll_next`](Self::poll_next) returns the bytes to append to the file, the same fMP4 an @@ -67,8 +89,9 @@ pub struct Finish { /// fragment. moov_offset: u64, - /// The regular `moov`. - moov: Bytes, + /// What follows the media: the chapter text's `mdat` if there are chapters, then the regular + /// `moov`. + tail: Bytes, /// Where `mdat` overwrites the reserved box, from the start of the file. mdat_offset: u64, @@ -102,7 +125,7 @@ impl Finish { // Cut before the moov lands where the cut bytes were, which could otherwise mix with it. file.set_len(self.moov_offset)?; file.sync()?; - file.write_at(self.moov_offset, &self.moov)?; + file.write_at(self.moov_offset, &self.tail)?; // The size goes into the body of the `free` box, so only the type needs the moov first. let (kind, size) = self.mdat.split_at(8); file.write_at(self.mdat_offset + 8, size)?; @@ -182,12 +205,13 @@ impl Export { self.fmp4.stop(); } - /// The writes that turn the file into a regular MP4, or `None` when nothing was written. + /// The writes that turn the file into a regular MP4 listing `chapters`, or `None` when + /// nothing was written. /// /// Media still buffered for a fragment that has not closed is left out, unless /// [`stop`](Self::stop) wrote it first. - pub fn finish(self) -> Result> { - self.index.map(|index| index.finish(self.written)).transpose() + pub fn finish(self, chapters: &[Chapter]) -> Result> { + self.index.map(|index| index.finish(self.written, chapters)).transpose() } /// The file header: the init segment's `ftyp`, the reserved box, then its `moov`. @@ -260,8 +284,10 @@ impl Export { /// The final `moov` indexes every complete fragment, the same as finishing the recording /// right after writing the last one. Whatever follows it, such as a partial fragment or /// `moov`, is cut, so a repair cut short can itself be repaired. A file only counts as finished -/// when its `mdat` header and trailing `moov` are exactly what finishing it writes. -pub fn repair(file: &mut F) -> Result> { +/// when its `mdat` header and trailing `moov` are exactly what finishing it writes, `chapters` +/// included. Given none, a finished file's own chapters stay, so repairing a file never +/// drops them; given some, they replace the file's. +pub fn repair(file: &mut F, chapters: &[Chapter]) -> Result> { let eof = file.seek(SeekFrom::End(0))?; let Some(ftyp) = Atom::read(file, 0, eof)?.filter(|atom| atom.kind == *b"ftyp") else { return Err(Error::Unrepairable("it does not start with an ftyp box").into()); @@ -315,8 +341,9 @@ pub fn repair(file: &mut F) -> Result> { _ => return Err(Error::Unrepairable("a free box holds neither a fragment nor a moov").into()), } } - // A finish appends a moov, which ends the recording, or a cut-short one part of it. - b"moov" => break, + // A finish appends the chapter text and a moov, which end the recording, or a + // cut-short one part of it. + b"mdat" | b"moov" => break, _ => return Err(Error::Unrepairable("it holds a box a recording never writes").into()), }; for moov in described.drain(..) { @@ -325,7 +352,7 @@ pub fn repair(file: &mut F) -> Result> { index.fragment(moof.start, &moof.decode(file)?)?; end = position; } - let mut finish = index.finish(end)?; + let mut finish = index.finish(end, chapters)?; // Each byte is the reserved box's or the mdat header's, whichever reached the disk. let torn = slot @@ -335,9 +362,11 @@ pub fn repair(file: &mut F) -> Result> { if !torn { return Err(Error::Unrepairable("its reserved box is neither free nor its mdat header").into()); } - // The moov a finish appended, unless anything differs, a cut-short one included. - let indexed = eof - end == finish.moov.len() as u64 - && read_at(file, end, eof - end, eof)?.is_some_and(|trailer| trailer == finish.moov); + // What a finish appended, unless anything differs, a cut-short one included. Given no + // chapters, a finish that listed some counts too, which they would otherwise lose. + let indexed = (eof - end == finish.tail.len() as u64 + && read_at(file, end, eof - end, eof)?.is_some_and(|trailer| trailer == finish.tail)) + || (chapters.is_empty() && keeps_chapters(file, end, eof, &finish.tail)?); if indexed && slot == finish.mdat { return Ok(None); } @@ -352,6 +381,40 @@ pub fn repair(file: &mut F) -> Result> { Ok(Some(finish)) } +/// Whether the file's trailer from `end` is a finish that listed chapters: its text and a +/// `moov` that is `plain`, the `moov` of a finish without chapters, apart from the chapters. +fn keeps_chapters(file: &mut F, end: u64, eof: u64, plain: &[u8]) -> Result { + let mut atom = Atom::read(file, end, eof)?; + if let Some(text) = atom.as_ref().filter(|atom| atom.kind == *b"mdat") { + atom = Atom::read(file, text.end, eof)?; + } + let Some(atom) = atom.filter(|atom| atom.kind == *b"moov" && atom.end == eof) else { + return Ok(false); + }; + let Some(data) = read_at(file, atom.start, atom.end - atom.start, eof)? else { + return Ok(false); + }; + // A moov that does not decode is damage for the finish to replace, not a failure. + let Ok(mut moov) = mp4_atom::Moov::decode(&mut std::io::Cursor::new(data.as_slice())) else { + return Ok(false); + }; + let text = moov + .trak + .iter() + .rposition(|trak| trak.mdia.hdlr.handler == b"text".into()); + let (Some(text), Some(_)) = (text, moov.udta.take().and_then(|udta| udta.chpl)) else { + return Ok(false); + }; + moov.trak.remove(text); + for trak in &mut moov.trak { + trak.tref = None; + } + moov.mvhd.next_track_id = moov.trak.iter().map(|trak| trak.tkhd.track_id).max().unwrap_or(0) + 1; + let mut buf = Vec::new(); + moov.encode(&mut buf).map_err(crate::Error::mp4)?; + Ok(buf == plain) +} + /// The `len` bytes at `start`, or `None` when they do not end by `limit`. fn read_at(file: &mut F, start: u64, len: u64, limit: u64) -> Result>> { if start.checked_add(len).is_none_or(|end| end > limit) { @@ -576,8 +639,8 @@ impl Index { Ok(()) } - /// The writes that make the file a regular MP4 with its `moov` at `end`. - fn finish(&self, end: u64) -> Result { + /// The writes that make the file a regular MP4 with its chapter text and `moov` at `end`. + fn finish(&self, end: u64, chapters: &[Chapter]) -> Result { let mut timelines = Vec::new(); for (&id, track) in self.tracks.iter().filter(|(_, track)| !track.samples.is_empty()) { let Some(Description { trak, .. }) = self.descriptions.get(&id) else { @@ -610,21 +673,69 @@ impl Index { } traks.sort_by_key(|trak| (trak.mdia.hdlr.handler != b"vide".into(), trak.tkhd.track_id)); + let duration = traks.iter().map(|trak| trak.tkhd.duration).max().unwrap_or(0); + let mut next_track_id = traks.iter().map(|trak| trak.tkhd.track_id).max().unwrap_or(0) + 1; + + // The chapter text lies in an mdat of its own, right where the moov would start. + let mut tail = BytesMut::new(); + let mut udta = None; + let chapters = self.place(chapters, origin, duration)?; + if !chapters.is_empty() { + let id = next_track_id; + next_track_id += 1; + let (trak, text) = chapter_track(id, self.timescale, duration, &chapters, end)?; + mp4_atom::Header { + kind: b"mdat".into(), + size: Some(text.len()), + } + .encode(&mut tail) + .map_err(crate::Error::mp4)?; + tail.put_slice(&text); + + let movie = u128::from(self.timescale); + udta = Some(mp4_atom::Udta { + chpl: Some(mp4_atom::Chpl { + entries: chapters + .iter() + .map(|&(start, title)| { + // The list counts 100 ns units. + let start = u128::from(start) * 10_000_000 / movie; + Ok(mp4_atom::ChplEntry { + start: u64::try_from(start).map_err(|_| Error::PtsOverflow)?, + title: title.to_string(), + }) + }) + .collect::>()?, + }), + ..Default::default() + }); + // The first track, a video one if there is any, points at the text. + if let Some(first) = traks.first_mut() { + first.tref = Some(mp4_atom::Tref { + track_reference_type_boxes: vec![mp4_atom::TrackReferenceTypeBox { + reference_type: b"chap".into(), + track_ids: vec![id], + }], + }); + } + traks.push(trak); + } + let moov = mp4_atom::Moov { mvhd: mp4_atom::Mvhd { timescale: self.timescale, - duration: traks.iter().map(|trak| trak.tkhd.duration).max().unwrap_or(0), + duration, // Normal playback rate and full volume; both default to 0. rate: mp4_atom::FixedPoint::from(1), volume: mp4_atom::FixedPoint::from(1), - next_track_id: traks.iter().map(|trak| trak.tkhd.track_id).max().unwrap_or(0) + 1, + next_track_id, ..Default::default() }, trak: traks, + udta, ..Default::default() }; - let mut buf = Vec::new(); - moov.encode(&mut buf).map_err(crate::Error::mp4)?; + moov.encode(&mut tail).map_err(crate::Error::mp4)?; // Everything from the reserved box to the moov, the fragmented moov and every moof // included, becomes one mdat that a reader skips through to the moov. @@ -635,7 +746,7 @@ impl Index { Ok(Finish { moov_offset: end, - moov: Bytes::from(buf), + tail: tail.freeze(), mdat_offset: self.reserved, mdat: mdat.freeze(), reopen: false, @@ -643,6 +754,136 @@ impl Index { } } +impl Index { + /// Each chapter that falls within the movie, sorted, as its start in movie ticks from the + /// movie's start at `origin` and its title. + fn place<'a>(&self, chapters: &'a [Chapter], origin: u64, duration: u64) -> Result> { + let timescale = moq_net::Timescale::new(u64::from(self.timescale))?; + let mut placed = Vec::new(); + for chapter in chapters { + let ticks = u64::try_from(chapter.time.as_scale(timescale)?).map_err(|_| Error::PtsOverflow)?; + // A chapter before the first sample or after the last has nothing to start on. + if let Some(start) = ticks.checked_sub(origin).filter(|&start| start < duration) { + placed.push((start, chapter.title.as_str())); + } + } + // A list a player reads in order, equal starts keeping the order given. + placed.sort_by_key(|&(start, _)| start); + if placed.len() > CHPL_LIMIT { + return Err(Error::TooManyChapters(placed.len()).into()); + } + if let Some((_, title)) = placed.iter().find(|(_, title)| title.len() > CHPL_LIMIT) { + return Err(Error::ChapterTitleTooLong(title.len()).into()); + } + Ok(placed) + } +} + +/// The QuickTime chapter track, whose samples are the titles in the order given, and the +/// text they lie in, which starts at `offset` in the file after the `mdat` header. +/// +/// A chapter lasts until the next starts, the last until the movie ends, and the first +/// starts after an empty edit if it does not start the movie. Each start is in movie ticks. +fn chapter_track( + id: u32, + movie: u32, + duration: u64, + chapters: &[(u64, &str)], + offset: u64, +) -> Result<(mp4_atom::Trak, Vec)> { + let first = chapters.first().map_or(0, |&(start, _)| start); + // Milliseconds, not the movie's ticks, which a long recording at 90 kHz overflows a + // 32-bit sample duration in hours. The edits stay in the movie's. + let scale = |ticks: u64| ticks * u64::from(TEXT_TIMESCALE) / u64::from(movie); + let mut text = Vec::new(); + let mut sizes = Vec::new(); + let mut deltas: Vec = Vec::new(); + for (i, &(start, title)) in chapters.iter().enumerate() { + let before = text.len(); + // A 16-bit length and the title, then `encd` declaring it UTF-8 as ffmpeg writes it. + text.extend_from_slice(&(title.len() as u16).to_be_bytes()); + text.extend_from_slice(title.as_bytes()); + text.extend_from_slice(&[0, 0, 0, 12, b'e', b'n', b'c', b'd', 0, 0, 1, 0]); + sizes.push((text.len() - before) as u32); + + let end = chapters.get(i + 1).map_or(duration, |&(next, _)| next); + let delta = u32::try_from(scale(end) - scale(start)).map_err(|_| Error::PtsOverflow)?; + deltas.push(delta); + } + + let mut stts: Vec = Vec::new(); + for delta in deltas { + match stts.last_mut() { + Some(last) if last.sample_delta == delta => last.sample_count += 1, + _ => stts.push(mp4_atom::SttsEntry { + sample_count: 1, + sample_delta: delta, + }), + } + } + + let trak = mp4_atom::Trak { + tkhd: mp4_atom::Tkhd { + track_id: id, + // Text a player lists rather than shows, so it is not enabled. + enabled: false, + in_movie: true, + duration, + ..Default::default() + }, + edts: edits(first, 0, duration - first), + mdia: mp4_atom::Mdia { + mdhd: mp4_atom::Mdhd { + timescale: TEXT_TIMESCALE, + duration: scale(duration) - scale(first), + ..Default::default() + }, + hdlr: mp4_atom::Hdlr { + handler: b"text".into(), + name: String::new(), + }, + minf: mp4_atom::Minf { + nmhd: Some(mp4_atom::Nmhd {}), + dinf: mp4_atom::Dinf { + dref: mp4_atom::Dref { + entries: vec![mp4_atom::DrefEntry::Url(mp4_atom::Url::default())], + }, + }, + stbl: mp4_atom::Stbl { + stsd: mp4_atom::Stsd { + codecs: vec![ + mp4_atom::Tx3g { + data_reference_index: 1, + ..Default::default() + } + .into(), + ], + }, + stts: mp4_atom::Stts { entries: stts }, + stsc: mp4_atom::Stsc { + entries: vec![mp4_atom::StscEntry { + first_chunk: 1, + samples_per_chunk: chapters.len() as u32, + sample_description_index: 1, + }], + }, + stsz: mp4_atom::Stsz { + samples: mp4_atom::StszSamples::Different { sizes }, + }, + // The mdat's header is 8 bytes. + co64: Some(mp4_atom::Co64 { + entries: vec![offset + 8], + }), + ..Default::default() + }, + ..Default::default() + }, + }, + ..Default::default() + }; + Ok((trak, text)) +} + fn unindexable(track: u32, reason: &'static str) -> Error { Error::Unindexable { track, reason } } @@ -919,7 +1160,10 @@ mod tests { /// Finish the recording in `file` the way a writer does. fn finish(export: super::Export, file: &mut Vec) { - apply(export.finish().expect("finish").expect("something was written"), file); + apply( + export.finish(&[]).expect("finish").expect("something was written"), + file, + ); } /// Make the writes of `finish` to `file`. @@ -1043,13 +1287,22 @@ mod tests { None } + /// The `moov` a finish appends, after the chapter text if it wrote any. + fn final_moov(finish: &super::Finish) -> &[u8] { + let tail = &finish.tail[..]; + match &tail[4..8] { + b"mdat" => &tail[u32::from_be_bytes(tail[..4].try_into().unwrap()) as usize..], + _ => tail, + } + } + fn repair(file: &[u8]) -> crate::Result> { - super::repair(&mut std::io::Cursor::new(file)) + super::repair(&mut std::io::Cursor::new(file), &[]) } fn assert_same(finish: &super::Finish, expected: &super::Finish, cut: usize) { assert_eq!(finish.moov_offset, expected.moov_offset, "cut at {cut}"); - assert_eq!(finish.moov, expected.moov, "cut at {cut}"); + assert_eq!(finish.tail, expected.tail, "cut at {cut}"); assert_eq!(finish.mdat_offset, expected.mdat_offset, "cut at {cut}"); assert_eq!(finish.mdat, expected.mdat, "cut at {cut}"); } @@ -1452,7 +1705,7 @@ mod tests { export.stop(); assert!(export.next().await.expect("stopping is not a failure").is_none()); - assert!(export.finish().unwrap().is_none()); + assert!(export.finish(&[]).unwrap().is_none()); } /// Nothing to finish before the header is written. @@ -1460,7 +1713,7 @@ mod tests { async fn nothing_written_needs_no_finish() { let live = Live::avc3(); let export = export(&live).await; - assert!(export.finish().unwrap().is_none()); + assert!(export.finish(&[]).unwrap().is_none()); } /// A recording cut anywhere, mid-fragment or right after a late rendition joins, repairs @@ -1495,12 +1748,12 @@ mod tests { continue; }; let repaired = repaired.unwrap().expect("unfinished"); - assert_same(&repaired, &index.finish(*end as u64).unwrap(), cut); + assert_same(&repaired, &index.finish(*end as u64, &[]).unwrap(), cut); // A repair cut short after its moov leaves a file that repairs the same. let mut half = file[..cut].to_vec(); half.truncate(repaired.moov_offset as usize); - half.extend_from_slice(&repaired.moov); + half.extend_from_slice(&repaired.tail); assert_same(&repair(&half).unwrap().expect("unfinished"), &repaired, cut); } @@ -1516,7 +1769,7 @@ mod tests { /// A recording with audio and video and nothing past its last fragment, and the bytes that /// finish it. - async fn recorded() -> (Vec, super::Finish) { + async fn recorded(chapters: &[super::Chapter]) -> (Vec, super::Finish) { let mut live = Live::avc3(); let mut audio = add_opus(&mut live, 0); for i in 0..6u64 { @@ -1528,7 +1781,7 @@ mod tests { let mut export = export(&live).await; let mut file = Vec::new(); record(&mut export, &mut file).await; - let finish = export.finish().unwrap().expect("something was written"); + let finish = export.finish(chapters).unwrap().expect("something was written"); (file, finish) } @@ -1537,7 +1790,19 @@ mod tests { /// reads, through the fragmented moov or the final one, and that repairs to the same finish. #[tokio::test(start_paused = true)] async fn a_finish_interrupted_anywhere_plays_and_repairs() { - let (file, finish) = recorded().await; + interrupted(&[], ["ftyp", "mdat", "moov"]).await; + } + + /// The same with chapters, whose text goes down in an `mdat` before the moov. + #[tokio::test(start_paused = true)] + async fn a_finish_with_chapters_interrupted_anywhere_plays_and_repairs() { + interrupted(&chapters(), ["ftyp", "mdat", "mdat", "moov"]).await; + } + + async fn interrupted(chapters: &[super::Chapter], layout: [&str; N]) { + // The repair of a recording lists the chapters its finish would. + let repair = |file: &[u8]| super::repair(&mut std::io::Cursor::new(file), chapters); + let (file, finish) = recorded(chapters).await; let mut finished = file.clone(); apply(finish.clone(), &mut finished); let fragmented = played(&file).expect("the unfinished file plays").to_vec(); @@ -1547,12 +1812,12 @@ mod tests { let partial = repair(&cut).unwrap().expect("unfinished"); let mut repaired = cut.clone(); apply(partial.clone(), &mut repaired); - assert_eq!(kinds(&atoms(&repaired)), ["ftyp", "mdat", "moov"]); + assert_eq!(kinds(&atoms(&repaired)), layout); // Part of a late rendition's description, longer than the final moov. let mut described = file.clone(); described.extend_from_slice(&[0, 0, 0x40, 0, b'f', b'r', b'e', b'e']); - described.resize(described.len() + 4 * finish.moov.len(), 0); + described.resize(described.len() + 4 * finish.tail.len(), 0); for (start, finish, done) in [ (&file, &finish, &finished), @@ -1562,7 +1827,7 @@ mod tests { for (i, crashed) in crashes(start, finish).iter().enumerate() { let moov = played(crashed).unwrap_or_else(|| panic!("crash {i} left a file no player reads")); assert!( - moov == fragmented || moov == finish.moov, + moov == fragmented || moov == final_moov(finish), "crash {i} plays a broken moov" ); let Some(again) = repair(crashed).unwrap_or_else(|err| panic!("crash {i}: {err}")) else { @@ -1578,7 +1843,7 @@ mod tests { for (j, twice) in twice.iter().enumerate() { let moov = played(twice).unwrap_or_else(|| panic!("crash {i}, {j} left a file no player reads")); assert!( - moov == fragmented || moov == finish.moov, + moov == fragmented || moov == final_moov(finish), "crash {i}, {j} plays a broken moov" ); match repair(twice).unwrap_or_else(|err| panic!("crash {i}, {j}: {err}")) { @@ -1591,7 +1856,7 @@ mod tests { assert_eq!(&fixed, done, "crash {i}"); } } - assert_eq!(kinds(&atoms(&finished)), ["ftyp", "mdat", "moov"]); + assert_eq!(kinds(&atoms(&finished)), layout); } /// A finished file counts only when its header and trailing moov are exactly what its @@ -1599,7 +1864,7 @@ mod tests { /// changed, is finished again. #[tokio::test(start_paused = true)] async fn only_a_verified_index_counts_as_finished() { - let (file, finish) = recorded().await; + let (file, finish) = recorded(&[]).await; let mut finished = file.clone(); apply(finish.clone(), &mut finished); assert!(repair(&finished).unwrap().is_none(), "already a regular MP4"); @@ -1617,7 +1882,7 @@ mod tests { damaged.extend([first, last].into_iter().filter(|torn| *torn != finished)); } let moov = finish.moov_offset as usize; - for len in [0, 1, 8, finish.moov.len() - 1] { + for len in [0, 1, 8, finish.tail.len() - 1] { let mut cut = finished.clone(); cut.truncate(moov + len); damaged.push(cut); @@ -1644,4 +1909,210 @@ mod tests { let fmp4 = [&file[..reserved], &file[reserved + 16..]].concat(); assert!(repair(&fmp4).is_err()); } + + fn chapter(millis: u64, title: &str) -> super::Chapter { + super::Chapter { + time: moq_net::Timestamp::from_millis(millis).unwrap(), + title: title.to_string(), + } + } + + /// Chapters in the order a studio might mark them, the last one past the recording. + fn chapters() -> Vec { + vec![ + chapter(100, "Second"), + chapter(0, "First \u{1f3ac}"), + chapter(100, "Also second"), + chapter(10_000, "Past the end"), + ] + } + + /// Where a chapter starts in milliseconds, and its title. + type Listed = Vec<(u64, String)>; + + /// The chapters a file lists in each form: its `chpl` box, then its text track. + fn listed(file: &[u8], moov: &mp4_atom::Moov) -> (Listed, Listed) { + let movie = u64::from(moov.mvhd.timescale); + let chpl = moov.udta.as_ref().and_then(|udta| udta.chpl.as_ref()).expect("a chpl"); + let chpl = chpl + .entries + .iter() + .map(|entry| (entry.start / 10_000, entry.title.clone())) + .collect(); + + let text = trak(moov, b"text"); + let mut start = 0; + // The text track counts milliseconds, its edits the movie's ticks. + let edits = text + .edts + .as_ref() + .and_then(|edts| edts.elst.as_ref()) + .map(|elst| elst.entries.as_slice()); + // An empty edit delays the first chapter. + if let Some([delay, _]) = edits { + start = delay.segment_duration * 1000 / movie; + } + let mut starts = Vec::new(); + for entry in &text.mdia.minf.stbl.stts.entries { + for _ in 0..entry.sample_count { + starts.push(start); + start += u64::from(entry.sample_delta); + } + } + let track = samples(file, text) + .iter() + .zip(starts) + .map(|(sample, start)| { + let len = u16::from_be_bytes([sample[0], sample[1]]) as usize; + (start, String::from_utf8(sample[2..2 + len].to_vec()).unwrap()) + }) + .collect(); + (chpl, track) + } + + /// Both forms list the chapters that fall in the recording, in order, and the first track + /// points at the text. + #[tokio::test(start_paused = true)] + async fn a_finish_lists_chapters_both_ways() { + let (mut file, finish) = recorded(&chapters()).await; + apply(finish, &mut file); + let moov = moov(atoms(&file)); + let expected = vec![ + (0, "First \u{1f3ac}".to_string()), + (100, "Second".to_string()), + (100, "Also second".to_string()), + ]; + let (chpl, track) = listed(&file, &moov); + assert_eq!(chpl, expected); + assert_eq!(track, expected); + + let text = trak(&moov, b"text"); + assert!(!text.tkhd.enabled, "text a player lists is not shown"); + assert_eq!(moov.trak[0].mdia.hdlr.handler, b"vide".into()); + let reference = moov.trak[0].tref.as_ref().expect("the first track references the text"); + assert_eq!(reference.track_reference_type_boxes.len(), 1); + assert_eq!(reference.track_reference_type_boxes[0].reference_type, b"chap".into()); + assert_eq!(reference.track_reference_type_boxes[0].track_ids, [text.tkhd.track_id]); + assert_eq!(moov.mvhd.next_track_id, text.tkhd.track_id + 1); + assert_eq!(text.tkhd.duration, moov.mvhd.duration); + } + + /// A chapter before the first sample or after the last is dropped, not clamped, and a + /// movie that starts late lists chapters from its own start. + #[tokio::test(start_paused = true)] + async fn chapters_are_placed_on_the_broadcast_timeline() { + let mut live = Live::avc3(); + for i in 0..6u64 { + live.track + .write(video_frame(1_000_000 + i * 33_000, i % 3 == 0)) + .unwrap(); + } + live.track.end_with(None).unwrap(); + let mut export = export(&live).await; + let mut file = Vec::new(); + record(&mut export, &mut file).await; + + let chapters = [ + chapter(999, "Before"), + chapter(1_000, "Start"), + chapter(1_066, "Middle"), + chapter(1_198, "After"), + ]; + apply( + export.finish(&chapters).unwrap().expect("something was written"), + &mut file, + ); + let moov = moov(atoms(&file)); + let (chpl, track) = listed(&file, &moov); + let expected = vec![(0, "Start".to_string()), (66, "Middle".to_string())]; + assert_eq!(chpl, expected); + assert_eq!(track, expected); + } + + /// A recording that no chapter falls in is the same file as one given none. + #[tokio::test(start_paused = true)] + async fn no_chapter_in_range_adds_nothing() { + let (_, plain) = recorded(&[]).await; + let (_, past) = recorded(&[chapter(60_000, "Past the end")]).await; + assert_same(&past, &plain, 0); + } + + /// What a chapter list's one-byte counts cannot hold is refused instead of cut. + #[tokio::test(start_paused = true)] + async fn a_chapter_list_over_its_limits_is_refused() { + let mut live = Live::avc3(); + for i in 0..6u64 { + live.track.write(video_frame(i * 33_000, i % 3 == 0)).unwrap(); + } + let mut export = export(&live).await; + let mut file = Vec::new(); + record(&mut export, &mut file).await; + let index = export.index.expect("a header"); + let end = file.len() as u64; + assert!(index.finish(end, &vec![chapter(0, "a"); 255]).is_ok()); + assert!(index.finish(end, &vec![chapter(0, "a"); 256]).is_err()); + assert!(index.finish(end, &[chapter(0, &"a".repeat(255))]).is_ok()); + assert!(index.finish(end, &[chapter(0, &"a".repeat(256))]).is_err()); + } + + /// A chapter's duration in the text track holds for a recording of hours at the finest + /// movie timescale. + #[test] + fn a_long_movie_keeps_its_chapters() { + let hour = 3600 * 90_000; + let (trak, _) = super::chapter_track(2, 90_000, 14 * hour, &[(0, "a"), (hour, "b")], 0).unwrap(); + assert_eq!(trak.mdia.mdhd.duration, 14 * 3600 * 1000); + } + + /// A repair given no chapters leaves a finished file's alone, and finishes one whose + /// chapters are damaged. + #[tokio::test(start_paused = true)] + async fn repairing_without_chapters_keeps_a_finished_files() { + let (mut file, finish) = recorded(&chapters()).await; + apply(finish, &mut file); + assert!(repair(&file).unwrap().is_none()); + + // A chunk offset inside the moov changed, which still decodes. + let mut changed = file.clone(); + let moov = changed.windows(4).rposition(|kind| kind == b"moov").expect("a moov"); + let co64 = moov + + changed[moov..] + .windows(4) + .position(|kind| kind == b"co64") + .expect("a co64"); + changed[co64 + 19] ^= 1; + assert!(repair(&changed).unwrap().is_some()); + + // A moov that no longer decodes is finished again too. + let mut broken = file.clone(); + let tkhd = moov + + broken[moov..] + .windows(4) + .position(|kind| kind == b"tkhd") + .expect("a tkhd"); + broken[tkhd + 4] = 9; + assert!(::decode(&mut std::io::Cursor::new(&broken[moov..])).is_err()); + assert!(repair(&broken).unwrap().is_some()); + } + + /// A repair given other chapters than a finished file lists finishes it again. + #[tokio::test(start_paused = true)] + async fn repairing_with_other_chapters_replaces_them() { + let (mut file, finish) = recorded(&[]).await; + apply(finish, &mut file); + let chapters = chapters(); + let again = super::repair(&mut std::io::Cursor::new(file.as_slice()), &chapters) + .unwrap() + .expect("other chapters"); + apply(again, &mut file); + + let (mut expected, finish) = recorded(&chapters).await; + apply(finish, &mut expected); + assert_eq!(file, expected); + assert!( + super::repair(&mut std::io::Cursor::new(file.as_slice()), &chapters) + .unwrap() + .is_none() + ); + } }