Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion quest/m1/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,6 @@ audit's receive-memory, priority, and lease bugs rank ahead of that reshape.
- [Interop audit](/quest/m1/audit-interop.md) - our interop runner entries match what we ship, and every red interop cell with a moq.dev side is fixed, reported, or noted as down
- [Lint override and deprecation audit](/quest/m1/audit-lints.md) - repeat the suppression and deprecation audit, starting with the conditional allows and the lanes clippy never runs
- [Refuse unknown MOQ_* environment variables](/quest/m1/env-unknown.md) - every binary exits at startup naming a `MOQ_*` variable it does not read
- [HLS cursors error on every incomplete recording](/quest/m1/hls-teardown.md) - the recorder cursors return `Ok(None)` only for a complete recording
- [HLS recorder in the CLI](/quest/m1/hls-record-cli.md) - `export hls --output <dir>` records playlists and segments to a directory
- [Group storage benchmark](/quest/m1/group-storage-bench.md) - numbers pick the append-only log, a ring buffer, or sorted runs for `GroupState` before either lands
- [Unbounded groups](/quest/m1/unbounded-groups/README.md) - groups and frames lose their size caps; the cache sheds parts of a group under pressure and refills the holes in place
Expand Down
4 changes: 0 additions & 4 deletions quest/m1/hls-record-cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,3 @@ Decided 2026-10-23 (maintainer interview):
- This is the cursors' first consumer outside moq-hls's tests.
- Update `doc/bin/cli.md` and `doc/bin/hls.md`, and every example that uses
`export hls`.

## Required

- [HLS cursors error on every incomplete recording](/quest/m1/hls-teardown.md) - an abrupt end or a missed segment must surface as an error before the recorder relies on it
38 changes: 0 additions & 38 deletions quest/m1/hls-teardown.md

This file was deleted.

6 changes: 3 additions & 3 deletions quest/m2/hls-restart-drain-cutoff.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,9 @@ Decided 2026-10-24 (iteration, on the merits):
- Rejected: an unbounded drain of the old run; an immediate cutoff at
`Restart` that discards listed rows; treating a catalog error as media EOF.
- A cutoff recording is complete: its cursor returns `Ok(None)` after the last
listed row, as after `End`. Keep this consistent with the end semantics the
[cursor audit](/quest/m1/hls-teardown.md) settles.
listed row, as after `End`. The cursors return `Ok(None)` only for a
confirmed end and `Error::IncompleteRecording` for any other close, so the
cutoff must end the old windows, not just close them.
- No new public API: the boundary shows up as the cursor's end. No wire change.

Tests: an old publisher that keeps sending after `Restart` still finalizes
Expand All @@ -45,5 +46,4 @@ catalog EOF and catalog-error behavior are unchanged.

## Related

- [HLS cursors error on every incomplete recording](/quest/m1/hls-teardown.md) - decides which window closes are clean finishes
- [HLS recorder in the CLI](/quest/m1/hls-record-cli.md) - the recorder that would see a replaced run
14 changes: 14 additions & 0 deletions rs/moq-hls/src/error.rs
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,20 @@ pub enum Error {
/// The rewind the importer refused.
source: moq_mux::Error,
},
/// A recording source closed without confirming its complete end.
#[error("recording closed before finishing")]
IncompleteRecording,

/// A recording cursor missed rows that left its window or were reset.
#[error("recording cursor missed segments")]
RecordingLagged,

/// A listed recording segment no longer has all its media available.
#[error("recording segment {segment} is unavailable")]
SegmentUnavailable {
/// The segment whose media cannot be recorded.
segment: u64,
},
}

impl Error {
Expand Down
192 changes: 173 additions & 19 deletions rs/moq-hls/src/export/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -191,8 +191,8 @@ impl Broadcaster {

/// A cursor over rendition add/remove events, for a consumer that mirrors or records the
/// whole broadcast. It replays the current renditions, then yields changes, and returns
/// `None` once the source closes. A rendition is added once the catalog advertises its own
/// timeline, which an unhinted publisher declares only after its first record.
/// `Ok(None)` only once the source finishes cleanly. A rendition is added once the catalog
/// advertises its own timeline, which an unhinted publisher declares after its first record.
pub fn renditions(&self) -> renditions::Consumer {
self.renditions.subscribe()
}
Expand Down Expand Up @@ -443,7 +443,7 @@ async fn watch_catalog(

let mut unbound = false;
let mut reference: Option<Reference> = None;
loop {
let result = loop {
match kio::wait(|waiter| consumer.poll_next(waiter)).await {
Ok(Some(catalog)) => {
// Decide the playlists' retention before any rendition starts following its timeline.
Expand Down Expand Up @@ -483,20 +483,20 @@ async fn watch_catalog(
reference = next;
}
}
Ok(None) => break,
Ok(None) => break Ok(()),
Err(err) => {
tracing::warn!(%err, "broadcast catalog stream ended with error");
break;
break Err(err.into());
}
}
}
};

// The source is done (or errored): let recording cursors finish. With every rendition
// disabled, no reference watcher is left to end the windows.
if reference.is_none() {
// With every rendition disabled, no reference watcher is left to end the windows.
// Only a clean catalog finish confirms completion; a catalog failure fails the windows.
if reference.is_none() && result.is_ok() {
renditions.fanout().finish(None);
}
renditions.close();
renditions.finish(result);
}

/// The rendition every segment boundary comes from, with the track its timeline indexes.
Expand Down Expand Up @@ -584,7 +584,7 @@ async fn watch_timeline(
track: reference.2.clone(),
source: Box::new(err),
};
if renditions.fail(&reference, err.clone()) {
if renditions.fail(Some(&reference), err.clone()) {
tracing::error!(%err, "reference timeline failed; failing every playlist");
}
}
Expand Down Expand Up @@ -915,7 +915,7 @@ mod tests {
tokio::time::timeout(Duration::from_millis(50), cursor.next())
.await
.ok()
.flatten()
.and_then(Result::unwrap)
}

/// The average bitrate encoded in a synthesized AAC init segment.
Expand Down Expand Up @@ -1914,6 +1914,20 @@ mod tests {
video0.push(&gop(sequence)).unwrap();
video1.push(&gop(sequence)).unwrap();
}
let media = broadcast.create_track("video0", timed()).unwrap();
for sequence in 0..4 {
let mut group = media.append_group().unwrap();
let frame = vp8_frame(sequence * 2_000_000, true);
hang::container::Frame {
timestamp: frame.timestamp,
payload: frame.payload,
}
.write_to(&mut group)
.unwrap();
group.finish().unwrap();
}
media.finish().unwrap();

let broadcaster = Broadcaster::new(crate::resolved(&origin, "live").await, Config::default());
broadcaster.advertised().await;
let rendition = broadcaster.rendition(Kind::Video, "video0").unwrap();
Expand Down Expand Up @@ -4050,7 +4064,7 @@ mod tests {
let added = tokio::time::timeout(Duration::from_secs(5), renditions.next())
.await
.expect("a rendition is discovered");
let Some(renditions::Event::Added(_rendition)) = added else {
let Ok(Some(renditions::Event::Added(_rendition))) = added else {
panic!("expected an Added event");
};

Expand All @@ -4061,7 +4075,7 @@ mod tests {
.await
.expect("the cursor resolves after the broadcaster drops");
assert!(
matches!(next, Some(renditions::Event::Removed { .. })),
matches!(next, Ok(Some(renditions::Event::Removed { .. }))),
"dropping the broadcaster releases its renditions to a live cursor"
);

Expand All @@ -4070,14 +4084,14 @@ mod tests {
let ended = tokio::time::timeout(Duration::from_secs(5), renditions.next())
.await
.expect("the cursor terminates after the broadcaster drops");
assert!(ended.is_none(), "the cursor ends once the broadcaster is gone");
assert!(matches!(ended, Err(crate::Error::IncompleteRecording)));

drop((catalog, media, registration, broadcast));
}

// The record path: the renditions cursor yields the video rendition, its segment cursor
// yields each finalized segment WITH its media, and closing the publisher drains the final
// segment and then ends both cursors.
// yields each finalized segment with its media, and losing the publisher mid-group fails
// both cursors instead of completing a truncated recording.
#[tokio::test]
async fn record_cursors_yield_renditions_and_segments() {
let origin = produce_origin();
Expand Down Expand Up @@ -4110,6 +4124,7 @@ mod tests {
let rendition = match tokio::time::timeout(Duration::from_secs(5), renditions.next())
.await
.expect("a rendition is discovered")
.unwrap()
{
Some(renditions::Event::Added(rendition)) => rendition,
other => panic!("expected an Added event, got {:?}", other.is_some()),
Expand Down Expand Up @@ -4149,11 +4164,150 @@ mod tests {
end.map(|segment| segment.map(|segment| segment.segment))
);

// The renditions cursor also ends once the source closes.
// The renditions cursor also fails when the source closes abruptly.
let ended = tokio::time::timeout(Duration::from_secs(5), renditions.next())
.await
.expect("renditions cursor resolves");
assert!(ended.is_none(), "renditions cursor ends when the broadcast closes");
assert!(ended.is_err(), "an abrupt broadcast close fails the renditions cursor");
}

#[tokio::test(start_paused = true)]
async fn a_segment_cursor_rejects_an_unfinished_window() {
let test = Timelines::new();
let rendition = test.rendition("video0");
let mut cursor = rendition.segments();
rendition.close();
assert!(matches!(cursor.next().await, Err(crate::Error::IncompleteRecording)));
assert!(matches!(cursor.next().await, Err(crate::Error::IncompleteRecording)));
}

#[tokio::test(start_paused = true)]
async fn a_failed_catalog_fails_unfinished_segment_windows() {
let test = Timelines::new();
let rendition = test.rendition("video0");
let mut segments = rendition.segments();
test.renditions.finish(Err(crate::Error::NoVariants));
assert!(matches!(segments.next().await, Err(crate::Error::NoVariants)));
}

#[tokio::test(start_paused = true)]
async fn a_slow_cursor_refuses_evicted_rows_before_its_first_read() {
let test = Timelines::new();
let rendition = test.rendition("video0");
let mut slow = rendition.segments();
let reference = reference(&test.catalog, None).unwrap();
for sequence in 0..20 {
test.renditions.fanout().push(
sequence,
moq_mux::timeline::Entry::decode(gop(sequence), moq_net::Timescale::MILLI).unwrap(),
&reference,
);
}
assert!(rendition.snapshot().media_sequence > 0);
assert!(matches!(slow.next().await, Err(crate::Error::RecordingLagged)));
assert!(matches!(slow.next().await, Err(crate::Error::RecordingLagged)));
}

#[tokio::test(start_paused = true)]
async fn a_recording_refuses_missing_media_but_skips_rendition_gaps() {
let test = Timelines::new();
let (_, mut video0) = test.publish("video0");
let (_, mut video1) = test.publish("video1");
let media = test.media("video0", 2);
let sibling = test.media("video1", 2);
media.finish().unwrap();
sibling.finish().unwrap();
for sequence in 0..3 {
video0.push(&gop(sequence)).unwrap();
}
// The sibling timeline explicitly has a gap for segment 0, then covers segment 1.
video1
.push(&hang::timeline::Record::new(
0,
2_000,
2_000,
hang::timeline::Position::group(1),
hang::timeline::Position::group(2),
))
.unwrap();
video1.finish().unwrap();
video0.finish().unwrap();
test.finished("video0").await;
let rendition = test.rendition("video0");
let mut cursor = rendition.segments();
assert_eq!(cursor.next().await.unwrap().unwrap().segment, 0);
assert_eq!(cursor.next().await.unwrap().unwrap().segment, 1);
assert!(matches!(
cursor.next().await,
Err(crate::Error::SegmentUnavailable { segment: 2 })
));
assert!(matches!(
cursor.next().await,
Err(crate::Error::SegmentUnavailable { segment: 2 })
));

let rendition = test.rendition("video1");
let mut cursor = rendition.segments();
assert_eq!(cursor.next().await.unwrap().unwrap().segment, 1);
assert!(cursor.next().await.unwrap().is_none());
}

#[tokio::test(start_paused = true)]
async fn catalog_changes_report_removal_and_drain_old_cursors() {
for reconfigure in [false, true] {
let mut test = Timelines::new();
let (_, mut timeline) = test.publish("video0");
let _media = test.media("video0", 2);
timeline.push(&gop(0)).unwrap();
test.listed("video0", 0).await;
let mut changes = test.renditions.subscribe();
let mut old = None;
for _ in 0..2 {
if let Some(renditions::Event::Added(rendition)) = changes.next().await.unwrap()
&& rendition.name == "video0"
{
old = Some(rendition.segments());
}
}
let mut old = old.unwrap();
if reconfigure {
test.catalog.video.renditions.get_mut("video0").unwrap().coded_width = Some(1280);
} else {
test.catalog.video.renditions.remove("video0");
}
let source = moq_mux::Source::new(produce_origin().consume(), test.broadcast.consume());
test.renditions.sync(&source, &test.catalog);
assert!(
matches!(changes.next().await.unwrap(), Some(renditions::Event::Removed { name, .. }) if name == "video0")
);
assert_eq!(old.next().await.unwrap().unwrap().segment, 0);
assert!(old.next().await.unwrap().is_none());
if reconfigure {
assert!(
matches!(changes.next().await.unwrap(), Some(renditions::Event::Added(rendition)) if rendition.name == "video0")
);
}
}
}

#[tokio::test(start_paused = true)]
async fn a_removed_rendition_keeps_draining_its_delayed_timeline() {
let mut test = Timelines::new();
let (_, mut reference) = test.publish("video0");
let (_, mut sibling) = test.publish("video1");
let _media = test.media("video1", 1);
reference.push(&gop(0)).unwrap();
test.listed("video0", 0).await;
let rendition = test.rendition("video1");
let mut cursor = rendition.segments();
test.catalog.video.renditions.remove("video1");
let source = moq_mux::Source::new(produce_origin().consume(), test.broadcast.consume());
test.renditions.sync(&source, &test.catalog);
// Removal closes the window, while its existing own-timeline watcher stays alive.
sibling.push(&gop(0)).unwrap();
sibling.finish().unwrap();
assert_eq!(cursor.next().await.unwrap().unwrap().segment, 0);
assert!(cursor.next().await.unwrap().is_none());
}

/// A two-second GOP record: record `n` covers group `n` from `2n` seconds.
Expand Down
Loading
Loading