From 47150619ec3a2893c70829e157434ccaa164ef64 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Thu, 24 Sep 2026 22:18:41 -0700 Subject: [PATCH 1/2] fix(net): hold a parked track's warm cache until the upstream confirms it A relay front keeps what an unread track delivered as a warm cache for 30s. A returning reader was served that cache before the fresh upstream copy answered, judged only against the cache's own frozen edge, so a rejoining player got seconds-old groups first. Lite-06 made it worse by resolving the start from the budget (several stale groups instead of one). Now the re-splice asks the upstream for the end of the cache's newest group, and live readers hold the cache until the copy's SUBSCRIBE_START resolves: at the cache's edge the cache is spliced in (an open group continues from its next frame), past it the cache is skipped. A local source keeps its own cache, so it releases instead of parking. Co-Authored-By: Claude Opus 5.5 --- doc/lib/c/index.md | 2 +- rs/moq-net/src/lite/subscriber.rs | 15 +- rs/moq-net/src/model/front.rs | 10 +- rs/moq-net/src/model/origin.rs | 361 ++++++++++++++++++++++++++++-- rs/moq-net/src/model/resume.rs | 114 +++++++++- rs/moq-net/src/model/track.rs | 57 ++++- rs/moq-tokio/tests/broadcast.rs | 190 ++++++++++++++++ 7 files changed, 717 insertions(+), 32 deletions(-) diff --git a/doc/lib/c/index.md b/doc/lib/c/index.md index 15de2348946..efae3e08d5a 100644 --- a/doc/lib/c/index.md +++ b/doc/lib/c/index.md @@ -40,7 +40,7 @@ and `target/include/moq.h`. - **Encoded video metadata.** `moq_video_init.hint` is a zero-initialized `moq_video_hint` with `has_*` flags for coded dimensions, bitrate (bits per second), frame rate, and latency preference. Hints seed a video codec track's catalog; detected dimensions take precedence. - **Client config.** A zeroed `moq_client_config` means the defaults for every knob, which is what lets a new one be appended without disturbing callers. Fields cover protocol (`versions`), TLS (`tls_fingerprints`, `tls_roots`, `tls_cert`/`_key`, `tls_host_name`), transport (`bind`, `connect_timeout_us`, the Happy Eyeballs delays, `websocket_enabled`/`_delay_us`), and tuning (reconnect backoff, `quic_*`). Every duration is in microseconds. A knob whose default isn't zero carries a `has_*` flag, so setting `backoff_timeout_us = 0` needs `has_backoff_timeout = true` to mean "retry forever" rather than "use the default". `moq_client_defaults()` reports what a NULL config dials with. - **Server.** `moq_server_listen` binds before it returns (a bad address or certificate fails there) and hands each incoming session to `on_request` as a request handle. Read `moq_session_request_path` and `_query` to route and authenticate, then `moq_session_request_accept` (a session handle, with origins like `moq_session_connect`) or `moq_session_request_reject` with an HTTP-style code (401 and 403 become the protocol's unauthorized close). An accepted session reports `1` once SETUP completes and never reconnects. `moq_server_addr` reports an ephemeral port and `moq_server_fingerprints` the hashes a client pins for a `tls_generate` certificate. `moq_server_close` stops listening; its terminal callback fires once the sockets are released. -- **Demand.** A watcher on a published track (`moq_publish_track_demand`, `moq_publish_media_demand`, `moq_encode_video_demand`, `moq_encode_audio_demand`) calls `on_demand` with `MOQ_DEMAND_USED` or `MOQ_DEMAND_UNUSED` right away and again on every change, so an encoder on a battery-powered device runs only while someone is watching. The first call is the current state, so a track that went unused before the watcher existed still reports it. `moq_publish_demand_cancel` stops it; the terminal callback still fires. A container has no single demand and is refused. Demand follows the last real subscriber: an origin that served the track drops its source copy on the unused edge and keeps only the finished groups it already cached warm for 30 seconds, so the cache linger does not delay the unused edge. +- **Demand.** A watcher on a published track (`moq_publish_track_demand`, `moq_publish_media_demand`, `moq_encode_video_demand`, `moq_encode_audio_demand`) calls `on_demand` with `MOQ_DEMAND_USED` or `MOQ_DEMAND_UNUSED` right away and again on every change, so an encoder on a battery-powered device runs only while someone is watching. The first call is the current state, so a track that went unused before the watcher existed still reports it. `moq_publish_demand_cancel` stops it; the terminal callback still fires. A container has no single demand and is refused. Demand follows the last real subscriber: an origin that served the track drops its source copy on the unused edge, so a cache linger never delays it. Only a relay keeps what it already delivered warm for 30 seconds, and a returning subscriber is served that cache only once the publisher confirms it is still current. - **Requests.** `moq_publish_dynamic` serves subscriptions to tracks the broadcast never declared: each arrives as a request handle, read its name with `moq_track_request_name`, then `moq_track_request_accept` (a raw track handle), `moq_track_request_video` / `_audio` (the media handle `moq_publish_video` / `_audio` return), or `moq_track_request_abort` with an application code the subscriber sees. Without a live handler an unknown name is refused. `moq_publish_track_dynamic` does the same for fetches of groups a track no longer has cached, delivered as `moq_group_request_*` (`sequence`, `priority`, `frame_start`); `moq_group_request_accept` starts the producer at `frame_start` so written frames keep their group indices. Register it with `moq_track_request_dynamic` before accepting a track that was itself requested by a fetch, so that pending group survives the transition. Both handlers stop with `moq_publish_dynamic_cancel`. - **Everything the bindings can do** ([list](/lib/#what-every-binding-can-do)): media publish and consume with the catalog managed for you, raw pixels and PCM with the codec inside (`moq_encode_video`, `moq_encode_audio`, and the `moq_decode_*` mirrors), raw tracks with timestamps and datagrams, JSON snapshot and stream tracks, group fetch, catalog sections, shared video properties, and stalled hints. The three advertising operations are `moq_origin_create_broadcast` (unannounced producer, invisible to everyone), `moq_publish_announce` / `moq_publish_unannounce` (exact-path advertisement), and `moq_origin_dynamic` (a claim over a path prefix and everything beneath it; `""` for everything). A route is a capability, not an inventory. `moq_origin_announced` takes a literal prefix and an optional relative pattern filter; `moq_announce_update.prefix` stays relative to the origin, while `captures` reports what each wildcard matched when `has_captures` is true. Paths with a `.`-prefixed segment below the prefix are [hidden](/concept/moq-lite#hidden-broadcasts); name the dot segment in `prefix` to list them. diff --git a/rs/moq-net/src/lite/subscriber.rs b/rs/moq-net/src/lite/subscriber.rs index 387ba189b8d..9013f381ccb 100644 --- a/rs/moq-net/src/lite/subscriber.rs +++ b/rs/moq-net/src/lite/subscriber.rs @@ -2777,8 +2777,13 @@ impl TrackServe { // The floor tracks the requested start until this subscription's own // SUBSCRIBE_START refines it: the peer never serves below the request, a // previous subscription's declaration must not outlive its demand, and - // live-edge demand (None) starts with no floor at all. - let _ = producer.start_at(subscription.start.map(|start| start.group)); + // live-edge demand (None) starts with no floor at all. Lite-06+ only resolves + // the start with its SUBSCRIBE_START, so until then the floor is just a request. + let floor = subscription.start.map(|start| start.group); + let _ = match self.subscriber.version.resolves_start() { + true => producer.request_start(floor), + false => producer.start_at(floor), + }; tracing::info!(id, broadcast = %self.subscriber.log_path(&self.path), track = %self.name, "subscribe started"); @@ -3200,6 +3205,12 @@ impl ServeLoop { // request's own fetch gate, and a cache-miss fetch queued while TRACK_INFO // was in flight would be drained as NotFound in the gap. let dynamic = request.dynamic(); + // Lite-06+ resolves each subscription's start from its budget, so the floor + // the SUBSCRIBE asks for is not where delivery begins until SUBSCRIBE_START says. + let request = match serve.subscriber.version.resolves_start() { + true => request.resolving_start(), + false => request, + }; let serving = request.accept(info); Self { serving, diff --git a/rs/moq-net/src/model/front.rs b/rs/moq-net/src/model/front.rs index c651082fb75..df6a582a656 100644 --- a/rs/moq-net/src/model/front.rs +++ b/rs/moq-net/src/model/front.rs @@ -105,7 +105,8 @@ pub(super) enum Action { /// Drop the source copy of `track` but keep the delivered groups spliced, /// so resume stays seamless while nobody reads. Park { track: Arc }, - /// Drop the delivered groups of `track` too: the linger expired unread. + /// Drop the source copy of `track` and every delivered group: the linger expired + /// unread, or the source is local and keeps its own cache. Release { track: Arc }, /// The logical track completed. Finish { track: Arc }, @@ -587,6 +588,13 @@ impl Front { }; track.used = false; match track.state { + // A local source keeps its own cache, so a warm copy would only be a staler + // duplicate of it: drop the copy outright, and a returning reader re-splices + // the source and reads its cache against the real live edge. + TrackState::Spliced { .. } if self.identity == Identity::Local => { + track.state = TrackState::Idle; + actions.push(Action::Release { track: name }); + } // Drop the copy so the source goes idle at once; the delivered // groups stay spliced for the linger. TrackState::Spliced { .. } => { diff --git a/rs/moq-net/src/model/origin.rs b/rs/moq-net/src/model/origin.rs index 8beaa32a753..f0ae48583da 100644 --- a/rs/moq-net/src/model/origin.rs +++ b/rs/moq-net/src/model/origin.rs @@ -1,4 +1,4 @@ -use crate::{broadcast, cache, stats, track}; +use crate::{broadcast, cache, group, stats, track}; use kio::Pollable; use std::{ cmp::Reverse, @@ -1815,6 +1815,9 @@ const TRACK_IDLE_LINGER: Duration = Duration::from_secs(30); struct WarmCopy { track: track::Producer, _dynamic: track::Dynamic, + /// The newest group, where the copy spliced after this one picks up (see + /// [`TrackIo::head`]). + edge: Option, } impl Drop for WarmCopy { @@ -1823,25 +1826,137 @@ impl Drop for WarmCopy { } } -/// Cache `source`'s finished groups on a new local track the origin owns. -fn warm_copy(source: &track::Consumer) -> Option { +/// A warm copy's newest group, which the copy spliced after it continues. +/// +/// Kept past that splice: an open one must stay open for the continuation (dropping +/// an unfinished producer clears its frames), and either kind supplies the head the +/// continuation lacks when the next park rebuilds the group. Nothing will ever finish +/// an open one, so it aborts on drop. +struct WarmGroup(group::Producer); + +impl Drop for WarmGroup { + fn drop(&mut self) { + if !self.0.is_finished() { + let _ = self.0.clone().abort(Error::Cancel); + } + } +} + +/// Cache what `source` delivered on a new local track the origin owns: its complete +/// groups, and its open live edge rebuilt from the frames already delivered. +/// +/// `head` is the previous park's edge. A copy spliced after a warm cache continues its +/// edge group from the next frame, so its copy of that group lacks the head, which +/// `head` supplies. +fn warm_copy(source: &track::Consumer, head: Option<&WarmGroup>) -> Option { let info = source.cached_info()?; let mut track = track::Producer::new(Arc::new(source.broadcast().clone()), source.name(), info); - for (group, visible) in source.cached_groups() { - // An open group is left for the re-splice to deliver whole. Dropping the source - // copy resets it mid-transfer, and its dead head would anchor the next takeover - // mid-group, asking upstream for a tail no returning reader can use. - if group.is_finished() { - let _ = track.adopt_group(group, visible); + let head = head.map(|head| &head.0); + let latest = source.latest(); + let groups = source.cached_groups(); + let mut edge = None; + + // A spliced copy hides the group it continued (its halves sit in two segments), so + // carry the previous edge over when the copy has no version of it at all. First, + // since it arrived before anything the copy holds. + if let Some(head) = head + && !groups.iter().any(|(group, _)| group.sequence == head.sequence) + { + let is_latest = latest.is_none_or(|latest| head.sequence >= latest); + if head.is_finished() { + let _ = track.adopt_group(head.clone(), true); + if is_latest { + edge = Some(WarmGroup(head.clone())); + } + } else if is_latest { + edge = warm_rebuild(&track, head, None); + } + } + + for (group, visible) in groups { + let finished = group.is_finished(); + let whole = group.live_first_frame() == Some(0); + let is_latest = visible && Some(group.sequence) == latest; + let warm = if finished && whole { + let _ = track.adopt_group(group.clone(), visible); + Some(WarmGroup(group)) + } else if (finished || is_latest) && (whole || head.is_some_and(|head| head.sequence == group.sequence)) { + // Dropping the source copy resets an open live edge mid-transfer, so rebuild + // it from the frames already delivered: the re-splice asks for the next + // frame, and a group that stays open for good (a JSON log in group 0) + // continues instead of being re-sent whole on every resume. A continuation + // is rebuilt whole from the previous edge's head the same way. + warm_rebuild(&track, &group, head) + } else { + // Mid-transfer backlog, or a continuation with no head to complete it. + None + }; + if is_latest { + edge = warm; } } let dynamic = track.dynamic(); Some(WarmCopy { track, _dynamic: dynamic, + edge, }) } +/// Rebuild `live` on `track` from its delivered frames, prefixed by `head`'s when `live` +/// only holds a continuation of it. Finished like `live`, or left open. +fn warm_rebuild(track: &track::Producer, live: &group::Producer, head: Option<&group::Producer>) -> Option { + // A continuation holds nothing below its offset. + let mut start = live.live_first_frame()? as u64; + let mut tail = live.consume(); + tail.start_at(start); + let mut frames = Vec::new(); + if start > 0 + && let Some(head) = head.filter(|head| head.sequence == live.sequence) + && let Some(head_start) = head.live_first_frame() + { + let mut head = head.consume(); + head.start_at(head_start as u64); + while head.index() < start { + match head.poll_read_frame(&kio::Waiter::noop()) { + Poll::Ready(Ok(Some(frame))) => frames.push(frame), + _ => break, + } + } + match head.index() == start { + true => start = head_start as u64, + // The head doesn't reach the continuation: keep only the continuation. + false => frames.clear(), + } + } + while let Poll::Ready(Ok(Some(frame))) = tail.poll_read_frame(&kio::Waiter::noop()) { + frames.push(frame); + } + if frames.is_empty() { + return None; + } + + // Wrapped first, so a failed write aborts it rather than dropping it unfinished. + let rebuilt = WarmGroup( + track + .create_group(group::Info { + sequence: live.sequence, + }) + .ok()?, + ); + let mut writer = rebuilt.0.clone(); + if start > 0 { + writer.start_at(start).ok()?; + } + for frame in frames { + writer.write_frame(frame.timestamp, frame.payload).ok()?; + } + if live.is_finished() { + writer.finish().ok()?; + } + Some(rebuilt) +} + /// Everything [`run_front`] owns, queued by [`Consumer::request_broadcast`]. struct FrontTask { /// The route table the front selects from. @@ -1879,6 +1994,9 @@ struct TrackIo { /// Delivered groups kept after the copy was dropped, so resume stays spliced /// through the linger without pinning the source as a reader. warm: Option, + /// The last warm copy's newest group, outliving it so the copy spliced after it + /// can continue the group (see [`WarmGroup`]). Released at the next park. + head: Option, /// Whether the track had a reader as of the last demand edge. used: bool, } @@ -2106,7 +2224,7 @@ async fn run_front(task: FrontTask) { tracks.remove(&name); continue; } - io.warm = None; + io.head = io.warm.take().and_then(|mut warm| warm.edge.take()); // The new segment has produced nothing yet: this is the // edge the copy is asked to advance. io.edge = io.resume.resume_position(); @@ -2118,24 +2236,25 @@ async fn run_front(task: FrontTask) { // Drop the source copy so its producer goes idle at once; keep // the groups it delivered on a local track so resume stays // spliced until the linger expires. - let warm = warm_copy(©); + let warm = warm_copy(©, io.head.as_ref()); drop(copy); - if io.resume.release().is_err() { + io.head = None; + let parked = match &warm { + Some(warm) => io.resume.park(&warm.track), + None => io.resume.release(), + }; + if parked.is_err() { tracks.remove(&name); continue; } - if let Some(warm) = warm { - if let Err(err) = io.resume.takeover(&warm.track) { - let _ = io.resume.abort(err); - tracks.remove(&name); - continue; - } - io.warm = Some(warm); - } + io.warm = warm; } Action::Release { track: name } => { let Some(io) = tracks.get_mut(&name) else { continue }; + // A local source releases straight from the spliced copy. + io.copy = None; io.warm = None; + io.head = None; if io.resume.release().is_err() { tracks.remove(&name); } @@ -2252,6 +2371,7 @@ async fn run_front(task: FrontTask) { copy: None, edge: None, warm: None, + head: None, used: false, }, ); @@ -4615,6 +4735,207 @@ mod tests { assert!(matches!(subscription.recv_group().await, Ok(None)), "ends cleanly"); } + /// A front resolved through a served route, as a relay's upstream session serves + /// one: the upstream broadcast's track requests arrive on the returned handle. + async fn served_front() -> (Dynamic, broadcast::Producer, broadcast::Dynamic, broadcast::Consumer) { + let producer = origin(1).produce(); + let consumer = producer.consume(); + let server = producer + .dynamic("room/alice", Route::default().with_hops(hops(&[10]))) + .unwrap(); + let pending = consumer.request_broadcast("room/alice"); + let upstream = broadcast::Info::new().produce(); + let dynamic = upstream.dynamic(); + queued(&server).await.accept(&upstream); + let resolved = pending.await.expect("resolves"); + (server, upstream, dynamic, resolved) + } + + /// A reader returning to a parked track waits for the fresh copy to resolve its + /// start, and skips the warm cache when the copy resolves past it: the source + /// judged the groups in between stale, so the older cache is stale too. Without the + /// hold the reader was handed the whole warm cache first, seconds behind live. + #[tokio::test] + async fn returning_reader_skips_a_warm_cache_the_copy_resolved_past() { + let ms = |v: u64| crate::Timestamp::from_millis(v).unwrap(); + let (_server, _upstream, mut dynamic, resolved) = served_front().await; + let budget = track::Subscription::default().with_max_age(Duration::from_millis(100)); + + let track = resolved.track("audio").unwrap(); + let b = budget.clone(); + let subscribing = tokio::spawn(async move { track.subscribe(b).await }); + let request = tokio::time::timeout(Duration::from_secs(1), dynamic.requested_track()) + .await + .expect("the front asked the source") + .expect("request"); + let source = request.resolving_start().accept(None); + for seq in 0..4u64 { + let mut group = source.create_group(seq.into()).unwrap(); + group.write_frame(ms(seq * 20), b"old".as_ref()).unwrap(); + group.finish().unwrap(); + } + let mut subscription = subscribing.await.unwrap().expect("subscribe"); + subscription.recv_group().await.unwrap().expect("the live group"); + drop(subscription); + tokio::time::timeout(Duration::from_secs(1), source.unused()) + .await + .expect("parked") + .expect("source open"); + drop(source); + + let track = resolved.track("audio").unwrap(); + let subscribing = tokio::spawn(async move { track.subscribe(budget).await }); + let request = tokio::time::timeout(Duration::from_secs(1), dynamic.requested_track()) + .await + .expect("the front asked the source again") + .expect("request"); + let mut source = request.resolving_start().accept(None); + let mut subscription = subscribing.await.unwrap().expect("resubscribe"); + + // The copy has not resolved its start: nothing is handed out yet. + assert!( + tokio::time::timeout(Duration::from_millis(50), subscription.recv_group()) + .await + .is_err(), + "the warm cache was served before the copy resolved its start" + ); + + // The source resolves past the floor (lite-06 skipped 4..20 as stale). + source.start_at(20).unwrap(); + let mut group = source.create_group(20u64.into()).unwrap(); + group.write_frame(ms(2000), b"new".as_ref()).unwrap(); + group.finish().unwrap(); + let group = subscription.recv_group().await.unwrap().expect("the live group"); + assert_eq!(group.sequence, 20, "a stale warm group was served"); + } + + /// A warm cache whose newest group finished still resumes when the source has + /// nothing newer: the re-splice asks for that group's tail, which a source that + /// resolves starts lazily (with its first served group) can answer at once. Asking + /// past it left a returning catalog reader waiting for the next catalog change. + #[tokio::test] + async fn returning_reader_replays_a_current_warm_cache() { + let (_server, _upstream, mut dynamic, resolved) = served_front().await; + + let track = resolved.track("catalog").unwrap(); + let subscribing = tokio::spawn(async move { track.subscribe(None).await }); + let request = tokio::time::timeout(Duration::from_secs(1), dynamic.requested_track()) + .await + .expect("the front asked the source") + .expect("request"); + let source = request.resolving_start().accept(None); + let mut group = source.create_group(0u64.into()).unwrap(); + group.write_frame(crate::Timestamp::ZERO, b"snapshot".as_ref()).unwrap(); + group.finish().unwrap(); + let mut subscription = subscribing.await.unwrap().expect("subscribe"); + subscription.recv_group().await.unwrap().expect("the catalog"); + drop(subscription); + tokio::time::timeout(Duration::from_secs(1), source.unused()) + .await + .expect("parked") + .expect("source open"); + drop(source); + + let track = resolved.track("catalog").unwrap(); + let subscribing = tokio::spawn(async move { track.subscribe(None).await }); + let request = tokio::time::timeout(Duration::from_secs(1), dynamic.requested_track()) + .await + .expect("the front asked the source again") + .expect("request"); + let mut source = request.resolving_start().accept(None); + let mut subscription = subscribing.await.unwrap().expect("resubscribe"); + + // The source still has group 0 as its newest: it serves the empty tail, and + // that is when its start resolves. + let reading = tokio::spawn(async move { + let mut group = subscription.recv_group().await.unwrap().expect("the catalog"); + assert_eq!(group.sequence, 0); + group.read_frame().await.unwrap().expect("the snapshot").payload + }); + tokio::task::yield_now().await; + assert_eq!( + source.subscription().and_then(|sub| sub.start), + Some(track::Position { group: 0, frame: 1 }), + "the re-splice asked past the cached catalog" + ); + source.start_at(0).unwrap(); + let mut tail = source.create_group(0u64.into()).unwrap(); + tail.start_at(1).unwrap(); + tail.finish().unwrap(); + let payload = tokio::time::timeout(Duration::from_secs(1), reading) + .await + .expect("the returning reader never got the catalog") + .unwrap(); + assert_eq!(&payload[..], b"snapshot"); + } + + /// A group that stays open for good (a JSON log in group 0) survives a park: the + /// returning reader gets the frames delivered before it from the warm cache, and the + /// re-splice asks the source only for the frames after them, across repeated parks. + #[tokio::test] + async fn returning_reader_continues_an_open_warm_group() { + let (_server, _upstream, mut dynamic, resolved) = served_front().await; + + async fn read(group: &mut group::Consumer) -> Vec { + let frame = tokio::time::timeout(Duration::from_secs(1), group.read_frame()) + .await + .expect("frame") + .unwrap() + .expect("group ended"); + frame.payload.to_vec() + } + + let mut expect: Vec<&[u8]> = Vec::new(); + let mut floor: Option = None; + for (round, payload) in [b"a".as_ref(), b"b", b"c"].into_iter().enumerate() { + let track = resolved.track("log").unwrap(); + let subscribing = tokio::spawn(async move { track.subscribe(None).await }); + let request = tokio::time::timeout(Duration::from_secs(1), dynamic.requested_track()) + .await + .expect("the front asked the source") + .expect("request"); + let mut source = request.resolving_start().accept(None); + let mut subscription = subscribing.await.unwrap().expect("subscribe"); + + // The source resolves at the floor's group, continuing group 0. + source.start_at(0).unwrap(); + let mut group = source.create_group(0u64.into()).unwrap(); + if let Some(floor) = floor { + group.start_at(floor.frame).unwrap(); + } + group.write_frame(crate::Timestamp::ZERO, payload).unwrap(); + expect.push(payload); + + let mut reading = tokio::time::timeout(Duration::from_secs(1), subscription.recv_group()) + .await + .expect("group 0") + .unwrap() + .expect("track ended"); + assert_eq!(reading.sequence, 0); + for frame in &expect { + assert_eq!(read(&mut reading).await, *frame, "round {round}"); + } + assert_eq!( + source.subscription().and_then(|sub| sub.start), + floor, + "round {round} asked for the wrong continuation" + ); + + drop(reading); + drop(subscription); + tokio::time::timeout(Duration::from_secs(1), source.unused()) + .await + .expect("parked") + .expect("source open"); + drop(group); + drop(source); + floor = Some(track::Position { + group: 0, + frame: expect.len() as u64, + }); + } + } + /// The same holds for a reader returning to a parked track: its warm cache /// does not stand in for the copy it is waiting on. #[tokio::test] diff --git a/rs/moq-net/src/model/resume.rs b/rs/moq-net/src/model/resume.rs index da250e34bee..2fa3d0c0482 100644 --- a/rs/moq-net/src/model/resume.rs +++ b/rs/moq-net/src/model/resume.rs @@ -43,6 +43,12 @@ struct Segment { end: Option, /// The underlying per-session track. track: track::Consumer, + /// Where the source is asked to start: `start`, except after a warm cache (see + /// [`Segment::warm_edge`]). + ask: Option, + /// A parked track's warm cache (see [`Producer::park`]), which live readers hold + /// until the next segment's copy resolves its start. + warm: bool, } impl Segment { @@ -72,6 +78,23 @@ impl Segment { }) } + /// Where a copy spliced after this warm cache is asked to start: the end of the + /// cache's newest group, even a finished one, rather than the head of the next. + /// + /// A source only resolves a start once it has a group to serve, so asking past its + /// newest group would leave a returning reader waiting on the next one. Asking for the + /// newest group's tail lets a source that is still there answer at once, and one that + /// moved on answer past it. Only the ask moves: the boundary stays where the cache + /// stops, so whatever the source sends for a finished group's empty tail (a FIN, or a + /// reset from a publisher that refuses empty ranges) sits outside the new segment. + fn warm_edge(&self) -> Option { + let group = self.track.peek_latest()?; + Some(Position { + group: group.sequence, + frame: group.frame_count() as u64, + }) + } + /// The newest cached group this segment serves below `before` (exclusive), or its /// newest cached group at all when `before` is `None`. /// @@ -234,6 +257,8 @@ impl ResumeState { start, end: None, track, + ask: start, + warm: false, }); self.epoch += 1; self.prune(); @@ -339,11 +364,23 @@ impl Producer { // segments at all) there is nothing to splice around, so the replacement // replaces them outright and starts unbounded, exactly like a first splice. // `switch` rejects a `None` start once a segment exists, hence the clear. + let ask = state + .segments + .last() + .filter(|last| last.warm) + .and_then(Segment::warm_edge); let start = state.resume_position(); if start.is_none() { state.segments.clear(); } - state.switch(track, start) + state.switch(track, start)?; + if let Some(ask) = ask + && start.is_some() + && let Some(last) = state.segments.last_mut() + { + last.ask = Some(ask); + } + Ok(()) } /// Drop every segment, releasing the underlying tracks while keeping the @@ -370,6 +407,30 @@ impl Producer { Ok(()) } + /// Replace every segment with `warm`, a cache of what they delivered, for a track + /// nobody reads anymore: [`Self::release`] followed by an unbounded first splice. + /// + /// Live readers hold the cache until the next [`Self::takeover`]'s copy resolves + /// where its feed starts. A copy that picks up at the cache's edge proves the cache + /// still leads into the live feed, so it is read as usual (and a group the cache + /// holds open continues from its next frame). A copy that starts past the edge + /// skipped groups its source already judged stale, so the older cache is stale too + /// and live readers skip it. + pub(crate) fn park(&mut self, warm: impl super::origin_impl::Consume) -> Result<()> { + let track = warm.consume(); + let mut state = self.state.write().map_err(|_| Error::Dropped)?; + if state.finished || state.abort.is_some() { + return Err(Error::Closed); + } + state.segments.clear(); + state.pruned = None; + state.switch(track, None)?; + if let Some(segment) = state.segments.last_mut() { + segment.warm = true; + } + Ok(()) + } + /// Whether any segment is spliced in, and so whether there is anything for /// [`Self::release`] to drop. /// @@ -1250,6 +1311,8 @@ struct SegmentSub { id: u64, start: Option, end: Option, + /// Where the source is asked to start; see [`Segment::ask`]. + ask: Option, sub: SubState, /// A completed segment's cursor, retained while parked groups may need their /// max age budget re-evaluated after the outer cap rises. @@ -1264,6 +1327,16 @@ struct SegmentSub { /// the lowest is re-offered first; holding them here (rather than blocking on /// the first) keeps in-range groups that arrive behind a capped one flowing. parked: BTreeMap, + /// Set while this is a warm segment (see [`Producer::park`]) that has not been + /// cleared for live reads: the copy spliced after it, once there is one. + warm: Option, +} + +/// A warm segment waiting on the copy spliced after it; see [`Subscriber::poll_activate`]. +struct Warm { + /// The cache's newest group. + edge: Option, + next: Option, } impl SegmentSub { @@ -1432,7 +1505,7 @@ impl Subscriber { }; self.last_prefs = prefs; for seg in &mut self.segments { - let prefs = slice(&self.last_prefs, seg.start, seg.end); + let prefs = slice(&self.last_prefs, seg.ask, seg.end); if let Some(sub) = seg.stale_sub_mut() { let _ = sub.update(prefs); } @@ -1499,9 +1572,14 @@ impl Subscriber { self.segments.retain(|s| !s.retired()); let anchor = self.anchor_end(); - for segment in segments { + let nexts: Vec<_> = segments.iter().skip(1).map(|next| Some(next.track.clone())).collect(); + let nexts = nexts.into_iter().chain(std::iter::once(None)); + for (segment, next) in segments.into_iter().zip(nexts) { match self.segments.iter_mut().find(|s| s.id == segment.id) { Some(existing) => { + if let Some(warm) = &mut existing.warm { + warm.next = next; + } if existing.end != segment.end { existing.end = segment.end; let cap = Self::stale_cap(existing, anchor); @@ -1512,7 +1590,7 @@ impl Subscriber { // read bounds stay on this subscriber (see `poll_recv_group`): // an inner `end_at` would park boundary-crossing groups in the // inner cursor, hiding the segment's completion. - let _ = sub.update(slice(&self.last_prefs, segment.start, segment.end)); + let _ = sub.update(slice(&self.last_prefs, segment.ask, segment.end)); } // A still-pending subscription picks the moved boundary up // when it activates (see `poll_activate`). Groups already handed @@ -1523,15 +1601,20 @@ impl Subscriber { None => { let sub = segment .track - .subscribe(slice(&self.last_prefs, segment.start, segment.end)); + .subscribe(slice(&self.last_prefs, segment.ask, segment.end)); self.segments.push(SegmentSub { id: segment.id, start: segment.start, end: segment.end, + ask: segment.ask, sub: SubState::Pending(sub), terminal: None, pruned: false, parked: BTreeMap::new(), + warm: segment.warm.then(|| Warm { + edge: segment.track.latest(), + next, + }), }); } } @@ -1657,6 +1740,25 @@ impl Subscriber { anchor_end: Option, waiter: &kio::Waiter, ) -> Poll<()> { + if matches!(seg.sub, SubState::Pending(_)) + && let Some(warm) = &seg.warm + { + // Nothing spliced after the cache yet, so nothing says it still leads into + // the live feed. A splice bumps the epoch, which wakes this waiter. + let Some(next) = &warm.next else { + return Poll::Pending; + }; + let start = ready!(next.poll_start(waiter)); + let edge = warm.edge; + seg.warm = None; + // The copy was asked for the cache's newest group and started past it: its + // source judged that group stale, so the older cache is no use to live reads. + if start.is_some_and(|start| edge.is_some_and(|edge| start > edge)) { + seg.complete(None); + return Poll::Ready(()); + } + } + if let SubState::Pending(pending) = &mut seg.sub { match ready!(pending.poll_ok(waiter)) { Ok(mut sub) => { @@ -1669,7 +1771,7 @@ impl Subscriber { // budget and floor, and this must not rewind past it. sub.raise_start_to(seg.first_group().max(min_sequence)); sub.set_stale_cap(Self::stale_cap(seg, anchor_end)); - let _ = sub.update(slice(prefs, seg.start, seg.end)); + let _ = sub.update(slice(prefs, seg.ask, seg.end)); seg.sub = SubState::Active(Box::new(sub)); } // The underlying track was rejected or closed: stall, not error. diff --git a/rs/moq-net/src/model/track.rs b/rs/moq-net/src/model/track.rs index 531a71db121..17bec9149bc 100644 --- a/rs/moq-net/src/model/track.rs +++ b/rs/moq-net/src/model/track.rs @@ -223,6 +223,12 @@ pub(crate) struct TrackState { // fetch can still create them. start_sequence: Option, + // Whether `start_sequence` is only the floor a subscription asked for, still + // waiting on the serving session to resolve where the live feed begins (a + // lite-06+ SUBSCRIBE_START). Readers that must know the resolved start (see + // [`Consumer::poll_start`]) wait on it; everything else treats the floor as usual. + start_pending: bool, + // Where production stopped, snapshotted when the cached groups are released (an // abort, or the last producer dropping). Computed live from the cache otherwise; // see [`Self::resume_position`]. @@ -986,8 +992,9 @@ impl TrackState { /// declaration: the signal is scoped to the current subscription's demand, /// which may legitimately move in either direction. `None` clears it (the /// demand dropped to the live edge, whose floor is unknown until declared). - fn set_start(&mut self, start_sequence: Option) { + fn set_start(&mut self, start_sequence: Option, pending: bool) { self.start_sequence = start_sequence; + self.start_pending = pending; } /// Record the exclusive final sequence, rejecting a re-finish or a boundary that @@ -1381,7 +1388,16 @@ impl Producer { /// still promised. Pass `None` to clear it, for demand at the live edge: /// its floor is unknown until the feed declares one. pub fn start_at(&mut self, sequence: impl Into>) -> Result<()> { - self.modify()?.set_start(sequence.into()); + self.modify()?.set_start(sequence.into(), false); + Ok(()) + } + + /// Declare the floor a subscription asked for while the serving session has yet to + /// resolve its start: nothing below `sequence` arrives, exactly as [`Self::start_at`], + /// but [`Consumer::poll_start`] keeps waiting until a later [`Self::start_at`] + /// resolves it. + pub(crate) fn request_start(&mut self, sequence: Option) -> Result<()> { + self.modify()?.set_start(sequence, true); Ok(()) } @@ -2318,6 +2334,28 @@ impl Consumer { }) } + /// Poll for the first group the live feed serves, once the serving session has + /// resolved it: `Some` for a declared start, `None` for none (the live edge, or a + /// source that never declares one). Parks while a lite-06+ session still owes its + /// SUBSCRIBE_START (see [`Producer::request_start`]); a closed track is ready with + /// whatever it last declared, since nothing will resolve it anymore. + pub(crate) fn poll_start(&self, waiter: &kio::Waiter) -> Poll> { + match &self.inner { + ConsumerKind::Plain(state) => { + let res = state.poll(waiter, |state| match state.start_pending && state.abort.is_none() { + true => Poll::Pending, + false => Poll::Ready(state.start_sequence), + }); + match res { + Poll::Ready(Ok(start)) => Poll::Ready(start), + Poll::Ready(Err(state)) => Poll::Ready(state.start_sequence), + Poll::Pending => Poll::Pending, + } + } + ConsumerKind::Spliced(_) => Poll::Ready(None), + } + } + /// The newest group, when it is already cached: resolved synchronously, without /// counting as a fetch or a delivery. The IETF publisher snapshots its frame count to /// resolve Largest Object; a group that is not immediately available reads as no edge. @@ -3908,6 +3946,10 @@ pub struct Request { // Ingress stats scope, threaded into the accepted [`Producer`]. Empty (no-op) // unless this request was reserved on a tagged broadcast. stats: stats::Scope, + + // The serving session resolves the start of each subscription itself, so the + // accepted track's start is unknown until it says (see [`Self::resolving_start`]). + resolving_start: bool, } impl Request { @@ -3924,9 +3966,19 @@ impl Request { alive, _dynamic: dynamic, stats: stats::Scope::default(), + resolving_start: false, } } + /// Mark the track as served by a session that resolves each subscription's start + /// (lite-06+), so [`Consumer::poll_start`] waits for its declaration instead of + /// reading the requested floor as the start. Applied atomically with + /// [`Self::accept`], before any reader can see the track. + pub(crate) fn resolving_start(mut self) -> Self { + self.resolving_start = true; + self + } + /// Attach an ingress stats scope, applied to the [`Producer`] on accept. Set by /// a tagged [`broadcast::Producer::reserve_track`]. pub(crate) fn with_stats(mut self, scope: stats::Scope) -> Self { @@ -3991,6 +4043,7 @@ impl Request { // tolerate it: the Producer we hand back simply can't write. if let Ok(mut state) = self.state.write() { state.accept(info.clone()); + state.start_pending = self.resolving_start; } // Accepting the request creates the track producer: count it as one ingress // subscription (closed when the last handle drops). No-op when untagged. diff --git a/rs/moq-tokio/tests/broadcast.rs b/rs/moq-tokio/tests/broadcast.rs index 3ae72b0740e..50043686d3d 100644 --- a/rs/moq-tokio/tests/broadcast.rs +++ b/rs/moq-tokio/tests/broadcast.rs @@ -994,6 +994,196 @@ async fn broadcast_route_migration() { handle_b.await.expect("server b panicked").expect("server b failed"); } +/// A subscriber returning to a parked track is not handed the parked cache when the +/// upstream resolves its start past it (lite-06+ resolves the start from the budget). +/// +/// The front keeps what an unread track delivered as a warm cache. While parked, the +/// publisher moved on, so the resumed upstream subscription starts well past that cache. +/// The cache was only fresh against its own frozen edge; serving it first put a +/// rejoining player seconds behind live. +#[tracing_test::traced_test] +#[tokio::test] +async fn broadcast_rejoin_skips_a_stale_warm_cache() { + use moq_net::Timestamp; + + let ms = |ms: u64| Timestamp::from_millis(ms).unwrap(); + let write = |track: &moq_net::track::Producer, sequence: u64, at: u64| { + let mut group = track + .create_group(moq_net::group::Info { sequence }) + .expect("create group"); + group.write_frame(ms(at), b"frame".as_ref()).expect("write frame"); + group.finish().expect("finish group"); + }; + + let pub_origin = moq_tokio::origin::spawn(); + let broadcast = pub_origin.create_broadcast("test").expect("create broadcast"); + broadcast.announce(Default::default()).expect("announce"); + let track = broadcast.create_track("audio", None).expect("create track"); + let live = track.clone(); + for sequence in 0..4u64 { + write(&track, sequence, sequence * 20); + } + + let mut config = moq_tokio::listen::Config::default(); + config.bind = Some("[::]:0".parse().unwrap()); + config.tls.generate = vec!["localhost".into()]; + let mut server = config + .init(Default::default()) + .expect("init server") + .listen() + .await + .expect("listen"); + let addr = server.local_addr().expect("local addr"); + let server = tokio::spawn(async move { + let request = server.accept().await.expect("accept"); + let session = request.with_publisher(&pub_origin).ok().await?; + let _broadcast = broadcast; + let _track = track; + let _ = session.closed().await; + Ok::<_, anyhow::Error>(()) + }); + + let sub_origin = moq_tokio::origin::spawn(); + let sub_consumer = sub_origin.consume(); + let mut announcements = sub_consumer.announced(); + let mut config = moq_tokio::connect::Config::default(); + config.tls.insecure = Some(true); + let client = config.init(Default::default()).expect("init client"); + let url: url::Url = format!("moqt://localhost:{}", addr.port()).parse().unwrap(); + let (_client, session) = tokio::time::timeout(TIMEOUT, connect_once(client.with_subscriber(sub_origin), url)) + .await + .expect("connect timeout") + .expect("connect failed"); + + assert!(next_announce(&mut announcements).await.kind.is_active()); + let remote = tokio::time::timeout(TIMEOUT, sub_consumer.request_broadcast("test")) + .await + .expect("request timeout") + .expect("broadcast resolves"); + let budget = moq_net::track::Subscription::default().with_max_age(Duration::from_millis(100)); + async fn recv(sub: &mut moq_net::track::Subscriber) -> u64 { + tokio::time::timeout(TIMEOUT, sub.recv_group()) + .await + .expect("recv timeout") + .expect("recv failed") + .expect("track ended") + .sequence + } + + let mut sub = remote + .track("audio") + .unwrap() + .subscribe(budget.clone()) + .await + .expect("subscribe"); + recv(&mut sub).await; + drop(sub); + + // The front parks the track and cancels upstream, while the publisher moves on. + tokio::time::timeout(TIMEOUT, live.unused()) + .await + .expect("upstream never canceled") + .expect("track open"); + for sequence in 4..=20u64 { + write(&live, sequence, 10_000 + (sequence - 4) * 20); + } + + let mut sub = remote + .track("audio") + .unwrap() + .subscribe(budget) + .await + .expect("resubscribe"); + let first = recv(&mut sub).await; + assert!( + first > 4, + "a rejoining reader was served the stale cache first: group {first}" + ); + let mut sequence = first; + while sequence < 20 { + sequence = recv(&mut sub).await; + assert!(sequence >= 4, "a rejoining reader was served stale group {sequence}"); + } + + drop(sub); + drop(session); + server.await.expect("server panicked").expect("server failed"); +} + +/// A subscriber returning to a parked track whose newest group is still current gets it +/// back: the re-splice asks for that group's tail, which the publisher answers at once. +/// Asking past it would wait for a group that a quiet track (a catalog) may never send. +#[tracing_test::traced_test] +#[tokio::test] +async fn broadcast_rejoin_replays_a_current_warm_cache() { + let pub_origin = moq_tokio::origin::spawn(); + let broadcast = pub_origin.create_broadcast("test").expect("create broadcast"); + broadcast.announce(Default::default()).expect("announce"); + let track = broadcast.create_track("catalog.json", None).expect("create track"); + let live = track.clone(); + let mut group = track.append_group().expect("append group"); + group + .write_frame(moq_net::Timestamp::ZERO, b"snapshot".as_ref()) + .expect("write frame"); + group.finish().expect("finish group"); + + let mut config = moq_tokio::listen::Config::default(); + config.bind = Some("[::]:0".parse().unwrap()); + config.tls.generate = vec!["localhost".into()]; + let mut server = config + .init(Default::default()) + .expect("init server") + .listen() + .await + .expect("listen"); + let addr = server.local_addr().expect("local addr"); + let server = tokio::spawn(async move { + let request = server.accept().await.expect("accept"); + let session = request.with_publisher(&pub_origin).ok().await?; + let _broadcast = broadcast; + let _track = track; + let _ = session.closed().await; + Ok::<_, anyhow::Error>(()) + }); + + let sub_origin = moq_tokio::origin::spawn(); + let sub_consumer = sub_origin.consume(); + let mut announcements = sub_consumer.announced(); + let mut config = moq_tokio::connect::Config::default(); + config.tls.insecure = Some(true); + let client = config.init(Default::default()).expect("init client"); + let url: url::Url = format!("moqt://localhost:{}", addr.port()).parse().unwrap(); + let (_client, session) = tokio::time::timeout(TIMEOUT, connect_once(client.with_subscriber(sub_origin), url)) + .await + .expect("connect timeout") + .expect("connect failed"); + + assert!(next_announce(&mut announcements).await.kind.is_active()); + let remote = tokio::time::timeout(TIMEOUT, sub_consumer.request_broadcast("test")) + .await + .expect("request timeout") + .expect("broadcast resolves"); + + for round in 0..2 { + let mut sub = remote + .track("catalog.json") + .unwrap() + .subscribe(None) + .await + .expect("subscribe"); + assert_eq!(read_payloads(&mut sub, 1).await, ["snapshot"], "round {round}"); + drop(sub); + // The front parks the track and cancels upstream before the next round rejoins. + tokio::time::timeout(TIMEOUT, live.unused()) + .await + .expect("upstream never canceled") + .expect("track open"); + } + + drop(session); + server.await.expect("server panicked").expect("server failed"); +} + /// A publisher-side route update re-advertises downstream as a restart. /// /// The publisher re-prices its announced route with a longer chain; the From 803bfdb017fc7ea372167aed3a6ee8a123e4757d Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 25 Sep 2026 05:56:15 -0700 Subject: [PATCH 2/2] fix(net): derive a warm copy's edge from its groups, not the datagram counter Datagrams share the sequence counter, so one sequenced past the newest group hid it as the live edge: an open edge was dropped from the warm copy. Also sweeps the quiet-track rejoin test over every version, with a finished and an open newest group, requiring the live feed to resume after each rejoin. Co-Authored-By: Claude Opus 5.5 --- rs/moq-net/src/model/origin.rs | 11 +++++- rs/moq-tokio/tests/broadcast.rs | 66 +++++++++++++++++++++++++++++---- 2 files changed, 68 insertions(+), 9 deletions(-) diff --git a/rs/moq-net/src/model/origin.rs b/rs/moq-net/src/model/origin.rs index f0ae48583da..2f4c0af8400 100644 --- a/rs/moq-net/src/model/origin.rs +++ b/rs/moq-net/src/model/origin.rs @@ -1852,8 +1852,13 @@ fn warm_copy(source: &track::Consumer, head: Option<&WarmGroup>) -> Option moq_net::group::Consumer { + tokio::time::timeout(TIMEOUT, sub.recv_group()) + .await + .unwrap_or_else(|_| panic!("{ctx}: recv_group timeout")) + .expect("recv_group failed") + .expect("track closed") + } + async fn read(group: &mut moq_net::group::Consumer, ctx: &str) -> String { + let frame = tokio::time::timeout(TIMEOUT, group.read_frame()) + .await + .unwrap_or_else(|_| panic!("{ctx}: read_frame timeout")) + .expect("read_frame failed") + .expect("group ended"); + String::from_utf8(frame.payload.to_vec()).unwrap() + } + + for round in 1..=3 { + let ctx = format!("{version} open={open} round {round}"); + let update = format!("v{round}"); let mut sub = remote .track("catalog.json") .unwrap() .subscribe(None) .await .expect("subscribe"); - assert_eq!(read_payloads(&mut sub, 1).await, ["snapshot"], "round {round}"); + let mut reading = recv(&mut sub, &ctx).await; + if open { + for frame in 0..round { + assert_eq!(read(&mut reading, &ctx).await, format!("v{frame}"), "{ctx}"); + } + group + .write_frame(moq_net::Timestamp::ZERO, update.as_bytes()) + .expect("write frame"); + } else { + assert_eq!(read(&mut reading, &ctx).await, format!("v{}", round - 1), "{ctx}"); + let mut next = live.append_group().expect("append group"); + next.write_frame(moq_net::Timestamp::ZERO, update.as_bytes()) + .expect("write frame"); + next.finish().expect("finish group"); + reading = recv(&mut sub, &ctx).await; + } + // The live feed resumed behind the replayed cache. + assert_eq!(read(&mut reading, &ctx).await, update, "{ctx}"); + drop(reading); drop(sub); // The front parks the track and cancels upstream before the next round rejoins. tokio::time::timeout(TIMEOUT, live.unused()) .await - .expect("upstream never canceled") + .unwrap_or_else(|_| panic!("{ctx}: upstream never canceled")) .expect("track open"); }