Repository navigation
fix(net): resolve a relayed subscription's start from its source #4387
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
34da014
15ff7cd
82b4138
d1113f3
db37ef8
a49f510
b34fc71
c8e676d
3727d30
71bc0d6
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| # [M] Late joiner history | ||
|
|
||
| ## Goal | ||
|
|
||
| A second subscriber that joins a relay's track from group 0 while a first | ||
| subscriber still holds it receives every group the relay has cached, | ||
| including a finished group older than the open live one. A regression test | ||
| that joins late through a relay fails before the fix. | ||
|
|
||
| ## Plan | ||
|
|
||
| Found downstream on moq.pro: its billing rollup tracks replay a finished | ||
| history group followed by an open live group, and a second browser opening | ||
| the Cost page on the same edge intermittently loses the older group, so | ||
| prior-period usage rows are missing. Production dashboards are affected, | ||
| not just tests. #4387 fixed the first viewer's version of this; what it | ||
| leaves is the viewer that joins later. | ||
|
|
||
| What we saw: | ||
|
|
||
| - With #4387 applied, moq.pro's `e2e/tests/rollup-history.spec.ts | ||
| --repeat-each 12` still fails about a quarter of runs, always on the | ||
| second viewer. The edge logs `serving group` only for the newer group on | ||
| that viewer's subscription, while the first viewer got both. | ||
| - A late-joiner variant of `rs/moq-net/tests/history_groups.rs` showed the | ||
| relay's late cursor spliced across two segments with its floor at the | ||
| newer group, so the cached older group sat below it. The segment churn | ||
| comes through `Action::Query`, `Splice`, `Park`, and `Release` in | ||
| `rs/moq-net/src/model/origin.rs`, triggered by the TRACK_INFO request that | ||
| precedes each subscription and then goes idle. | ||
| - That variant is not a faithful reproduction yet: under paused time the | ||
| origin's linger timers fire during the test's idle waits, so it also failed | ||
| for lite-03 and the IETF drafts, which the real stack does not. A | ||
| reproduction needs to keep the relay's track warm without auto-advancing | ||
| past the linger, or drive the timers explicitly. | ||
|
|
||
| Start from how a newly spliced or parked segment derives its floor for a | ||
| subscriber that asked for group 0, and whether a warm copy's cached groups | ||
| below the new segment's first group stay reachable. The | ||
| [splice edge cases](/quest/m1/splice-edges.md) touch the same code. | ||
|
|
||
| ## Related | ||
|
|
||
| - [Splice edge cases](/quest/m1/splice-edges.md) - other spliced-track cases that lose or mis-judge groups |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,7 +1,7 @@ | ||
| use crate::runtime::Timers as _; | ||
| use crate::{SessionError, announce, frame, group, origin, track}; | ||
| use std::{ | ||
| collections::HashMap, | ||
| collections::{BTreeSet, HashMap}, | ||
| ops::Bound, | ||
| sync::{ | ||
| Arc, | ||
|
|
@@ -2214,6 +2214,10 @@ struct TrackRun<S: crate::transport::poll::Session> { | |
| // exclusive final sequence (which may be ahead of the live edge). | ||
| emit_range: bool, | ||
| start_sent: bool, | ||
| // The first servable group, held until the source resolves where its feed starts. | ||
| first: Option<group::Consumer>, | ||
| // Groups skipped for a missing head before the start resolved, which it must not name. | ||
| skipped: BTreeSet<u64>, | ||
| end_sent: bool, | ||
| // Lite07+ sends SUBSCRIBE_END with the stream count instead of as soon as the | ||
| // boundary is known, once every group below it has opened its stream. | ||
|
|
@@ -2250,6 +2254,8 @@ impl<S: crate::transport::poll::Session> TrackRun<S> { | |
| track_priority_tx, | ||
| emit_range, | ||
| start_sent: false, | ||
| first: None, | ||
| skipped: BTreeSet::new(), | ||
| end_sent: false, | ||
| count_streams, | ||
| datagrams, | ||
|
|
@@ -2300,6 +2306,27 @@ impl<S: crate::transport::poll::Session> TrackRun<S> { | |
| continue; | ||
| } | ||
|
|
||
| // The first group waits for the source to resolve where its feed starts; datagrams | ||
| // keep flowing meanwhile. | ||
| if let Some(group) = self.first.take() { | ||
| match self.track.poll_start(waiter) { | ||
| Poll::Ready(source) => { | ||
| self.start(group, source, stream)?; | ||
| continue; | ||
| } | ||
| Poll::Pending => { | ||
| self.first = Some(group); | ||
| if self.datagrams | ||
| && let Poll::Ready(Some(datagram)) = self.track.poll_recv_datagram(waiter)? | ||
| { | ||
| self.ctx.serve_datagram(datagram); | ||
| continue; | ||
| } | ||
| return Poll::Pending; | ||
| } | ||
| } | ||
| } | ||
|
|
||
| // One cursor drives the whole subscription: poll the cap-aware arrival-order | ||
| // group and, when enabled, the next best-effort datagram. Groups are polled | ||
| // first so a datagram burst can't starve them; datagrams flow whenever no | ||
|
|
@@ -2308,45 +2335,20 @@ impl<S: crate::transport::poll::Session> TrackRun<S> { | |
| if let Poll::Ready(res) = poll_recv_next(&mut self.track, self.datagrams, emit_boundary, waiter) { | ||
| match res? { | ||
| Recv::Group(mut group) => { | ||
| let sequence = group.sequence; | ||
| if !position_group(&mut group, self.start_frame, self.end_frame) { | ||
| // Its head is gone, and this subscriber didn't ask for a | ||
| // partial group. Skip it rather than open a stream that can | ||
| // only be reset; the next servable group resolves the start. | ||
| tracing::debug!(subscribe = self.ctx.id, track = %self.ctx.track_name, sequence, "skipping group with a missing head"); | ||
| tracing::debug!(subscribe = self.ctx.id, track = %self.ctx.track_name, sequence = group.sequence, "skipping group with a missing head"); | ||
| if self.emit_range && !self.start_sent { | ||
| self.skipped.insert(group.sequence); | ||
| } | ||
| continue; | ||
| } | ||
| if self.emit_range && !self.start_sent { | ||
| self.start_sent = true; | ||
| // Only the group: the subscriber derives the start frame from | ||
| // its own request (see `lite::SubscribeStart`). | ||
| stream | ||
| .writer | ||
| .buffer(&lite::SubscribeResponse::Start(lite::SubscribeStart { | ||
| group: sequence, | ||
| }))?; | ||
| // SUBSCRIBE_OK is an implicit drop of everything below the | ||
| // resolved start (the subscriber records it as a permanent | ||
| // miss), so a lower group arriving late must not be served | ||
| // after all. A widening SUBSCRIBE_UPDATE re-lowers the floor, | ||
| // renegotiating the resolved start along with the demand. | ||
| self.track.start_at(sequence); | ||
| match self.emit_range && !self.start_sent { | ||
| true => self.first = Some(group), | ||
| false => self.serve(group), | ||
| } | ||
|
|
||
| let frame_start = group.index(); | ||
| tracing::debug!(subscribe = self.ctx.id, track = %self.ctx.track_name, sequence, "serving group"); | ||
|
|
||
| // Use the latest priority for new groups so SUBSCRIBE_UPDATE applies to them too. | ||
| let current_priority = self.ctx.track_priority_current(); | ||
| // The subscribe id scopes the group tie-break: one queue serves every | ||
| // subscription on the session, and only groups of the same one may be | ||
| // ranked against each other by sequence. | ||
| let handle = self | ||
| .ctx | ||
| .priority | ||
| .insert(Priority::new(current_priority, self.ctx.id, sequence)); | ||
| self.children | ||
| .push(GroupServe::new(self.ctx.clone(), sequence, frame_start, handle, group)); | ||
| } | ||
| Recv::Datagram(datagram) => self.ctx.serve_datagram(datagram), | ||
| Recv::Boundary(group) => { | ||
|
|
@@ -2383,6 +2385,63 @@ impl<S: crate::transport::poll::Session> TrackRun<S> { | |
| } | ||
| } | ||
|
|
||
| impl<S: crate::transport::poll::Session> TrackRun<S> { | ||
| /// Send SUBSCRIBE_START for the first servable group, then serve it. | ||
| /// | ||
| /// A relay caches groups in upstream arrival order, and a newer group's stream can | ||
| /// beat an older one, so the first group here need not be the oldest the source | ||
| /// serves. `source` is where the source's feed starts, raised to this cursor's floor, | ||
| /// so the resolved start is the lower of the two: resolving from the later group would | ||
| /// drop the older one for good. | ||
| fn start( | ||
| &mut self, | ||
| group: group::Consumer, | ||
| source: Option<u64>, | ||
| stream: &mut Stream<S, Version>, | ||
| ) -> Result<(), Error> { | ||
| let mut start = source.map_or(group.sequence, |source| source.min(group.sequence)); | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When the source has resolved its start to group N but QUIC delivers a valid N+1 stream before a headless N stream, this commits SUBSCRIBE_START at N immediately. The late N subsequently fails AGENTS.md reference: rs/moq-net/AGENTS.md:L9-L9 Useful? React with 👍 / 👎.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I disagree with this one. SUBSCRIBE_START is a lower bound: nothing below it arrives. It does not promise a stream for the START group. It resolves before the groups above it have arrived, and any of them, the START group included, can still turn out headless, be evicted, or be aborted. The subscriber already handles that as an ordinary gap above START. The previous fix only corrects what the publisher already knows when it resolves: a group it has skipped cannot be where the feed starts. Accounting for a headless N arriving after N+1 would mean holding START until every group between the source's start and the held group has arrived. That brings back the stall this design avoids, and some of those groups may never exist. (Written by Opus 5.5) |
||
| // A skipped group is never served, so it cannot be where the feed starts. This stops | ||
| // at the held group at the latest, since it was not skipped. | ||
| while self.skipped.contains(&start) { | ||
| start += 1; | ||
| } | ||
| self.skipped.clear(); | ||
| self.start_sent = true; | ||
| // Only the group: the subscriber derives the start frame from its own request | ||
| // (see `lite::SubscribeStart`). | ||
| stream | ||
| .writer | ||
| .buffer(&lite::SubscribeResponse::Start(lite::SubscribeStart { group: start }))?; | ||
| // SUBSCRIBE_START is an implicit drop of everything below the resolved start (the | ||
| // subscriber records it as a permanent miss), so a lower group arriving late must | ||
| // not be served after all. A widening SUBSCRIBE_UPDATE re-lowers the floor, | ||
| // renegotiating the resolved start along with the demand. Raised, not assigned: an | ||
| // update that landed while the group was held may already have raised it past. | ||
| self.track.raise_start_to(start); | ||
| self.serve(group); | ||
| Ok(()) | ||
| } | ||
|
|
||
| /// Open a group machine for `group`. | ||
| fn serve(&mut self, group: group::Consumer) { | ||
| let sequence = group.sequence; | ||
| let frame_start = group.index(); | ||
| tracing::debug!(subscribe = self.ctx.id, track = %self.ctx.track_name, sequence, "serving group"); | ||
|
|
||
| // Use the latest priority for new groups so SUBSCRIBE_UPDATE applies to them too. | ||
| let current_priority = self.ctx.track_priority_current(); | ||
| // The subscribe id scopes the group tie-break: one queue serves every | ||
| // subscription on the session, and only groups of the same one may be | ||
| // ranked against each other by sequence. | ||
| let handle = self | ||
| .ctx | ||
| .priority | ||
| .insert(Priority::new(current_priority, self.ctx.id, sequence)); | ||
| self.children | ||
| .push(GroupServe::new(self.ctx.clone(), sequence, frame_start, handle, group)); | ||
| } | ||
| } | ||
|
|
||
|
Comment on lines
+2388
to
+2444
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win 🔎 Supported by static analysis🏁 Script executed: sed -n '2280,2445p' rs/moq-net/src/lite/publisher.rs
sed -n '3140,3225p' rs/moq-net/src/lite/publisher.rsRepository: moq-dev/moq Length of output: 9922 🏁 Script executed: sed -n '3140,3315p' rs/moq-net/src/lite/publisher.rs
rg -n "fn (poll_start|raise_start_to|start_at)|raise_start_to|poll_start|position_group|struct TrackRun|impl.*TrackRun" rs/moq-net/src rs/moq-net/testsRepository: moq-dev/moq Length of output: 11036 🏁 Script executed: sed -n '1995,2035p' rs/moq-net/src/lite/publisher.rs
sed -n '3970,4100p' rs/moq-net/src/model/track.rs
sed -n '2260,2350p' rs/moq-net/src/model/resume.rs
sed -n '7500,7585p' rs/moq-net/src/model/track.rsRepository: moq-dev/moq Length of output: 14411 🏁 Script executed: rg -n -C 5 "pub struct Subscriber|struct Subscriber|min_sequence|fn .*sequence|start_sequence" rs/moq-net/src/model/track.rs
rg -n -C 4 "self\.track\.[A-Za-z_]+|track\.[A-Za-z_]+" rs/moq-net/src/lite/publisher.rs | head -n 160Repository: moq-dev/moq Length of output: 36135 Discard a held group below the updated start floor. When Discard the held group when its sequence is below the subscriber’s effective floor before calling 🤖 Prompt for AI Agents |
||
| /// Serves one group on its own unidirectional stream: the header, then every | ||
| /// frame, applying queue and SUBSCRIBE_UPDATE priority changes as they land. | ||
| struct GroupServe<S: crate::transport::poll::Session> { | ||
|
|
@@ -3102,6 +3161,155 @@ mod serve_group_test { | |
| assert_eq!(*log.writes.lock().unwrap(), [0, 1, 0, 1, 2, 3, 2]); | ||
| } | ||
|
|
||
| /// A lite-07 run over a relay's track, whose upstream subscription still waits on the | ||
| /// source's SUBSCRIBE_START, on a subscribe stream the test can push updates onto. | ||
| struct RelayRun { | ||
| run: TrackRun<ScriptedSession>, | ||
| stream: Stream<ScriptedSession, Version>, | ||
| session: ScriptedSession, | ||
| opens: Arc<Opens>, | ||
| } | ||
|
|
||
| impl RelayRun { | ||
| /// Subscribe to `track` from `start_group`. | ||
| async fn new(track: &mut track::Producer, start_group: u64) -> Self { | ||
| track.request_start(Some(0)).unwrap(); | ||
| let subscription = track::Subscription::default() | ||
| .with_start(track::Position::group(start_group)) | ||
| .with_max_age(Duration::from_secs(30)); | ||
| let subscriber = track.subscribe(subscription); | ||
|
|
||
| let mut session = ScriptedSession::new(Vec::new()); | ||
| let (send, recv) = futures::future::poll_fn(|cx| { | ||
| <ScriptedSession as web_transport_trait::poll::Session>::poll_open_bi(&mut session, cx) | ||
| }) | ||
| .await | ||
| .unwrap(); | ||
| let stream = Stream::<ScriptedSession, Version> { | ||
| writer: Writer::new(send, Version::Lite07), | ||
| reader: crate::coding::Reader::new(recv, Version::Lite07), | ||
| }; | ||
| let track_priority = kio::Producer::new(0u8); | ||
| let opens = Arc::<Opens>::default(); | ||
| let ctx = Subscription { | ||
| session: session.clone(), | ||
| id: 0, | ||
| track_name: "test".into(), | ||
| priority: PriorityQueue::default(), | ||
| track_priority: track_priority.consume(), | ||
| track_priority_seen: 0, | ||
| version: Version::Lite07, | ||
| timescale: Some(crate::Timescale::default()), | ||
| opens: opens.clone(), | ||
| }; | ||
| let bounds = Bounds { | ||
| start_group: Some(start_group), | ||
| start_frame: 0, | ||
| end_group: None, | ||
| end_frame: None, | ||
| }; | ||
| Self { | ||
| run: TrackRun::new(ctx, subscriber, bounds, track_priority), | ||
| stream, | ||
| session, | ||
| opens, | ||
| } | ||
| } | ||
|
|
||
| /// Drive the run until it parks, which it must. | ||
| fn settle(&mut self) { | ||
| let Self { run, stream, .. } = self; | ||
| let res = kio::wait(|waiter| run.poll(stream, waiter)).now_or_never(); | ||
| assert!(res.is_none(), "the run ended"); | ||
| } | ||
|
|
||
| /// How many group streams the run opened. | ||
| fn opened(&self) -> u64 { | ||
| self.opens.opened.load(Ordering::Relaxed) | ||
| } | ||
|
|
||
| /// Whether the first thing written was SUBSCRIBE_START at `group`. | ||
| fn started_at(&self, group: u64) -> bool { | ||
| let start = lite::SubscribeResponse::Start(lite::SubscribeStart { group }); | ||
| let start = start.encode_bytes(Version::Lite07).unwrap(); | ||
| self.session.log.writes.lock().unwrap().starts_with(&start) | ||
| } | ||
| } | ||
|
|
||
| /// A SUBSCRIBE_UPDATE landing while the first group waits on the source's start keeps | ||
| /// the floor it raised: resolving the start from the held group must not lower it. | ||
| #[tokio::test] | ||
| async fn held_first_group_keeps_an_updated_floor() { | ||
| let mut track = track::Producer::new(Arc::new(broadcast::Info::default()), "test", None); | ||
| let mut relay = RelayRun::new(&mut track, 0).await; | ||
|
|
||
| write_group(&mut track, 5, 0); | ||
| relay.settle(); | ||
|
|
||
| // The subscriber moves its start past the held group before the source resolves. | ||
| let update = lite::SubscribeUpdate { | ||
| priority: 0, | ||
| max_age: Duration::ZERO, | ||
| start_group: Some(7), | ||
| end_group: None, | ||
| start_frame: 0, | ||
| end_frame: None, | ||
| }; | ||
| relay.session.push(&update.encode_bytes(Version::Lite07).unwrap()); | ||
| relay.settle(); | ||
|
|
||
| track.start_at(0).unwrap(); | ||
| relay.settle(); | ||
| assert_eq!(relay.opened(), 1, "the held group is served"); | ||
|
|
||
| // Group 6 is below the updated floor. | ||
| write_group(&mut track, 6, 6); | ||
| relay.settle(); | ||
| assert_eq!(relay.opened(), 1, "served a group below the floor"); | ||
| } | ||
|
|
||
| /// A source whose feed starts below the subscriber's floor serves the floor's group, so | ||
| /// a newer group arriving first must not resolve the start past it. | ||
| #[tokio::test] | ||
| async fn held_first_group_resolves_to_the_floor_under_the_source() { | ||
| let mut track = track::Producer::new(Arc::new(broadcast::Info::default()), "test", None); | ||
| let mut relay = RelayRun::new(&mut track, 5).await; | ||
|
|
||
| write_group(&mut track, 6, 6); | ||
| relay.settle(); | ||
| track.start_at(0).unwrap(); | ||
| relay.settle(); | ||
| assert_eq!(relay.opened(), 1, "the held group is served"); | ||
|
|
||
| write_group(&mut track, 5, 5); | ||
| relay.settle(); | ||
| assert_eq!(relay.opened(), 2, "dropped the floor's group"); | ||
| assert!(relay.started_at(5)); | ||
| } | ||
|
|
||
| /// A group skipped for a missing head before the start resolves is never served, so | ||
| /// SUBSCRIBE_START must not name it, even where the source's feed starts. | ||
| #[tokio::test] | ||
| async fn held_first_group_starts_past_a_skipped_head() { | ||
| let mut track = track::Producer::new(Arc::new(broadcast::Info::default()), "test", None); | ||
| let mut relay = RelayRun::new(&mut track, 5).await; | ||
|
|
||
| // Group 5's first frame is gone. | ||
| let mut headless = track.create_group(group::Info { sequence: 5 }).unwrap(); | ||
| headless.start_at(1).unwrap(); | ||
| headless | ||
| .write_frame(Timestamp::from_millis(5).unwrap(), b"x".as_slice()) | ||
| .unwrap(); | ||
| headless.finish().unwrap(); | ||
| write_group(&mut track, 6, 6); | ||
| relay.settle(); | ||
|
|
||
| track.start_at(5).unwrap(); | ||
| relay.settle(); | ||
| assert_eq!(relay.opened(), 1, "the held group is served"); | ||
| assert!(relay.started_at(6), "named the skipped group"); | ||
| } | ||
|
|
||
| /// A track that ends without a group still ends the subscription, with no stream owed. | ||
| #[tokio::test] | ||
| async fn lite07_end_counts_zero_streams() { | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If a SUBSCRIBE_UPDATE arrives while this group is waiting for the source's START, the update changes
start_frame,end_frame, and the sequence bounds, but the already-positioned group bypassesposition_groupand is unconditionally served here. A narrowing update can therefore deliver frames outside the new range, while a widening update cannot restore frames discarded under the old range; same-group frame bounds are not filtered by the receiving track model. Defer positioning or reacquire and validate the held group against the latest bounds before serving it.Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Partly agreed, fixed in d1113f3.
The real bug:
start()assigned the cursor floor withstart_at(start). If an update raised the floor above the held group, resolving the start lowered it again, so later groups below the updated floor were served. It now callsraise_start_to, which does the same thing when there is no race, since the resolved start is never below the floor. Regression test:lite::publisher::serve_group_test::held_first_group_keeps_an_updated_floor(it served group 6 under a floor of 7 before the fix).I disagree with repositioning the held group. The cursor already handed it out, so it is in the same position as a
GroupServewaiting inOpenfor stream credit, which can also take arbitrarily long. SUBSCRIBE_UPDATE has never repositioned groups that were handed out before it arrived. It only changes the priority of groups in flight. The subscriber gets the same result as an update arriving just after the group was served, a race it already has to handle.(Written by Opus 5.5)