Skip to content

moq-net: a group held across a route switch is not woken when its successor gets its first timestamp #4950

Closed

Description

@fperex

What happens

A reader holds an open group whose route went quiet, and the front has switched to a route that serves from a later group. Recover::poll can only give the held group up (Error::Old) once the successor group has a timestamp, because the group's reach ends where its immediate successor begins. While the successor has no timestamp the poll returns Pending. When the successor's first frame arrives, the held reader is not woken, although a fresh poll would now return Err(Old).

It is a missed wakeup (a Pending that does not arrange a wake for the change that makes progress possible), not a wrong verdict: the verdict is right as soon as something polls again.

Minimal reproduction

Added to the test module in rs/moq-net/src/model/resume.rs, next to a_group_no_route_continues_is_given_up_once_stale. It uses only the existing helpers and the existing Wake counting waker, no sleeps.

#[test]
fn a_group_no_route_continues_wakes_when_its_successor_is_stamped() {
	let routes = Producer::new();
	let logical = logical(&routes);
	let a = copy();
	routes.serve(a.consume());
	let mut sub = subscribe(&logical, Duration::from_millis(100));
	for (sequence, ms) in [(0, 0), (1, 1000)] {
		let mut group = a.create_group(group::Info { sequence }).unwrap();
		group.write_frame(ts(ms), b"x".as_ref()).unwrap();
		group.finish().unwrap();
	}
	assert_eq!(recv(&mut sub).sequence, 0);
	assert_eq!(recv(&mut sub).sequence, 1);

	// The route that serves group 2 never finishes it.
	let mut open = a.create_group(group::Info { sequence: 2 }).unwrap();
	open.write_frame(ts(2000), b"held".as_ref()).unwrap();
	let mut held = recv(&mut sub);
	assert_eq!(read(&mut held), Some(b"held".to_vec()));

	// The replacement serves from group 3, which has no frame yet, and has group 4 already.
	let b = copy();
	routes.serve(b.consume());
	let mut successor = b.create_group(group::Info { sequence: 3 }).unwrap();
	let mut live = b.create_group(group::Info { sequence: 4 }).unwrap();
	live.write_frame(ts(4000), b"live".as_ref()).unwrap();
	live.finish().unwrap();

	let wake = Arc::new(Wake::default());
	let waiter = kio::Waiter::new(wake.clone().into());
	assert!(held.poll_read_frame(&waiter).is_pending());
	wake.0.store(false, std::sync::atomic::Ordering::Relaxed);

	// Group 2 ends where group 3 begins, so this is when it is known to be 1s behind.
	successor.write_frame(ts(3000), b"successor".as_ref()).unwrap();
	assert!(
		wake.0.load(std::sync::atomic::Ordering::Relaxed),
		"the successor's first timestamp wakes the held group"
	);
	assert!(matches!(held.poll_read_frame(&waiter), Poll::Ready(Err(Error::Old))));
}

cargo test -p moq-net --lib a_group_no_route_continues on the commit below:

test model::resume::test::a_group_no_route_continues_wakes_when_its_successor_is_stamped ... FAILED
panicked at rs/moq-net/src/model/resume.rs:1302:9:
the successor's first timestamp wakes the held group

It fails the same way on every run (20 of 20). Writing the successor's timestamp before the first poll gives Ready(Err(Old)) immediately, and so does the poll after the missed wake, so the verdict itself is fine.

Expected vs actual

  • Expected: the first timestamp of group 3 wakes the reader parked on group 2, which then reads Err(Old).
  • Actual: no wake. The reader stays parked until something else wakes it: another group created on the serving route, the route or its track ending, a frame or close on the held group's own route, or another route change. Under the simulated executor (moq_net_sim) with nothing else happening, the run stops with deadlock: every task is parked and no timer is armed.

Suspected cause

Recover::poll judges staleness through serving.poll_stale(...) (resume.rs:877-881), which is Consumer::poll_stale (track.rs:2680). That polls only the track state, so it registers on the track state's waiters. The successor's first timestamp is written to the group state (group.rs:516-538, GroupState::stamp at group.rs:284), and a kio write only wakes the waiters of the state it modified (rs/kio/src/producer.rs:401-426). The track state is untouched, so nobody is woken.

The plain, no route switch path already handles this: GroupExpiry::is_expired (track.rs:3296) registers on the candidate, its immediate successor (poll_timestamp, and poll_closed for an abort) and the groups past the edge, with the comment "A first timestamp can change the verdict without mutating the track" (track.rs:3333), and has wake tests for it (track.rs:5788, track.rs:5822). The route switch path does the same judgement without that registration. The existing test a_group_no_route_continues_is_given_up_once_stale (resume.rs:1239) polls with now_or_never, so it checks the verdict but not the wake.

The same missing registration also affects a newer group's first timestamp (the new edge) and an abort of the successor on a rewound timeline; I checked both with the same setup and they also do not wake the reader.

Impact

Narrow and bounded as far as I can tell, but I have only reproduced it in process, not on a real session. The poll has to land between the successor group's creation and its first frame header (the wire subscriber creates the group on the group header, lite/subscriber.rs:843, and stamps it when the first frame header is decoded), or the first frame has to be late, for example behind a loss on that stream. The default subscription (max_age zero) is affected as well: the held group is stale as soon as its successor has a timestamp. The consequence is that the held group is given up at the next unrelated wake, in practice the next group on the new route, instead of when it became stale. I did not find a way to make it permanent with a live publisher.

Affected version

main at 0574f5b. Consumer::poll_stale came in with #4741 (0382d30).

Possible fix

Move the registration loop of GroupExpiry::is_expired into one TrackState method and use it from both is_expired and Consumer::poll_stale. With that change the test above passes 20 of 20, and cargo test -p moq-net, cargo clippy -p moq-net --all-targets -- -D warnings and cargo fmt --check are clean. I can open a PR with the test and this change if that direction is acceptable.

Found while porting regression tests.

(Written by Claude Sonnet 5.5 and Claude Opus 5.5)

Metadata

Metadata

Assignees

No one assigned

    Labels

    questBeing tracked/planned in a quest. See `quest/`

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions