Skip to content

Origin-routed subscriber hangs in read_frame when a track is aborted mid-group and re-created under the same name #4365

Closed

Description

@palerikm

Summary. A subscriber reading a track through the origin (a route-fed broadcast, as a relay-fed client does) waits in group::Consumer::read_frame() for ever under one condition: the publisher aborts the track mid-group, and a fresh track with the same name is created on the broadcast before the open group is dropped. There is no error and no end-of-group, so a client that re-subscribes on track errors never gets the chance.

The same abort without the fresh track surfaces correctly (read_frame → dropped, then recv_group → error). A subscriber taking the track straight off the broadcast (broadcast::Producer::consume()) also sees the error in both cases. Only the origin-routed (spliced) track hangs.

Versions

Output of the repro below

via_origin=false remint_before_group_drop=false -> recv_group error: cancelled
via_origin=false remint_before_group_drop=true  -> recv_group error: cancelled
via_origin=true  remint_before_group_drop=false -> recv_group error: not found
via_origin=true  remint_before_group_drop=true  -> read_frame HANG in group 2

Where it seems to come from. broadcast.rs track_inner gives an origin-routed consumer a spliced logical track backed by resume::Producer. Its comment says an aborted logical track is dropped so "a publisher that had not yet created the track may have it now". It looks as if, when the underlying group dies, the splice tries to resume that group onto the fresh track, which will never produce it, and nothing ever gives up. resume.rs's give_up ("no route can serve the rest of this group") does fire in other cases; here nothing reaches it. I have not dug further than that.

Expected. The in-flight group ends with an error (or None) and recv_group returns an error, as when the fresh track does not exist. Then a client can re-subscribe and pick up the fresh track.

Real-world shape. A publisher that recovers from a track failure by re-creating the track (or a relay re-serving a track after its upstream reconnects) while a relay-fed subscriber is mid-group. That subscriber goes silent with no error. We are working around it by delaying the re-create and by treating a subscribed track that delivers nothing for a few seconds as an error.

Repro (cargo run; ~15 s)

Cargo.toml

[package]
name = "moq-splice-repro"
version = "0.1.0"
edition = "2024"

[dependencies]
moq-net = "0.3"          # 0.3.7; also reproduces on main @ ca47661f
moq-tokio = { version = "0.19", default-features = false }
tokio = { version = "1", features = ["full"] }
bytes = "1"

src/main.rs

//! A track aborted mid-group, with a fresh same-name track created before the
//! open group is dropped. A subscriber reading through the ORIGIN (a route-fed
//! broadcast) then waits in `read_frame` forever; a subscriber on the broadcast
//! directly gets an error. Without the fresh track both get an error.
use moq_net::group::Info as Group;
use std::time::Duration;

async fn last_event(remint: bool, via_origin: bool) -> String {
    let origin = moq_tokio::origin::spawn();
    let bc = origin.create_broadcast("demo/feed").expect("broadcast");
    bc.announce(moq_net::origin::Route::default()).expect("announce");
    let track = bc.create_track("video", None).expect("track");

    let reader_bc = if via_origin {
        let mut all = moq_net::Patterns::new();
        all.insert(moq_net::Pattern::all());
        let consumer = origin
            .consume()
            .scope(moq_net::Path::from("demo/feed"), &all)
            .expect("scope");
        let mut announced = consumer.announced();
        let update = announced.next().await.expect("announce update");
        consumer.request_broadcast(&update.prefix).await.expect("request")
    } else {
        bc.consume()
    };
    let mut sub = reader_bc
        .track("video")
        .expect("track name")
        .subscribe(moq_net::track::Subscription::default().with_max_age(Duration::from_millis(500)))
        .await
        .expect("subscribe");

    let bc_w = bc.clone();
    let writer = tokio::spawn(async move {
        let mut seq = 0u64;
        let mut group = track.create_group(Group { sequence: seq }).expect("group");
        for n in 0..10_000u64 {
            if n == 75 {
                // mid group 2
                track.clone().abort(moq_net::Error::Cancel).expect("abort");
                let fresh = remint.then(|| {
                    let t2 = bc_w.create_track("video", None).expect("fresh track");
                    let g2 = t2.create_group(Group { sequence: seq + 1 }).expect("fresh group");
                    (t2, g2)
                });
                drop(group);
                tokio::time::sleep(Duration::from_secs(4)).await;
                drop(fresh);
                return;
            }
            if n > 0 && n % 30 == 0 {
                let _ = group.finish();
                seq += 1;
                group = track.create_group(Group { sequence: seq }).expect("group");
            }
            group
                .write_frame(moq_net::Timestamp::now(), bytes::Bytes::from_static(b"x"))
                .expect("write");
            tokio::time::sleep(Duration::from_millis(2)).await;
        }
    });

    let wait = Duration::from_secs(2);
    let event = 'outer: loop {
        match tokio::time::timeout(wait, sub.recv_group()).await {
            Err(_) => break "recv_group HANG".to_string(),
            Ok(Err(e)) => break format!("recv_group error: {e}"),
            Ok(Ok(None)) => break "track end".to_string(),
            Ok(Ok(Some(mut g))) => loop {
                match tokio::time::timeout(wait, g.read_frame()).await {
                    Err(_) => break 'outer format!("read_frame HANG in group {}", g.sequence),
                    Ok(Err(_)) | Ok(Ok(None)) => break,
                    Ok(Ok(Some(_))) => {}
                }
            },
        }
    };
    writer.abort();
    event
}

#[tokio::main(flavor = "multi_thread", worker_threads = 2)]
async fn main() {
    for via_origin in [false, true] {
        for remint in [false, true] {
            println!(
                "via_origin={via_origin:<5} remint_before_group_drop={remint:<5} -> {}",
                last_event(remint, via_origin).await
            );
        }
    }
}

Activity

  1. kixelated commented on Sep 28, 2026

    @kixelated
    Collaborator

    What's the desired functionality? Obviously not hanging forever, but should we issue a fetch to try to recover the tail of the group? Or should we abort?

  2. palerikm commented on Sep 28, 2026

    @palerikm
    Author

    Abort, please, or at most a fetch that is bounded and fails fast.

    The fresh track is new content under the same name. The comment in track_inner says so for a finished track: a publisher that finishes a track and publishes it again "is serving new content, not resuming this one". A track that was aborted and then re-created is the same case. The new producer never had the aborted group's tail, so a fetch for it can only come back not-found. A fetch is fine if it can't hang and resolves quickly, but it can't recover anything here.

    What we'd want as a subscriber:

    1. The in-flight group ends with an error, as it already does when no fresh track exists (dropped, then an error from recv_group).
    2. recv_group then moves on to the fresh track's groups, or errors so the application can re-subscribe. Either works for us.

    For live media, the tail of an aborted group is worth very little by the time it could arrive; bounded latency matters more than completeness. We work around it today with a read timeout, plus polling recv_group alongside read_frame.

  3. added
    questBeing tracked/planned in a quest. See `quest/`
    on Sep 28, 2026
  4. added 2 commits that reference this issue on Sep 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    questBeing tracked/planned in a quest. See `quest/`

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions