Skip to content

moq-relay 0.17.3: quiet catalog lost after reader rejoin and cache expiry #5413

Closed

Description

@EBridgeford

Discovered this when migrating from ~3* month old versions of the moq family of libraries to the latest releases. Write up by Astra

Quiet catalog disappears after reader rejoin and cache expiry (relay 0.17.3)

Versions: moq-relay 0.17.3, moq-net 0.3.11; Python clients moq-rs 0.5.1 / moq-ffi 0.4.13. Reproduced over QUIC with unchanged clients.

Reproduction

Python script inline below

RELAY=/path/to/moq-relay uv run moq-upstream-catalog-repro.py

The script publishes one catalog snapshot, lets reader A read and disconnect, reconnects reader B and keeps it subscribed, waits past cache expiry, then starts reader C.

Expected: C receives the unchanged latest catalog.

Actual: the pre-expiry read passes, but C times out. The publisher remains connected. --cache-seconds 30 also reproduces with the default window; --takeover exercises publisher transport replacement.

Suggested patch

Patch inline below

  • Hold cache::Protection in WarmGroup: re-splicing drops WarmCopy, but its retained logical live edge still needs protection.
  • Release that protection when a newer group arrives, preserving history expiry.
  • Preserve finished snapshots in TrackIo::retire during takeover.

The attached regression covers Lite05/06 late replay and superseded-history expiry. Released relay fails the QUIC reproducer; the patched relay passes, including the default 30-second window. Patched native suite: 1,565 passed, six ignored.

Related #5100 is already included in this release; this idle/rejoin lifecycle still fails.

Python repro script

# /// script
# requires-python = ">=3.12"
# dependencies = ["moq-rs==0.5.1", "moq-ffi==0.4.13"]
# ///
"""RELAY=/path/to/moq-relay uv run repro.py [--cache-seconds 30] [--takeover]"""
import argparse
import asyncio
import os
import subprocess
import tempfile

import moq

parser = argparse.ArgumentParser()
parser.add_argument('--cache-seconds', type=int, default=1)
parser.add_argument('--takeover', action='store_true')
args = parser.parse_args()
url = 'https://127.0.0.1:18446'


def client(**kwargs):
    return moq.Client(url, tls_verify=False, websocket_enabled=False, **kwargs)


async def read(reader):
    async with asyncio.timeout(2):
        broadcast = await reader.announced_broadcast('repro/video')
        async with await broadcast.subscribe_track('catalog.json') as track:
            frame = await track.read_frame()
            assert frame and frame.payload
            return frame.payload


async def main():
    with tempfile.TemporaryFile() as log:
        relay = subprocess.Popen([
            os.environ.get('RELAY', 'moq-relay'), '--listen', '127.0.0.1:18446',
            '--listen-tls-generate', 'localhost', '--auth-public', 'repro/**',
            '--cache-duration', f'{args.cache_seconds}s',
        ], stdout=log, stderr=log)
        replacement = None
        try:
            await asyncio.sleep(0.5)
            origin = moq.OriginProducer()
            async with client(publish=origin) as publisher:
                broadcast = origin.create_broadcast('repro/video')
                broadcast.set_catalog_section('probe', {'value': 'retained'})
                broadcast.announce()
                if not args.takeover:
                    async with client() as first:
                        await read(first)
                    await asyncio.sleep(0.2)
                async with client() as holder:
                    remote = await holder.announced_broadcast('repro/video')
                    async with await remote.subscribe_track('catalog.json') as watching:
                        original = (await asyncio.wait_for(watching.read_frame(), 2)).payload
                        async with client() as baseline:
                            assert await read(baseline) == original
                        print('PASS: catalog available before cache expiry', flush=True)
                        if args.takeover:
                            replacement = await client(publish=origin).__aenter__()
                            await asyncio.sleep(0.1)
                            await publisher.__aexit__()
                        await asyncio.sleep(2 * args.cache_seconds + 2)
                        async with client() as later:
                            assert await read(later) == original
                        print('PASS: unchanged latest catalog survives expiry', flush=True)
        finally:
            if replacement:
                await replacement.__aexit__()
            relay.terminate()
            relay.wait(timeout=5)


asyncio.run(main())

Suggested patch

--- a/rs/moq-net/src/model/origin.rs
+++ b/rs/moq-net/src/model/origin.rs
@@ -2241,12 +2241,26 @@
 /// an unfinished producer clears its frames), and either kind supplies the head the
 /// continuation lacks when the next park rebuilds the group. Nothing will ever finish
 /// an open one, so it aborts on drop.
-struct WarmGroup(group::Producer);
+struct WarmGroup {
+	group: group::Producer,
+	// The logical live edge outlives the parked track that first protected it.
+	protection: Option<cache::Protection>,
+}
+
+impl WarmGroup {
+	fn new(group: group::Producer) -> Self {
+		let protection = group.cache_protect();
+		Self {
+			group,
+			protection: Some(protection),
+		}
+	}
+}
 
 impl Drop for WarmGroup {
 	fn drop(&mut self) {
-		if !self.0.is_finished() {
-			let _ = self.0.clone().abort(Error::Cancel);
+		if !self.group.is_finished() {
+			let _ = self.group.clone().abort(Error::Cancel);
 		}
 	}
 }
@@ -2275,7 +2289,7 @@
 fn warm_copy(source: &track::Consumer, head: Option<&WarmGroup>, held: Option<&group::Producer>) -> Option<WarmCopy> {
 	let info = source.cached_info()?;
 	let mut track = track::Producer::new(Arc::new(source.broadcast().clone()), source.name(), info);
-	let head = head.map(|head| &head.0);
+	let head = head.map(|head| &head.group);
 	let mut groups = source.cached_groups();
 	if let Some(held) = held
 		&& !groups.iter().any(|(group, _)| group.sequence == held.sequence)
@@ -2302,7 +2316,7 @@
 			let _ = track.adopt_group(head.clone(), true);
 			first = Some(head.sequence);
 			if is_latest {
-				edge = Some(WarmGroup(head.clone()));
+				edge = Some(WarmGroup::new(head.clone()));
 			}
 		} else if is_latest {
 			edge = warm_rebuild(&track, head, None);
@@ -2316,7 +2330,7 @@
 		let is_latest = visible && Some(group.sequence) == latest;
 		let warm = if finished && whole {
 			let _ = track.adopt_group(group.clone(), visible);
-			Some(WarmGroup(group))
+			Some(WarmGroup::new(group))
 		} else if (finished || is_latest) && (whole || head.is_some_and(|head| head.sequence == group.sequence)) {
 			// Dropping the source copy resets an open live edge mid-transfer, so rebuild
 			// it from the frames already delivered: the re-splice asks for the next
@@ -2329,7 +2343,7 @@
 			None
 		};
 		if visible && let Some(warm) = &warm {
-			first = Some(first.map_or(warm.0.sequence, |first: u64| first.min(warm.0.sequence)));
+			first = Some(first.map_or(warm.group.sequence, |first: u64| first.min(warm.group.sequence)));
 		}
 		if is_latest {
 			edge = warm;
@@ -2381,14 +2395,14 @@
 	}
 
 	// Wrapped first, so a failed write aborts it rather than dropping it unfinished.
-	let rebuilt = WarmGroup(
+	let rebuilt = WarmGroup::new(
 		track
 			.create_group(group::Info {
 				sequence: live.sequence,
 			})
 			.ok()?,
 	);
-	let mut writer = rebuilt.0.clone();
+	let mut writer = rebuilt.group.clone();
 	if start > 0 {
 		writer.start_at(start).ok()?;
 	}
@@ -2477,17 +2491,14 @@
 		self.dead = Some(copy);
 	}
 
-	/// Retire `copy` as a takeover replaces it. Mid-group, park its delivery warm, open
-	/// group included, so a reader arriving later still gets the group's head and the
-	/// takeover continues it. Between groups its own cache keeps everything finished, so
-	/// there is nothing to do.
+	/// Retire `copy` as a takeover replaces it. Keep its delivery warm so a late reader
+	/// can still read the current group, even after the retired copy's cache expires.
+	/// An open group continues from the retained head; a finished one remains the live
+	/// edge until the replacement produces a newer group.
 	fn retire(&mut self, copy: &track::Consumer) -> Result<(), Error> {
 		let Some(held) = self.held.take() else {
 			return Ok(());
 		};
-		if held.0.is_finished() {
-			return Ok(());
-		}
 		let Some(warm) = warm_copy(copy, self.head.as_ref(), Some(&held.0)) else {
 			return Ok(());
 		};
@@ -2968,6 +2979,11 @@
 			}
 			Step::Held(name, group) => {
 				if let Some(io) = tracks.get_mut(&name) {
+					if let Some(head) = &mut io.head
+						&& group.sequence > head.group.sequence
+					{
+						head.protection = None;
+					}
 					io.held = Some(HeldGroup(group));
 				}
 				continue;
--- a/rs/moq-net/tests/rejoin.rs
+++ b/rs/moq-net/tests/rejoin.rs
@@ -190,3 +190,75 @@
 		.unwrap_or_else(|_| panic!("{version}: timed out"));
 	}
 }
+
+/// A rejoined reader keeps the quiet catalog's logical live edge available to late joiners.
+#[tokio::test(start_paused = true)]
+async fn rejoined_quiet_catalog_survives_cache_expiry() {
+	for version in ["moq-lite-05", "moq-lite-06"] {
+		tokio::time::timeout(TEST_TIMEOUT + moq_net::cache::DEFAULT_EXPIRY * 5, async {
+			let version: Version = version.parse().unwrap();
+			let publisher = produce_origin(1);
+			let relay = produce_origin(2);
+			let broadcast = publisher.create_broadcast("bench").unwrap();
+			let track = broadcast.create_track("catalog", None).unwrap();
+			broadcast.announce(Default::default()).unwrap();
+			let mut group = track.append_group().unwrap();
+			group.write_frame(Timestamp::ZERO, b"catalog".as_ref()).unwrap();
+			group.finish().unwrap();
+			let mut options = MockConnectOptions::new(version);
+			options.server_publish = Some(publisher.consume());
+			options.client_subscribe = Some(relay.clone());
+			let _upstream = connect_mock(options).await;
+			let session = |hop| {
+				let relay = relay.clone();
+				async move {
+					let client = produce_origin(hop);
+					let mut options = MockConnectOptions::new(version);
+					options.server_publish = Some(relay.consume());
+					options.client_subscribe = Some(client.clone());
+					let pair = connect_mock(options).await;
+					let consumer = client.consume();
+					consumer.routed("bench").await.unwrap();
+					let remote = consumer.request_broadcast("bench").await.unwrap();
+					let sub = remote.track("catalog").unwrap().subscribe(None).await.unwrap();
+					(sub, remote, pair)
+				}
+			};
+			let (mut first, remote, pair) = session(3).await;
+			assert_eq!(first.recv_group().await.unwrap().unwrap().sequence, 0);
+			drop((first, remote, pair));
+			track.unused().await.unwrap();
+			let (mut holder, _remote, _pair) = session(4).await;
+			assert_eq!(holder.recv_group().await.unwrap().unwrap().sequence, 0);
+			tokio::time::sleep(moq_net::cache::DEFAULT_EXPIRY * 2 + Duration::from_secs(2)).await;
+			let (mut later, _remote, _pair) = session(5).await;
+			let mut group = tokio::time::timeout(Duration::from_secs(1), later.recv_group())
+				.await
+				.unwrap_or_else(|_| panic!("{version}: late reader lost retained catalog"))
+				.unwrap()
+				.unwrap();
+			assert_eq!(read_all(&mut group).await.unwrap(), [b"catalog"]);
+			drop(group);
+			let mut newer = track.append_group().unwrap();
+			newer
+				.write_frame(Timestamp::from_millis(1000).unwrap(), b"newer".as_ref())
+				.unwrap();
+			newer.finish().unwrap();
+			assert_eq!(holder.recv_group().await.unwrap().unwrap().sequence, 1);
+			tokio::time::sleep(moq_net::cache::DEFAULT_EXPIRY * 2 + Duration::from_secs(2)).await;
+			let old = async {
+				let mut group = _remote.track("catalog").unwrap().fetch_group(0, None).await?;
+				group.read_frame().await
+			};
+			assert!(
+				tokio::time::timeout(Duration::from_secs(1), old)
+					.await
+					.unwrap()
+					.is_err(),
+				"superseded catalog must no longer be protected from expiry"
+			);
+		})
+		.await
+		.unwrap_or_else(|_| panic!("{version}: timed out"));
+	}
+}

Activity

  1. added
    questBeing tracked/planned in a quest. See `quest/`
    on Oct 13, 2026
  2. kixelated commented on Oct 13, 2026

    @kixelated
    Collaborator

    Thanks for the repro and patch. Your test fails on the moq-net-v0.3.11 tag as described. A port of it passes on main, across lite-05/06/07 and transport-19/22, including a takeover variant. #4741 removed the warm-copy machinery (WarmGroup, WarmCopy, TrackIo), and readers on main now read the serving route's copy directly.

    Decision (planned in #5418):

    • Backport to the 0.17 line (quest/m0/release-22/quiet-edge), shipping with the next 0.17 patch; it closes this issue. Root cause on release: re-splicing drops the WarmCopy, which finishes its track and demotes its latest group out of cache protection while io.head still treats it as the live edge.
    • The backport keeps the WarmCopy alive until the new copy delivers a newer group. The existing rule (a live track's latest group is protected) then covers it, with no second protection path to keep in sync. retire gets the same treatment for a finished held group. Your cache::Protection patch is the fallback if that proves invasive. Your regression test lands with it.
    • main gets the ported test plus the takeover variant as a regression guard (quest/m1/quiet-edge-test).

    Once #5418 merges, anyone who wants to donate tokens can run /quest-start quest/m0/release-22/quiet-edge.

    (Written by Claude Opus 5.5)

  3. added 2 commits that reference this issue on Oct 13, 2026
    7246c4f
    8972488
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    questBeing tracked/planned in a quest. See `quest/`

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions