Repository navigation
Description
Discovered this when migrating from ~3* month old versions of the moq family of libraries to the latest releases. Write up by Astra
Quiet catalog disappears after reader rejoin and cache expiry (relay 0.17.3)
Versions: moq-relay 0.17.3, moq-net 0.3.11; Python clients moq-rs 0.5.1 / moq-ffi 0.4.13. Reproduced over QUIC with unchanged clients.
Reproduction
Python script inline below
RELAY=/path/to/moq-relay uv run moq-upstream-catalog-repro.pyThe script publishes one catalog snapshot, lets reader A read and disconnect, reconnects reader B and keeps it subscribed, waits past cache expiry, then starts reader C.
Expected: C receives the unchanged latest catalog.
Actual: the pre-expiry read passes, but C times out. The publisher remains connected. --cache-seconds 30 also reproduces with the default window; --takeover exercises publisher transport replacement.
Suggested patch
Patch inline below
- Hold
cache::ProtectioninWarmGroup: re-splicing dropsWarmCopy, but its retained logical live edge still needs protection. - Release that protection when a newer group arrives, preserving history expiry.
- Preserve finished snapshots in
TrackIo::retireduring takeover.
The attached regression covers Lite05/06 late replay and superseded-history expiry. Released relay fails the QUIC reproducer; the patched relay passes, including the default 30-second window. Patched native suite: 1,565 passed, six ignored.
Related #5100 is already included in this release; this idle/rejoin lifecycle still fails.
Python repro script
# /// script
# requires-python = ">=3.12"
# dependencies = ["moq-rs==0.5.1", "moq-ffi==0.4.13"]
# ///
"""RELAY=/path/to/moq-relay uv run repro.py [--cache-seconds 30] [--takeover]"""
import argparse
import asyncio
import os
import subprocess
import tempfile
import moq
parser = argparse.ArgumentParser()
parser.add_argument('--cache-seconds', type=int, default=1)
parser.add_argument('--takeover', action='store_true')
args = parser.parse_args()
url = 'https://127.0.0.1:18446'
def client(**kwargs):
return moq.Client(url, tls_verify=False, websocket_enabled=False, **kwargs)
async def read(reader):
async with asyncio.timeout(2):
broadcast = await reader.announced_broadcast('repro/video')
async with await broadcast.subscribe_track('catalog.json') as track:
frame = await track.read_frame()
assert frame and frame.payload
return frame.payload
async def main():
with tempfile.TemporaryFile() as log:
relay = subprocess.Popen([
os.environ.get('RELAY', 'moq-relay'), '--listen', '127.0.0.1:18446',
'--listen-tls-generate', 'localhost', '--auth-public', 'repro/**',
'--cache-duration', f'{args.cache_seconds}s',
], stdout=log, stderr=log)
replacement = None
try:
await asyncio.sleep(0.5)
origin = moq.OriginProducer()
async with client(publish=origin) as publisher:
broadcast = origin.create_broadcast('repro/video')
broadcast.set_catalog_section('probe', {'value': 'retained'})
broadcast.announce()
if not args.takeover:
async with client() as first:
await read(first)
await asyncio.sleep(0.2)
async with client() as holder:
remote = await holder.announced_broadcast('repro/video')
async with await remote.subscribe_track('catalog.json') as watching:
original = (await asyncio.wait_for(watching.read_frame(), 2)).payload
async with client() as baseline:
assert await read(baseline) == original
print('PASS: catalog available before cache expiry', flush=True)
if args.takeover:
replacement = await client(publish=origin).__aenter__()
await asyncio.sleep(0.1)
await publisher.__aexit__()
await asyncio.sleep(2 * args.cache_seconds + 2)
async with client() as later:
assert await read(later) == original
print('PASS: unchanged latest catalog survives expiry', flush=True)
finally:
if replacement:
await replacement.__aexit__()
relay.terminate()
relay.wait(timeout=5)
asyncio.run(main())Suggested patch
--- a/rs/moq-net/src/model/origin.rs
+++ b/rs/moq-net/src/model/origin.rs
@@ -2241,12 +2241,26 @@
/// an unfinished producer clears its frames), and either kind supplies the head the
/// continuation lacks when the next park rebuilds the group. Nothing will ever finish
/// an open one, so it aborts on drop.
-struct WarmGroup(group::Producer);
+struct WarmGroup {
+ group: group::Producer,
+ // The logical live edge outlives the parked track that first protected it.
+ protection: Option<cache::Protection>,
+}
+
+impl WarmGroup {
+ fn new(group: group::Producer) -> Self {
+ let protection = group.cache_protect();
+ Self {
+ group,
+ protection: Some(protection),
+ }
+ }
+}
impl Drop for WarmGroup {
fn drop(&mut self) {
- if !self.0.is_finished() {
- let _ = self.0.clone().abort(Error::Cancel);
+ if !self.group.is_finished() {
+ let _ = self.group.clone().abort(Error::Cancel);
}
}
}
@@ -2275,7 +2289,7 @@
fn warm_copy(source: &track::Consumer, head: Option<&WarmGroup>, held: Option<&group::Producer>) -> Option<WarmCopy> {
let info = source.cached_info()?;
let mut track = track::Producer::new(Arc::new(source.broadcast().clone()), source.name(), info);
- let head = head.map(|head| &head.0);
+ let head = head.map(|head| &head.group);
let mut groups = source.cached_groups();
if let Some(held) = held
&& !groups.iter().any(|(group, _)| group.sequence == held.sequence)
@@ -2302,7 +2316,7 @@
let _ = track.adopt_group(head.clone(), true);
first = Some(head.sequence);
if is_latest {
- edge = Some(WarmGroup(head.clone()));
+ edge = Some(WarmGroup::new(head.clone()));
}
} else if is_latest {
edge = warm_rebuild(&track, head, None);
@@ -2316,7 +2330,7 @@
let is_latest = visible && Some(group.sequence) == latest;
let warm = if finished && whole {
let _ = track.adopt_group(group.clone(), visible);
- Some(WarmGroup(group))
+ Some(WarmGroup::new(group))
} else if (finished || is_latest) && (whole || head.is_some_and(|head| head.sequence == group.sequence)) {
// Dropping the source copy resets an open live edge mid-transfer, so rebuild
// it from the frames already delivered: the re-splice asks for the next
@@ -2329,7 +2343,7 @@
None
};
if visible && let Some(warm) = &warm {
- first = Some(first.map_or(warm.0.sequence, |first: u64| first.min(warm.0.sequence)));
+ first = Some(first.map_or(warm.group.sequence, |first: u64| first.min(warm.group.sequence)));
}
if is_latest {
edge = warm;
@@ -2381,14 +2395,14 @@
}
// Wrapped first, so a failed write aborts it rather than dropping it unfinished.
- let rebuilt = WarmGroup(
+ let rebuilt = WarmGroup::new(
track
.create_group(group::Info {
sequence: live.sequence,
})
.ok()?,
);
- let mut writer = rebuilt.0.clone();
+ let mut writer = rebuilt.group.clone();
if start > 0 {
writer.start_at(start).ok()?;
}
@@ -2477,17 +2491,14 @@
self.dead = Some(copy);
}
- /// Retire `copy` as a takeover replaces it. Mid-group, park its delivery warm, open
- /// group included, so a reader arriving later still gets the group's head and the
- /// takeover continues it. Between groups its own cache keeps everything finished, so
- /// there is nothing to do.
+ /// Retire `copy` as a takeover replaces it. Keep its delivery warm so a late reader
+ /// can still read the current group, even after the retired copy's cache expires.
+ /// An open group continues from the retained head; a finished one remains the live
+ /// edge until the replacement produces a newer group.
fn retire(&mut self, copy: &track::Consumer) -> Result<(), Error> {
let Some(held) = self.held.take() else {
return Ok(());
};
- if held.0.is_finished() {
- return Ok(());
- }
let Some(warm) = warm_copy(copy, self.head.as_ref(), Some(&held.0)) else {
return Ok(());
};
@@ -2968,6 +2979,11 @@
}
Step::Held(name, group) => {
if let Some(io) = tracks.get_mut(&name) {
+ if let Some(head) = &mut io.head
+ && group.sequence > head.group.sequence
+ {
+ head.protection = None;
+ }
io.held = Some(HeldGroup(group));
}
continue;
--- a/rs/moq-net/tests/rejoin.rs
+++ b/rs/moq-net/tests/rejoin.rs
@@ -190,3 +190,75 @@
.unwrap_or_else(|_| panic!("{version}: timed out"));
}
}
+
+/// A rejoined reader keeps the quiet catalog's logical live edge available to late joiners.
+#[tokio::test(start_paused = true)]
+async fn rejoined_quiet_catalog_survives_cache_expiry() {
+ for version in ["moq-lite-05", "moq-lite-06"] {
+ tokio::time::timeout(TEST_TIMEOUT + moq_net::cache::DEFAULT_EXPIRY * 5, async {
+ let version: Version = version.parse().unwrap();
+ let publisher = produce_origin(1);
+ let relay = produce_origin(2);
+ let broadcast = publisher.create_broadcast("bench").unwrap();
+ let track = broadcast.create_track("catalog", None).unwrap();
+ broadcast.announce(Default::default()).unwrap();
+ let mut group = track.append_group().unwrap();
+ group.write_frame(Timestamp::ZERO, b"catalog".as_ref()).unwrap();
+ group.finish().unwrap();
+ let mut options = MockConnectOptions::new(version);
+ options.server_publish = Some(publisher.consume());
+ options.client_subscribe = Some(relay.clone());
+ let _upstream = connect_mock(options).await;
+ let session = |hop| {
+ let relay = relay.clone();
+ async move {
+ let client = produce_origin(hop);
+ let mut options = MockConnectOptions::new(version);
+ options.server_publish = Some(relay.consume());
+ options.client_subscribe = Some(client.clone());
+ let pair = connect_mock(options).await;
+ let consumer = client.consume();
+ consumer.routed("bench").await.unwrap();
+ let remote = consumer.request_broadcast("bench").await.unwrap();
+ let sub = remote.track("catalog").unwrap().subscribe(None).await.unwrap();
+ (sub, remote, pair)
+ }
+ };
+ let (mut first, remote, pair) = session(3).await;
+ assert_eq!(first.recv_group().await.unwrap().unwrap().sequence, 0);
+ drop((first, remote, pair));
+ track.unused().await.unwrap();
+ let (mut holder, _remote, _pair) = session(4).await;
+ assert_eq!(holder.recv_group().await.unwrap().unwrap().sequence, 0);
+ tokio::time::sleep(moq_net::cache::DEFAULT_EXPIRY * 2 + Duration::from_secs(2)).await;
+ let (mut later, _remote, _pair) = session(5).await;
+ let mut group = tokio::time::timeout(Duration::from_secs(1), later.recv_group())
+ .await
+ .unwrap_or_else(|_| panic!("{version}: late reader lost retained catalog"))
+ .unwrap()
+ .unwrap();
+ assert_eq!(read_all(&mut group).await.unwrap(), [b"catalog"]);
+ drop(group);
+ let mut newer = track.append_group().unwrap();
+ newer
+ .write_frame(Timestamp::from_millis(1000).unwrap(), b"newer".as_ref())
+ .unwrap();
+ newer.finish().unwrap();
+ assert_eq!(holder.recv_group().await.unwrap().unwrap().sequence, 1);
+ tokio::time::sleep(moq_net::cache::DEFAULT_EXPIRY * 2 + Duration::from_secs(2)).await;
+ let old = async {
+ let mut group = _remote.track("catalog").unwrap().fetch_group(0, None).await?;
+ group.read_frame().await
+ };
+ assert!(
+ tokio::time::timeout(Duration::from_secs(1), old)
+ .await
+ .unwrap()
+ .is_err(),
+ "superseded catalog must no longer be protected from expiry"
+ );
+ })
+ .await
+ .unwrap_or_else(|_| panic!("{version}: timed out"));
+ }
+}
Activity
kixelated commentedon Oct 13, 2026
Thanks for the repro and patch. Your test fails on the
moq-net-v0.3.11tag as described. A port of it passes onmain, across lite-05/06/07 and transport-19/22, including a takeover variant. #4741 removed the warm-copy machinery (WarmGroup,WarmCopy,TrackIo), and readers onmainnow read the serving route's copy directly.Decision (planned in #5418):
quest/m0/release-22/quiet-edge), shipping with the next 0.17 patch; it closes this issue. Root cause onrelease: re-splicing drops theWarmCopy, which finishes its track and demotes its latest group out of cache protection whileio.headstill treats it as the live edge.WarmCopyalive until the new copy delivers a newer group. The existing rule (a live track's latest group is protected) then covers it, with no second protection path to keep in sync.retiregets the same treatment for a finished held group. Yourcache::Protectionpatch is the fallback if that proves invasive. Your regression test lands with it.maingets the ported test plus the takeover variant as a regression guard (quest/m1/quiet-edge-test).Once #5418 merges, anyone who wants to donate tokens can run
/quest-start quest/m0/release-22/quiet-edge.(Written by Claude Opus 5.5)
fix(net): keep a quiet track's latest group past rejoin and takeover
quest: import moq-dev#5407 and moq-dev#5413 (moq-dev#5418)
docs(media): media area conventions and the two CMSF scenarios