Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion doc/concept/moq-lite.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,9 @@ authorization and hidden names work exactly as without one.
worker closed a path and serves it again. Such a worker keeps the path's group
sequence going, even when its own input starts over at 0. Otherwise a viewer
returning within a relay's cache window gets the old output's latest group,
then nothing until the new sequence passes it.
then nothing until the new sequence passes it. On moq-lite 07 and
moq-transport, the relay next to the worker sees the sequence go back and
gives its own viewers an error instead, so they request the path again.
- **Newest wins.** Among routes at the same prefix, the newest epoch ranks
first, ahead of cost, and a route without an epoch ranks last. Newest means
the latest UUIDv7 timestamp, so hosts with skewed clocks can lose a restart
Expand Down
2 changes: 1 addition & 1 deletion quest/m1/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ session outcomes, and embedder refusals inside session outcomes.
- [Interop close logs](/quest/m1/interop-close-log.md) - logging fix landed; delete once the required Interop run is green
- [One max_age meaning](/quest/m1/cache-max-age.md) - a superseded group goes stale on wall clock since its successor arrived or on media time, whichever is first, in Rust and js/net; fixes the untimed failover stall
- [FETCH max-delay](/quest/m1/fetch-max-delay.md) - historical readers obey publisher max-age and an optional tighter reader budget, including while holding a group
- [Upstream position regression](/quest/m1/largest-regression.md) - a relay copy that sees upstream's largest group go backwards on moq-transport or epochless lite-07 fails loud instead of serving the old instance's cache
- [Refused idle copy](/quest/m1/refused-copy.md) - behind any chain of relays a returning reader never gets a restarted publisher's old group: a refused idle copy withholds its cache and no hop answers from it
- [Untimed decisions](/quest/m1/untimed-decisions.md) - the maintainer decides whether moq-archive keeps refusing untimed tracks, and whether a malformed FETCH object ends its track
- [FFI shape](/quest/m1/ffi-shape/README.md) - the bindings mirror Rust's layers: net at the root, then media, json, flate, audio, and video namespaces built from the handle below
- [Publishing never invents a timestamp](/quest/m1/publish-timestamp.md) - no Rust or binding publish API fills in a timestamp; an untimed payload goes out untimed
Expand Down
65 changes: 0 additions & 65 deletions quest/m1/largest-regression.md

This file was deleted.

77 changes: 77 additions & 0 deletions quest/m1/refused-copy.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# [M] A refused idle copy withholds its cache at every hop

## Goal

Behind any chain of relays, a returning reader never gets a restarted
publisher's old group. An idle copy that its upstream refuses before
answering, or answers below its cache, withholds its cache and ends, and no
hop answers from a withheld cache. Covers lite-07 and moq-transport; lite-05
and 06 carry no largest position and keep today's behavior.

Today only the relay next to the worker catches the restart (#5057). With an
edge relay in between, the returning viewer still gets the old group on every
version: the edge's idle copy is refused, aborts, and an aborted copy lets its
readers drain the old cache.

## Plan

Facts as of #5057:

- lite: on lite-05+ the upstream's reset lands in `ServeLoop::poll`
(`lite/subscriber.rs`) as `ServeEnd::GiveBack(err)`, and `TrackServeRun`
calls `serving.abort(err)`. An aborted copy is readable
(`TrackState::readable`, `model/track.rs`), so readers drain the idle
cache. `Unroutable` is not in `route_failed` (`model/resume.rs`), and since
#5137 a front on a route without an epoch passes the abort on rather than
re-splicing.
- moq-transport: the middle relay waits on `track.poll_live`
(`ietf/publisher.rs`), but its own copy's abort unblocks that wait, and
`live_edge` reads `latest()` and `peek_latest`, which ignore the
`live_floor` that `withhold_cache` raises. Its SUBSCRIBE_OK carries the
hidden group as Largest, so the edge sees no regression, goes live on its
old group, and then gets PUBLISH_DONE `InternalError`, which aborts its
copy without closing the source (`ietf/subscriber.rs`, `subscribe_once`).
The error-answer branch there also aborts without withholding.
- Every source of a refusal toward a rejoining copy today: no covering route
(`model/origin.rs`), a lite-07 epoch mismatch (`model/origin.rs`), and
#5057's regression. On the wire they can't be told apart: lite resets with
0x36, and moq-transport sends DOES_NOT_EXIST, read back as `NotFound`
(`ietf/error.rs`).
- lite-07: a group header or a datagram (`lite/subscriber.rs`) makes the copy
live before START, after which `idle_cached()` returns `None` and the
regression check is skipped. A lite-07 publisher always writes START before
the first group.

Decided 2026-10-08:

- An idle copy refused before it was answered can't judge its cache, so it
withholds it, whatever the error. A verdict error closes the source so the
front ends (#5054's guarantee); a `route_failed` error fails over as today.
This is right for every refusal source above, and needs no wire change.
Rejected: matching UNROUTABLE only, which moq-transport can't carry
distinctly, and a new regression error code, a wire change that doesn't
help moq-transport.
- No hop answers from a withheld cache: `live_edge` honors the withheld
floor, and a moq-transport relay whose copy ended refuses the downstream
SUBSCRIBE instead of sending SUBSCRIBE_OK with the hidden group.
- lite-07 defers going live on a group until START. A datagram-only track
never gets a START, so it keeps the gap, which needs a reader starting
below the cache; document it as a known limit.
- Decided 2026-10-08: m1, beside the regression check it extends. A relay
that under-reports its upstream's largest (a cold copy answering from
older groups) can trip a false regression; that costs a spurious error and
a re-request, never stale data, and is accepted: cold-relay Largest stays
INVALID_RANGE (`quest/m0/README.md`).

Verification: mocked time, a viewer behind an edge relay and the relay next
to a claim worker that restarts at group 0 within the linger, on lite-07 and
moq-transport 14, 16 and 22: the returning viewer gets an error and then the
new output from group 0, never the old group. A lite-07 case where a group
stream arrives before START. A same-epoch standby still resumes, and a
`route_failed` refusal still fails over.

Public API: none expected. Wire: none new.

## Related

- [Announcement takeover](/quest/m1/announce-takeover.md) - a worker that wants readers kept across a restart proves continuity instead of relying on an epochless claim
62 changes: 47 additions & 15 deletions rs/moq-net/src/ietf/subscriber.rs
Original file line number Diff line number Diff line change
Expand Up @@ -1905,9 +1905,9 @@ where
let mut this = self.clone();

let path = path.to_owned();
let broadcast = broadcast.clone();
let source = (*source).clone();
subscribes.push(async move {
this.run_subscribe(path, broadcast, request).await;
this.run_subscribe(path, source, request).await;
});
}

Expand All @@ -1917,9 +1917,10 @@ where
async fn run_subscribe(
&mut self,
broadcast_path: Path<'_>,
// Held for the subscription's lifetime but never watched: the broadcast ending
// is a retraction, which does not disturb a subscription already in flight.
_broadcast: broadcast::Dynamic,
// The minted source, never watched: its end is a retraction, which does not disturb a
// subscription already in flight. Closed once the route shows it serves new content
// under the name.
source: broadcast::Producer,
request: track::Request,
) {
// Data streams wait on the alias bound by SUBSCRIBE_OK, so leave the model request
Expand Down Expand Up @@ -1964,7 +1965,14 @@ where
};
loop {
let Some(idle) = self
.subscribe_once(&broadcast_path, &track_name, target, &mut group_fetches, &mut gate)
.subscribe_once(
&broadcast_path,
&track_name,
&source,
target,
&mut group_fetches,
&mut gate,
)
.await
else {
return;
Expand All @@ -1986,6 +1994,7 @@ where
&mut self,
broadcast_path: &Path<'_>,
track_name: &str,
source: &broadcast::Producer,
target: Target,
group_fetches: &mut TaskSet,
gate: &mut crate::auth::Gate,
Expand Down Expand Up @@ -2250,6 +2259,20 @@ where
// Aborted with the session while the answer was in hand.
return None;
}
// moq-transport names no publisher instance, so a Largest below what the copy
// cached is a publisher reusing the name for new content. The source closes
// first: its front ends before any reader sees the copy fail, so a reader
// re-requesting gets a fresh one.
if idle.track.regresses(resuming.flatten()) {
tracing::warn!(broadcast = %self.origin.absolute(broadcast_path), track = %track_name, ?largest, "upstream went back in its group sequence");
let mut track = idle.track;
track.withhold_cache();
source.close();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Retire the old front before accepting another request

When another local requester asks for this path after source.close() but before the independently spawned origin driver polls SourceClosed (possible on the normal multithreaded Tokio runtime), OriginState::fronts still contains a resolved, open front, so request_broadcast hands out that old broadcast. Once Action::End runs, the newcomer receives an error from the retiring track instead of the fresh front promised by this branch. Source closure and front retirement need an atomic or acknowledged handoff rather than call ordering; the lite regression path has the same race.

AGENTS.md reference: AGENTS.md:L64-L64

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Declining. The guarantee is for a reader that saw the copy fail: a reader only surfaces a copy's error through the front's verdict (resume.rs poll_end: "a copy that failed is the front's to replace"), and the front's wait checks SourceClosed before a copy's end, so it detaches the closed source, finds nothing else to select, and leaves the front table before any reader sees UNROUTABLE. That reader's re-request mints a fresh front.

The window you describe is a newcomer arriving after source.close() but before the front task runs. It is indistinguishable from one that joined a moment before the check fired: both are in flight on the old front, both get one error, and the next request is fresh. Neither sees the old output's groups, since the copy's cache is withheld first. Closing that window would need the front table to track each front's serving source, an origin-wide change for a path that only exists to surface a publisher reusing a name, so the PR accepts it.

(Written by Claude Opus 5.5)

self.remove_subscribe(request_id);
let _ = track.abort(Error::Unroutable);
self.cancel_subscribe(stream, request_id).await;
return None;
}
(idle.track, idle.dynamic)
}
None => {
Expand Down Expand Up @@ -5352,8 +5375,9 @@ mod tests {
drop(track);
drop(consumer);

let source = producer.clone();
let serving = moq_net_sim::spawn(async move {
subscriber.run_subscribe(Path::new("broadcast"), dynamic, request).await;
subscriber.run_subscribe(Path::new("broadcast"), source, request).await;
});

moq_net_sim::timeout(std::time::Duration::from_secs(1), serving)
Expand Down Expand Up @@ -5402,8 +5426,9 @@ mod tests {

let request = dynamic.requested_track().await.expect("no track requested");

let source = producer.clone();
let serving = moq_net_sim::spawn(async move {
subscriber.run_subscribe(Path::new("broadcast"), dynamic, request).await;
subscriber.run_subscribe(Path::new("broadcast"), source, request).await;
});

// Let the SUBSCRIBE go out. No SUBSCRIBE_OK is coming, so the subscription never
Expand Down Expand Up @@ -5466,8 +5491,9 @@ mod tests {

let request = dynamic.requested_track().await.expect("no track requested");

let source = producer.clone();
let serving = moq_net_sim::spawn(async move {
subscriber.run_subscribe(Path::new("broadcast"), dynamic, request).await;
subscriber.run_subscribe(Path::new("broadcast"), source, request).await;
});

// Let the SUBSCRIBE go out, then retract the broadcast before any response.
Expand Down Expand Up @@ -5523,9 +5549,10 @@ mod tests {
let request = dynamic.requested_track().await.expect("no track requested");

let mut fetch = None;
let source = producer.clone();
let serving = match kind {
Allocated::Subscribe => moq_net_sim::spawn(async move {
subscriber.run_subscribe(Path::new("broadcast"), dynamic, request).await;
subscriber.run_subscribe(Path::new("broadcast"), source, request).await;
}),
Allocated::TrackStatus => moq_net_sim::spawn(async move {
assert!(
Expand Down Expand Up @@ -5650,8 +5677,9 @@ mod tests {

let request = dynamic.requested_track().await.expect("no track requested");

let source = producer.clone();
let serving = moq_net_sim::spawn(async move {
subscriber.run_subscribe(Path::new("broadcast"), dynamic, request).await;
subscriber.run_subscribe(Path::new("broadcast"), source, request).await;
});

settle().await;
Expand Down Expand Up @@ -5793,8 +5821,9 @@ mod tests {
// subscription reached Established rather than assume it.
let probe = subscriber.clone();

let source = producer.clone();
let serving = moq_net_sim::spawn(async move {
subscriber.run_subscribe(Path::new("broadcast"), dynamic, request).await;
subscriber.run_subscribe(Path::new("broadcast"), source, request).await;
});

// Let the SUBSCRIBE go out and the SUBSCRIBE_OK come back, so the subscription is
Expand Down Expand Up @@ -5872,8 +5901,9 @@ mod tests {
let request = dynamic.requested_track().await.expect("no track requested");

let probe = subscriber.clone();
let source = producer.clone();
let serving = moq_net_sim::spawn(async move {
subscriber.run_subscribe(Path::new("broadcast"), dynamic, request).await;
subscriber.run_subscribe(Path::new("broadcast"), source, request).await;
});

settle().await;
Expand Down Expand Up @@ -5971,8 +6001,9 @@ mod tests {
let request = dynamic.requested_track().await.expect("no track requested");

let probe = subscriber.clone();
let source = producer.clone();
let serving = moq_net_sim::spawn(async move {
subscriber.run_subscribe(Path::new("broadcast"), dynamic, request).await;
subscriber.run_subscribe(Path::new("broadcast"), source, request).await;
});
settle().await;
let _subscription = track.subscribe(None);
Expand Down Expand Up @@ -9662,9 +9693,10 @@ mod joining_fetch_tests {
let request = dynamic.requested_track().await.expect("no track requested");

let mut serving_subscriber = subscriber.clone();
let source = producer.clone();
let serving = moq_net_sim::spawn(async move {
serving_subscriber
.run_subscribe(Path::new("broadcast"), dynamic, request)
.run_subscribe(Path::new("broadcast"), source, request)
.await;
});

Expand Down
Loading
Loading