Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
8e50f5f
chore(quest): claim quest/m2/cluster-routes/acceptance.md
kixelated Nov 18, 2026
3c7c7ca
fix(moq-net): lite-07 routes skip the lite-06 churn mitigations, and …
kixelated Nov 18, 2026
d075782
test(moq-net): pin a grant too wide for one ANNOUNCE_REQUEST, and poi…
kixelated Nov 18, 2026
366afa8
test(moq-net): retarget the lost-publisher drill at retire, and widen…
kixelated Nov 18, 2026
b67e47c
fix(moq-net): a late retirement spares the copies a later showing of …
kixelated Nov 18, 2026
61d8e35
fix(moq-net): retiring a claim takes its copies at an older epoch too
kixelated Nov 18, 2026
1595e40
fix(moq-net): a copy ended while its claim stays shown is withdrawn a…
kixelated Nov 18, 2026
16fed2f
fix(moq-net): a live copy of the same claim keeps its claimant's hop …
kixelated Nov 18, 2026
f5690d8
Merge remote-tracking branch 'origin/main' into t3/retire-lite06-chur…
kixelated Nov 19, 2026
66c4be1
fix(moq-net): the route mesh harness lets go of a claim as a session …
kixelated Nov 19, 2026
3469e58
quest: weigh option D for a grant too wide to ask for, and recommend it
kixelated Nov 19, 2026
2c956db
Merge remote-tracking branch 'origin/main' into HEAD
kixelated Nov 19, 2026
861f189
quest: a wide grant is asked wider and answered in part
kixelated Nov 19, 2026
769e29f
Merge remote-tracking branch 'origin/main' into HEAD
kixelated Nov 19, 2026
bfd60e6
Merge remote-tracking branch 'origin/main' into t3/retire-lite06-chur…
kixelated Nov 19, 2026
9abc43e
fix(moq-net): a copy of an ended claim another grant brought never shows
kixelated Nov 19, 2026
354d1e1
fix(moq-net): a held copy goes with its holder's route, and a newer e…
kixelated Nov 19, 2026
b5e608f
fix(moq-net): a holder that moves to another announcer's claim holds …
kixelated Nov 19, 2026
554aa75
fix(moq-net): an ended claim holds back only its own copies, and a la…
kixelated Nov 19, 2026
4a3563e
Merge remote-tracking branch 'origin/main' into t3/retire-lite06-chur…
kixelated Nov 19, 2026
87c853e
fix(moq-net): a lagging copy releases holds only past two skipped rev…
kixelated Nov 19, 2026
6bc49fd
fix(moq-net): an ended claim holds back its copies at every path they…
kixelated Nov 19, 2026
6192adb
fix(moq-net): a reader below an ended claim ends at once only if it s…
kixelated Nov 19, 2026
88d9178
style(moq-net): format the mixed-batch test
kixelated Nov 19, 2026
36f832c
fix(moq-net): a reader below an ended claim keeps a lost claim's caus…
kixelated Nov 19, 2026
3baea32
fix(moq-net): a copy of an ended claim lost in the same message ends …
kixelated Nov 19, 2026
55c701f
Merge remote-tracking branch 'origin/main' into quest/m2/cluster-rout…
kixelated Nov 19, 2026
68a90f4
fix(moq-net): a copy lost beside its ended claim ends with no other h…
kixelated Nov 19, 2026
4c2921b
fix(moq-net): a relay passing skipped revisions on as one lets its ne…
kixelated Nov 19, 2026
01b27f5
fix(moq-net): an ended claim holds back copies only where its own pre…
kixelated Nov 19, 2026
0db8ae5
Merge remote-tracking branch 'origin/main' into quest/m2/cluster-rout…
kixelated Nov 19, 2026
004a4e7
quest: moq-relay's handoff tests hold under load
kixelated Nov 19, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions quest/m0/test-flakes-3/handoff-ready-load.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# [S] moq-relay's handoff tests hold under load

## Goal

`moq-relay`'s `handoff::` integration tests pass however loaded the machine is.

## Plan

Three of them (`a_killed_relay_s_clients_are_reset_by_the_one_it_replaced`, `a_restart_starts_the_new_relay_beside_the_old`, `a_relay_no_successor_takes_over_from_drains`) failed under a full `just check` on 2026-11-19, each with `no READY reached the service manager: Elapsed` after the test's 10s wall-clock `TIMEOUT`, on a branch that touched no moq-relay code ([moq-dev/moq#9112](https://github.com/moq-dev/moq/pull/9112)).
Run alone, all seven pass in 1.2s.
Reproduce under load first, then make the READY wait follow an event, such as the relay process exiting, instead of a deadline.
6 changes: 3 additions & 3 deletions quest/m2/cluster-routes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,9 +42,9 @@ cursor hold was the earlier mitigation. Main also contains
retraction quarantine and
[moq-dev/moq#5391](https://github.com/moq-dev/moq/pull/5391)'s same-clock
queue dedup. #5375 measured residual stale revivals, so neither longer holds
nor one passing mesh establish correctness. Retire the mitigations only
after the incident fixtures and failover tests demonstrate their
replacement, as [acceptance](/quest/m2/cluster-routes/acceptance.md) does.
nor one passing mesh establish correctness. All three now hold only routes
learned over a hop chain, and go with lite-06: lite-07 routes skip them,
proven by the incident fixtures and failover tests on the route layer.

## Related

Expand Down
144 changes: 91 additions & 53 deletions quest/m2/cluster-routes/acceptance.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,72 +2,110 @@

## Goal

The route layer is proven on the implementation, and the mitigations its
fixtures replace are retired: [moq-dev/moq#5375](https://github.com/moq-dev/moq/pull/5375)'s
retraction quarantine, [moq-dev/moq#5391](https://github.com/moq-dev/moq/pull/5391)'s
same-clock queue dedup, and the 300 ms update hold, all marked `lite06_churn` in
`rs/moq-net/src/model/origin.rs`.
A lite-07 subscriber whose grant permits more disjoint regions than one
ANNOUNCE_REQUEST can name either discovers every region or says it cannot:
no reader sees a region it never asked for as synced and empty.

## Plan

Before the lite-07 freeze, exercise a scoped subscriber whose permitted, disjoint prefixes
cannot fit one ANNOUNCE_REQUEST and cannot be widened within the grant. In
`lite/subscriber/routed.rs`, `asks` can discard a prefix while `release` treats an
empty set of matching replies as synchronized. Publish inside the last permitted region
and verify either complete discovery or a surfaced inability to answer. An empty,
synchronized snapshot for an unasked region is incorrect. The audit traced these branches
in [moq-dev/moq#8987](https://github.com/moq-dev/moq/pull/8987); confirm the behavior on real sessions and fix it if reproduced.

Decided 2026-11-14 (maintainer, /quest-complete on
[moq-dev/moq#8768](https://github.com/moq-dev/moq/pull/8768)): ✅ stays in
m2 and gates the cut, per lite07-finalize's 2026-10-13 rule that routes'
tier and partition tests run before the cut (rejected: moving it to m3 and
fixing late routing flaws in lite-08). A flaw found here is fixed on
lite-07-wip before it freezes.

The fixtures landed in [moq-dev/moq#8832](https://github.com/moq-dev/moq/pull/8832),
and the reference model is deleted. On real sessions with time mocked: the
topology and prior-art fixtures (`tests/integration/route_acceptance.rs`),
failover liveness and the live-graph tests of
[moq-dev/moq#4644](https://github.com/moq-dev/moq/pull/4644) (`route_failover.rs`),
the tiered cluster, partition, and scoped-interest checks (`route_cluster.rs`),
and mount continuity (`mount.rs`).
[moq-dev/moq#8987](https://github.com/moq-dev/moq/pull/8987) put lite-07 on
`route::relay::Router` and rewrote their assertions for it; the old
router's forwarding and operation-count fixtures went with it, and
`route/tests/multicast*.rs` covers the new one. The route fan-out quest
those pinned was deleted
with the old rule, its one surviving cost moved to
[route wakes](/quest/m4/route-wakes.md).
[moq-dev/moq#9112](https://github.com/moq-dev/moq/pull/9112) confined the
`lite06_churn` mitigations to routes learned over a hop chain, made a claim
the route node lets go of leave every copy at once, recorded why lite-06
keeps its `Start, End, Start` residual, and pointed the Node ID 0 coverage
at the native JS claims lanes of `just test interop`. What remains:

`auth::residual_lite_07_a_grant_too_wide_to_ask_syncs_without_its_last_region`
pins the flaw on real sessions. A relay grants its client 18 disjoint
regions over two tokens; each token's grant fits its own AUTH message, but
the union does not fit the one ANNOUNCE_REQUEST a subscriber's whole set
must (`lite::request_fits`, which the publisher enforces). `coarsen` in
`lite/subscriber/routed.rs` then drops the last region, and `release` treats
a want no asked prefix overlaps as answered: a whole-tree reader syncs
without that region's broadcast, and a reader of the region alone syncs
empty. Fix it per the decision under Waiting, and turn the residual into
the passing case.

What the publishers do today with an ask wider than the grant (traced in
code, then run on a server enforcing `r00/a/**` and `r01/b/**` with the
subscriber forced to ask for the root):

- The ask is legal. The draft already says a prefix outside the
subscriber's scope "lists nothing and is SYNCED at once"
(`doc/draft/moq-lite.md`, Announce). Confining the ask to the grant is
only `coarsen`'s choice, from
[moq-dev/moq#8987](https://github.com/moq-dev/moq/pull/8987). Neither
publisher checks an ask against the grant: both filter each claim by its
path (`Answer::shows` in `lite/publisher/routed.rs`, `#changes` in
`js/net/src/lite/answer.ts`), and `Answer::poll_permit` re-filters every
claim the node told as the grant changes. The root ask got ANNOUNCE_OK,
SYNCED, both granted broadcasts, nothing outside the grant, and no close.
- A claim above a granted region is lost. A publisher presents a broader
claim at the requested prefix itself (`relay.rs`, "presented at the
prefix itself"). Asked for a region, a service claim at `r00` arrives at
`r00/a`, which the grant covers. Asked for the root, it would arrive at
`r00`, which the grant does not cover, so the filter drops it. The reader
syncs without a claim that serves its region.
- `js/net`'s subscriber already asks for the root once its readers' set
does not fit (`#wanted` in `js/net/src/lite/routed.ts`), ignoring the
grant. So a confined JS client with that many readers has this gap
against either publisher today.

Decided 2026-11-16 (maintainer, /quest-plan of the pre-cut audit): the
switch makes `origin::Config::update_hold` crate-private, since nothing
outside moq-net sets it, and puts whatever of the three mitigations survives
here under one named lite-06 marker. So retiring them below changes no
public API, and one that a lite-06 mesh keeps is already marked temporary.
[moq-dev/moq#9053](https://github.com/moq-dev/moq/pull/9053) did both: the
three are marked by the `lite06_churn` module, and still run on every
version, so gating any that a lite-06 mesh keeps to the lite-06 session
path is part of retiring them below.
Decided 2026-11-19 (maintainer, in the coordinating session, on
[moq-dev/moq#9112](https://github.com/moq-dev/moq/pull/9112)): ✅ D, ask
wider and answer the granted part (rejected: A, the size exception for
granted regions; B, the public API change; C, leaving the reader to hang).
It changes the lite-07-wip draft and both publishers, so it lands before
the cut. The options as they were put:

What remains:
- How a subscriber handles a grant too wide for one ANNOUNCE_REQUEST.
- (D) Ask wider, answer the granted part. `coarsen` falls back to the
regions' longest shared ancestor, up to the root, and never drops a
region. Draft: a requested prefix wider than the subscriber's scope is
answered as if each granted region inside it were asked: a claim above a
region is presented at that region, and SYNCED follows every region's
initial set. Add one ordering rule: a publisher answers with a grant
change before the ANNOUNCE_OK of the next edit it reads after the
AUTH_OK, so a subscriber whose grant widens under a wide ask can send an
empty edit and hold new readers until its ANNOUNCE_OK. `rs/moq-net`:
the publisher reads each granted region inside an ask through its own
node key, as it reads mounts, re-narrowed on each grant change. That
also keeps the publisher from reading ungranted subtrees, which bounds
its work and leaves SYNCED's timing blind to what it hides. The
subscriber's `coarsen` widens instead of dropping. `js/net`: the answer
presents a claim above a granted region at the region; the subscriber
already falls back to the root, and gains the widening sync. No message
field changes and no public API changes. It also fixes JS's existing
root fallback.
- (A) Exception to the size rule. A subscriber's set may pass one message
when each prefix past it is a region the publisher granted. Draft: the
exception, plus how both ends derive a grant's regions from its patterns
(`interest_prefixes`, wildcards included), plus what happens when a
narrowing (an AUTH_OK replacing the grant, an AUTH_ERROR, or a lapse)
crosses an edit already sent. The acceptor judges a request when it
reads it, so that edit names a region no longer granted, and as written
the publisher must close the session. A narrowing between edits leaves
the set past the limit until the subscriber's next edit removes the
region. `rs/moq-net`: the publisher checks each extra prefix against its
grant, and `coarsen` keeps every granted region. `js/net`: the same
check, and its subscriber, which knows nothing of the grant today, has
to ask by the grant's regions.
- (B) API: a reader of an unasked region learns the session cannot answer
it, a public change in Rust, JS, and the bindings.
- (C) Neither: a reader overlapping an unasked region never syncs, a hang
in place of a wrong answer.

- Retire the quarantine, the dedup, and the hold, now that the fixtures
run on the route layer that replaces them. The quarantine
is what turns the flaws into a 2 s gap: a relay that shows a held claim
offline announces it again only after `RETRACT_QUARANTINE`. The hold and
the quarantine sit on the announce cursor, which every version shares, so
check what a lite-06 mesh loses without them
(`mesh_withdraw::cross_node_close_without_hold_reannounces` is its
positive control) before deleting them; if a lite-06 mesh still needs
them, retire them for lite-07 routes only.
- `mesh_withdraw::residual_late_better_path_restarts_on_lite06` pins a
lite-06 mesh where each relay mints its own instance, so a client sees
`Start, End, Start` while the broadcast stays live. Turn it into an update,
or record why a lite-06 mesh keeps it.
- A request naming Node ID 0 on a lite-07 session: Rust clients always name
a node, so only publisher unit tests cover a relay answering 0. Cover it
where `js/net` sends it, in `just test interop`.
D over A: D keeps the one-message rule whole, never names a region on the
wire, so a narrowing cannot turn an honest edit into a violation, and
needs no shared rule for deriving regions. It costs a presentation rule
and the publisher's narrowing, which JS's root fallback needs anyway. Done
once the choice is recorded in Plan.

## Related

Expand Down
4 changes: 2 additions & 2 deletions quest/m2/multicast/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,8 @@ and two flaws the mesh replay found,
[feeding an old relay's subscription mid-roll](/quest/m2/multicast/roll-echo-stall.md)
and [a reader starving after a lite-07 link reopens](/quest/m2/multicast/reopen-starve.md);
and on
[route layer acceptance](/quest/m2/cluster-routes/acceptance.md) retiring
the lite-06 mitigations. It does not wait on
[route layer acceptance](/quest/m2/cluster-routes/acceptance.md)'s
scoped subscriber whose grant is too wide to ask for. It does not wait on
[warmth's port](/quest/m3/cache-routing/select.md): its wire joins lite-07
only if the port has landed and passed the mesh replay when the cut is made,
and otherwise moves to the first wip version after it (maintainer,
Expand Down
25 changes: 16 additions & 9 deletions rs/moq-net/src/fuzz/mesh.rs
Original file line number Diff line number Diff line change
Expand Up @@ -74,11 +74,19 @@ struct Upstream {
struct Left(crate::origin::Claim, Arc<AtomicUsize>);

impl node::Orphan for Left {
fn release(self: Box<Self>, ended: bool) {
fn release(self: Box<Self>, gone: node::Gone) {
self.1.fetch_add(1, Ordering::Relaxed);
if ended {
self.0.withdrawn();
}
release(self.0, gone);
}
}

/// Let go of a served claim as a session does, for the reason `gone` gives.
fn release(claim: crate::origin::Claim, gone: node::Gone) {
match gone {
node::Gone::Moved => drop(claim),
node::Gone::Lost => claim.retire(false),
node::Gone::Ended => claim.retire(true),
node::Gone::Withdrawn => claim.withdrawn(),
}
}

Expand Down Expand Up @@ -376,6 +384,7 @@ impl RouteMesh {
primary: served.primary,
priority: Some(served.priority),
above: u8::try_from(served.attr.above).unwrap_or(u8::MAX),
lifecycle: served.lifecycle,
};
let route = crate::origin::RouteInfo::new()
.with_instance(instance.clone(), served.attr.epoch)
Expand Down Expand Up @@ -403,12 +412,10 @@ impl RouteMesh {
origin: node,
path,
instance,
ended,
gone,
} => {
if let Some(held) = up.served.remove(&(node, path, instance))
&& ended
{
held.claim.withdrawn();
if let Some(held) = up.served.remove(&(node, path, instance)) {
release(held.claim, gone);
}
}
}
Expand Down
28 changes: 17 additions & 11 deletions rs/moq-net/src/lite/subscriber/routed.rs
Original file line number Diff line number Diff line change
Expand Up @@ -138,11 +138,19 @@ type ServedKey = (NodeId, Path, Instance);
struct Left(crate::origin::Claim);

impl node::Orphan for Left {
fn release(self: Box<Self>, ended: bool) {
// Dropped, a claim the node let go of goes offline rather than ending.
if ended {
self.0.withdrawn();
}
fn release(self: Box<Self>, gone: node::Gone) {
release(self.0, gone);
}
}

/// Let go of a copy of a claim the origin holds, for the reason `gone` gives.
fn release(claim: crate::origin::Claim, gone: node::Gone) {
match gone {
// Dropped, it goes offline, while another copy serves.
node::Gone::Moved => drop(claim),
node::Gone::Lost => claim.retire(false),
node::Gone::Ended => claim.retire(true),
node::Gone::Withdrawn => claim.withdrawn(),
}
}

Expand Down Expand Up @@ -828,13 +836,10 @@ impl Routed {
origin,
path,
instance,
ended,
gone,
} => {
// Dropped, a claim the node let go of goes offline rather than ending.
if let Some(served) = self.served.remove(&(origin, path, instance))
&& ended
{
served.route.dynamic.withdrawn();
if let Some(served) = self.served.remove(&(origin, path, instance)) {
release(served.route.dynamic, gone);
}
}
}
Expand Down Expand Up @@ -886,6 +891,7 @@ impl Routed {
priority: Some(served.priority),
// A receiver refuses a claim above its path by more than the path's segments.
above: u8::try_from(served.attr.above).unwrap_or(u8::MAX),
lifecycle: served.lifecycle,
};
// A claim no route reaches is unavailable: it stays shown, and serves nothing.
let route = crate::origin::RouteInfo::new()
Expand Down
16 changes: 8 additions & 8 deletions rs/moq-net/src/model/follow.rs
Original file line number Diff line number Diff line change
Expand Up @@ -458,7 +458,7 @@ mod tests {
origin::{Cost, RouteInfo, RouteOptions},
};

/// The follower's next event, which must arrive within the origin's update hold.
/// The follower's next event, which must arrive within a second.
async fn followed(follow: &mut Follow) -> (&'static str, String) {
let event = kio::clock::timeout(Duration::from_secs(1), follow.next())
.await
Expand Down Expand Up @@ -488,7 +488,7 @@ mod tests {
}
}

/// The follower stays quiet past the origin's update hold.
/// The follower stays quiet for a second.
async fn quiet(follow: &mut Follow) {
if let Ok(event) = kio::clock::timeout(Duration::from_secs(1), follow.next()).await {
panic!("expected nothing, got {event:?}");
Expand Down Expand Up @@ -648,15 +648,15 @@ mod tests {
);
}

/// The next change handed out, which must arrive within the origin's update hold.
/// The next change handed out, which must arrive within a second.
async fn changed(announced: &mut Following) -> Option<broadcast::Consumer> {
kio::clock::timeout(Duration::from_secs(1), announced.available())
.await
.expect("no change")
.expect("the wait failed")
}

/// The next instance handed out, which must arrive within the origin's update hold.
/// The next instance handed out, which must arrive within a second.
async fn available(announced: &mut Following) -> broadcast::Consumer {
changed(announced).await.expect("an end, not an instance")
}
Expand All @@ -668,10 +668,9 @@ mod tests {
}
}

/// Nothing more is handed out past the origin's update hold.
/// Nothing more is handed out for a second.
async fn none(announced: &mut Following) {
let hold = super::super::origin_impl::lite06_churn::DEFAULT_UPDATE_HOLD * 2;
if let Ok(result) = kio::clock::timeout(hold, announced.available()).await {
if let Ok(result) = kio::clock::timeout(Duration::from_secs(1), announced.available()).await {
panic!(
"expected nothing, got {:?}",
result.map(|broadcast| broadcast.map(|broadcast| broadcast.info().instance.clone()))
Expand Down Expand Up @@ -746,7 +745,8 @@ mod tests {
let (_arrival, _publish) = futures::join!(available(&mut announced), async {
let _unanswered = requested(&dynamic).await;
drop(dynamic);
kio::clock::sleep(super::super::origin_impl::lite06_churn::DEFAULT_UPDATE_HOLD * 2).await;
// A gap, well inside the second `available` waits.
kio::clock::sleep(Duration::from_millis(600)).await;
origin.route("live", RouteOptions::default()).unwrap()
});
none(&mut announced).await;
Expand Down
Loading