Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion quest/m2/multicast/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ through a lost claim's debounce, so a reader resumes on a detour learned after
its session closed, and a route moving behind one next hop no longer resends a
group, since a lite-07 range may start partway through one. The lite-07 cut
still waits on this epic's children, a flaw the mesh replay found,
[a lite-07 front that stops delivering after it switches copies](/quest/m2/multicast/splice-stall.md);
[a lite-07 reader its relay refuses as unroutable and never re-asks](/quest/m2/multicast/splice-stall.md);
and on
[route layer acceptance](/quest/m2/cluster-routes/acceptance.md)'s
scoped subscriber whose grant is too wide to ask for. It does not wait on
Expand Down
112 changes: 72 additions & 40 deletions quest/m2/multicast/splice-stall.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
# [M] A front keeps delivering after it switches copies on lite-07
# [M] A reader refused as unroutable reads again on lite-07

## Goal

On lite-07, a relay's front that switches to a new copy after a link reopens
or a relay crashes keeps delivering every group to its readers, as the
release (lite-06) does. Today it can deliver for a few seconds, then hand
out one group every 15-25s for a minute or more, with its route unchanged.
On lite-07, a reader whose relay still shows its broadcast keeps reading after
a link reopens or a relay crashes, as the release (lite-06) does. Today, about
one run in twenty, a relay answers a reader's live subscription with
Unroutable while it keeps announcing the broadcast unchanged. The reader's
track strands, waiting for a route change that never comes, and starves for
the rest of the run without an error.

No reader starves in
`just test mesh-replay --mesh test/mesh-replay/ring4.toml --script test/mesh-replay/reopen.toml --broadcasts 32`
Expand All @@ -14,43 +16,73 @@ or in the nightly ring meshes, once `[[known]]` no longer names this quest in

## Plan

Found by [the mesh replay](/test/mesh-replay/README.md) while fixing
[reopen-starve](https://github.com/moq-dev/moq/pull/9179), whose fix leaves no
request refused and never asked again. With that fix, 9 of 13 `reopen.toml`
runs on ring4 at 32 broadcasts passed (2 of 3 with
[#9154](https://github.com/moq-dev/moq/pull/9154)), and the others starved 3
to 205 reader paths. Before it, 17 of 24 passed, and failures starved 1 to 32
paths, some of them stranded requests.

What an instrumented failing run shows, after every link of `a` reopened:

- `a`'s front on `b/0` switched to the direct route to `b` as `b`'s copy of
the claims caught up, spliced the new copy, and fed its reader for 13s.
From then on its route never changed and no request was refused, yet
`b`'s groups reached `a`'s reader once every 15-25s.
- Over the same minute, `d`'s reader stalled the same way on all of `c`'s
broadcasts, through a front on the `c`-`d` link that the fault never
touched and that never switched.
- No shaper dropped or overflowed a packet, and every reader's status kept
time.

Pre-fix runs showed the same signature on a handful of paths: a front
spliced a copy that delivered, then its reader got nothing for 57-75s.

Likely cause, untested: [roll-echo-stall](https://github.com/moq-dev/moq/pull/9174)'s
first fix. A shared lite-07 copy never places a time start that a wider
range already covers, so the subscription it feeds waits. A spliced copy
resumes from a time. Its head could not be tested here: it no longer builds
on main, and its merge is being redone. Once it lands, rerun `reopen.toml`
several times. If no reader starves, delete this quest and its `[[known]]`
entry. Otherwise, instrument the stalled subscription on both ends of the
link: which groups the publisher serves, and what the subscriber's window
waits on.
Found by [the mesh replay](/test/mesh-replay/README.md). On main after
[roll-echo-stall](https://github.com/moq-dev/moq/pull/9174) and
[reopen-starve](https://github.com/moq-dev/moq/pull/9179), 19 of 20
`reopen.toml` runs (ring4, 32 broadcasts, debug build) passed. The failing run
starved 2 reader paths. After
[resend](https://github.com/moq-dev/moq/pull/9155) landed, 2 of 3 passed. The
third shows the same strand: `a`'s reader got Unroutable for `c/5` as `a`'s
links reopened, then read nothing for 73s, across a re-follow of the path.
That run's 128 paths starved at the kill came while the host ran at 100% CPU,
so they are load, not this bug.

What that run's logs show. `b`'s reader followed `d/0` and `d/3` through `a`.
The `a`-`b` link paused, and its session idle-timed out 10s later. At that
moment, relay `b` reset the reader's live subscriptions for exactly those two
paths with Unroutable (`subscribe error ... err=unroutable` in the reader's
log). The reader's follow logged no end and stayed online. Relay `b` never
withdrew or updated `d/0`, so the reader's front, which strands a track
refused as unroutable until its table moves (`model/front.rs`
`track_ended` and `refuse`, `Track::stranded`, `unstrand`), never asked again.
The other 30 paths at `b` switched to the copy through `c`. That copy handed
out about one group every 5s until the `a`-`b` link returned, then fed every
reader again.

Reopen-starve's first plan named this strand: a refused track waits for
its table to move. #9179 removed the refusal between relays (a request follows
a claim only through a neighbour that holds it). The strand remains one hop
down, between a relay and its client.

Open, and why it could not be settled yet:

- Why relay `b` ended those two tracks Unroutable. The run had info logs only.
Candidates: `Front::selected` finding no candidate (#9179's `holds` filter
leaves none while a neighbour's copy of the claims lags its route) with no
orphaned claim to wait on, so the front ends `Unroutable`. Or the publisher's
node filter refusing. Reruns with debug logs on `lite::publisher`,
`lite::subscriber`, and `model` failed only on hosts saturated by other
work (100% CPU, relays at 5-30% of a core). There the measuring reader
itself starved on loopback, so those runs show nothing about this bug.
- A mock-time reproduction. A ring4 integration test on `peer_with_latency`
links passed: 8 to 32 broadcasts from `d`, readers on `a`, `b`, and `c`
over their own sessions, one link cut or paused. It also passed with the
cut swept from 0 to 1.5s after the readers start. The window needs timing
the mocked clock does not hit.

Recommended fix, pending the cause: a hop must not answer a live subscription
with Unroutable while it keeps showing that broadcast available. Either the
front waits for a way as it does for an orphaned claim (`Front::waiting`), or
the hop updates its announcement so the downstream's table moves. Fixing only
the client to re-ask on a quiet table would need a timer and could loop on a
standing refusal. Rerun `/verify-failover`, since either changes the front
lifecycle it covers.

Next steps: capture one failure with debug logs on a host that is not
saturated, and record per-relay CPU with it to tell load apart from logic.
Find which path in relay `b` produced the Unroutable. Write the test at the
lowest layer that shows it, likely a front unit test in `model/front.rs`.
Then fix it and rerun `reopen.toml` at least eight times.

Pauses are not part of this quest. A debug build pauses about as long on
lite-06 as on lite-07 under the same load: link 13-14s, relay reopen 24-35s,
kill 25-30s. A release build pauses the same on both versions: link 11-12s,
relay reopen 17-19s, kill 20-23s. The release 0.17.17 binary pauses 10s, 16s,
and 19s at a lower load.

The lite-07 cut waits on it, as decided for reopen-starve on 2026-11-19
(under the maintainer's delegation for lite-07 routing calls): it is the
same starved reader after a link flap or a crash, which a deployment sees
daily.
(under the maintainer's delegation for lite-07 routing calls): it is a
starved reader after a link flap or a crash, which a deployment sees daily.

Not a wire change.

Expand Down
6 changes: 3 additions & 3 deletions test/mesh-replay/script.toml
Original file line number Diff line number Diff line change
Expand Up @@ -45,9 +45,9 @@ kind = "stuck"
event = "roll"
quest = "/quest/m3/release/cut.md"

# On lite-07 alone, a front that switched copies after a reopened link or a crashed relay can
# deliver a group only every 15-25s on an unchanged route. Not on a roll, so a roll's starved
# reader still fails. `reopen.toml` reproduces it and lists nothing known.
# On lite-07 alone, a relay can answer a reader's live subscription Unroutable after a reopened
# link or a crashed relay while still showing the broadcast, and the reader never asks again.
# Not on a roll, so a roll's starved reader still fails. `reopen.toml` lists nothing known.
[[known]]
check = "unfed"
kind = "starved"
Expand Down