From 7b5220d1b465a589a5a4b105709a9cbdfa2c8178 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Thu, 19 Nov 2026 14:29:20 -0800 Subject: [PATCH 1/3] chore(quest): claim quest/m2/multicast/splice-stall.md Quest-Claim: 7614581735c6adc5 From 0e7fce025d392693bc8ee49a7f4770aabb162750 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Thu, 19 Nov 2026 18:51:01 -0800 Subject: [PATCH 2/3] quest: splice-stall narrows to a reader stranded by an Unroutable its relay sends 19 of 20 reopen.toml runs on main pass; the failure starved 2 reader paths whose relay reset them Unroutable while still showing the broadcast. Reverting #9179 brings back reopen-starve (4 of 8 failed). Pauses match lite-06 per build. Co-Authored-By: Claude Opus 5.5 --- quest/m2/multicast/README.md | 2 +- quest/m2/multicast/splice-stall.md | 107 ++++++++++++++++++----------- test/mesh-replay/script.toml | 6 +- 3 files changed, 71 insertions(+), 44 deletions(-) diff --git a/quest/m2/multicast/README.md b/quest/m2/multicast/README.md index b9c1ea92d7..fd7f170be2 100644 --- a/quest/m2/multicast/README.md +++ b/quest/m2/multicast/README.md @@ -41,7 +41,7 @@ waits on this epic's children, the other flaw, [a route moving behind one next hop resending a group](/quest/m2/multicast/resend.md), whose fix changes the wire, and a flaw the mesh replay found, -[a lite-07 front that stops delivering after it switches copies](/quest/m2/multicast/splice-stall.md); +[a lite-07 reader its relay refuses as unroutable and never re-asks](/quest/m2/multicast/splice-stall.md); and on [route layer acceptance](/quest/m2/cluster-routes/acceptance.md)'s scoped subscriber whose grant is too wide to ask for. It does not wait on diff --git a/quest/m2/multicast/splice-stall.md b/quest/m2/multicast/splice-stall.md index f41d505120..70951f7045 100644 --- a/quest/m2/multicast/splice-stall.md +++ b/quest/m2/multicast/splice-stall.md @@ -1,11 +1,13 @@ -# [M] A front keeps delivering after it switches copies on lite-07 +# [M] A reader refused as unroutable reads again on lite-07 ## Goal -On lite-07, a relay's front that switches to a new copy after a link reopens -or a relay crashes keeps delivering every group to its readers, as the -release (lite-06) does. Today it can deliver for a few seconds, then hand -out one group every 15-25s for a minute or more, with its route unchanged. +On lite-07, a reader whose relay still shows its broadcast keeps reading after +a link reopens or a relay crashes, as the release (lite-06) does. Today, about +one run in twenty, a relay answers a reader's live subscription with +Unroutable while it keeps announcing the broadcast unchanged. The reader's +track strands, waiting for a route change that never comes, and starves for +the rest of the run without an error. No reader starves in `just test mesh-replay --mesh test/mesh-replay/ring4.toml --script test/mesh-replay/reopen.toml --broadcasts 32` @@ -14,43 +16,68 @@ or in the nightly ring meshes, once `[[known]]` no longer names this quest in ## Plan -Found by [the mesh replay](/test/mesh-replay/README.md) while fixing -[reopen-starve](https://github.com/moq-dev/moq/pull/9179), whose fix leaves no -request refused and never asked again. With that fix, 9 of 13 `reopen.toml` -runs on ring4 at 32 broadcasts passed (2 of 3 with -[#9154](https://github.com/moq-dev/moq/pull/9154)), and the others starved 3 -to 205 reader paths. Before it, 17 of 24 passed, and failures starved 1 to 32 -paths, some of them stranded requests. - -What an instrumented failing run shows, after every link of `a` reopened: - -- `a`'s front on `b/0` switched to the direct route to `b` as `b`'s copy of - the claims caught up, spliced the new copy, and fed its reader for 13s. - From then on its route never changed and no request was refused, yet - `b`'s groups reached `a`'s reader once every 15-25s. -- Over the same minute, `d`'s reader stalled the same way on all of `c`'s - broadcasts, through a front on the `c`-`d` link that the fault never - touched and that never switched. -- No shaper dropped or overflowed a packet, and every reader's status kept - time. - -Pre-fix runs showed the same signature on a handful of paths: a front -spliced a copy that delivered, then its reader got nothing for 57-75s. - -Likely cause, untested: [roll-echo-stall](https://github.com/moq-dev/moq/pull/9174)'s -first fix. A shared lite-07 copy never places a time start that a wider -range already covers, so the subscription it feeds waits. A spliced copy -resumes from a time. Its head could not be tested here: it no longer builds -on main, and its merge is being redone. Once it lands, rerun `reopen.toml` -several times. If no reader starves, delete this quest and its `[[known]]` -entry. Otherwise, instrument the stalled subscription on both ends of the -link: which groups the publisher serves, and what the subscriber's window -waits on. +Found by [the mesh replay](/test/mesh-replay/README.md). On main after +[roll-echo-stall](https://github.com/moq-dev/moq/pull/9174) and +[reopen-starve](https://github.com/moq-dev/moq/pull/9179), 19 of 20 +`reopen.toml` runs (ring4, 32 broadcasts, debug build) passed. The failing run +starved 2 reader paths. + +What that run's logs show. `b`'s reader followed `d/0` and `d/3` through `a`. +The `a`-`b` link paused, and its session idle-timed out 10s later. At that +moment, relay `b` reset the reader's live subscriptions for exactly those two +paths with Unroutable (`subscribe error ... err=unroutable` in the reader's +log). The reader's follow logged no end and stayed online. Relay `b` never +withdrew or updated `d/0`, so the reader's front, which strands a track +refused as unroutable until its table moves (`model/front.rs` +`track_ended` and `refuse`, `Track::stranded`, `unstrand`), never asked again. +The other 30 paths at `b` switched to the copy through `c`. That copy handed +out about one group every 5s until the `a`-`b` link returned, then fed every +reader again. + +Reopen-starve's first plan named this strand: a refused track waits for +its table to move. #9179 removed the refusal between relays (a request follows +a claim only through a neighbour that holds it). The strand remains one hop +down, between a relay and its client. + +Open, and why it could not be settled yet: + +- Why relay `b` ended those two tracks Unroutable. The run had info logs only. + Candidates: `Front::selected` finding no candidate (#9179's `holds` filter + leaves none while a neighbour's copy of the claims lags its route) with no + orphaned claim to wait on, so the front ends `Unroutable`. Or the publisher's + node filter refusing. Reruns with debug logs on `lite::publisher`, + `lite::subscriber`, and `model` failed only on hosts saturated by other + work (100% CPU, relays at 5-30% of a core). There the measuring reader + itself starved on loopback, so those runs show nothing about this bug. +- A mock-time reproduction. A ring4 integration test on `peer_with_latency` + links passed: 8 to 32 broadcasts from `d`, readers on `a`, `b`, and `c` + over their own sessions, one link cut or paused. It also passed with the + cut swept from 0 to 1.5s after the readers start. The window needs timing + the mocked clock does not hit. + +Recommended fix, pending the cause: a hop must not answer a live subscription +with Unroutable while it keeps showing that broadcast available. Either the +front waits for a way as it does for an orphaned claim (`Front::waiting`), or +the hop updates its announcement so the downstream's table moves. Fixing only +the client to re-ask on a quiet table would need a timer and could loop on a +standing refusal. Rerun `/verify-failover`, since either changes the front +lifecycle it covers. + +Next steps: capture one failure with debug logs on a host that is not +saturated, and record per-relay CPU with it to tell load apart from logic. +Find which path in relay `b` produced the Unroutable. Write the test at the +lowest layer that shows it, likely a front unit test in `model/front.rs`. +Then fix it and rerun `reopen.toml` at least eight times. + +Pauses are not part of this quest. A debug build pauses about as long on +lite-06 as on lite-07 under the same load: link 13-14s, relay reopen 24-35s, +kill 25-30s. A release build pauses the same on both versions: link 11-12s, +relay reopen 17-19s, kill 20-23s. The release 0.17.17 binary pauses 10s, 16s, +and 19s at a lower load. The lite-07 cut waits on it, as decided for reopen-starve on 2026-11-19 -(under the maintainer's delegation for lite-07 routing calls): it is the -same starved reader after a link flap or a crash, which a deployment sees -daily. +(under the maintainer's delegation for lite-07 routing calls): it is a +starved reader after a link flap or a crash, which a deployment sees daily. Not a wire change. diff --git a/test/mesh-replay/script.toml b/test/mesh-replay/script.toml index 90f2a9c9a2..4919b0359b 100644 --- a/test/mesh-replay/script.toml +++ b/test/mesh-replay/script.toml @@ -45,9 +45,9 @@ kind = "stuck" event = "roll" quest = "/quest/m3/release/cut.md" -# On lite-07 alone, a front that switched copies after a reopened link or a crashed relay can -# deliver a group only every 15-25s on an unchanged route. Not on a roll, so a roll's starved -# reader still fails. `reopen.toml` reproduces it and lists nothing known. +# On lite-07 alone, a relay can answer a reader's live subscription Unroutable after a reopened +# link or a crashed relay while still showing the broadcast, and the reader never asks again. +# Not on a roll, so a roll's starved reader still fails. `reopen.toml` lists nothing known. [[known]] check = "unfed" kind = "starved" From 3235a226148ca23de0e1fd6197248d93db32ddd2 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Thu, 19 Nov 2026 19:02:15 -0800 Subject: [PATCH 3/3] quest: splice-stall records the runs after resend landed Co-Authored-By: Claude Opus 5.5 --- quest/m2/multicast/splice-stall.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/quest/m2/multicast/splice-stall.md b/quest/m2/multicast/splice-stall.md index 70951f7045..f48499d04a 100644 --- a/quest/m2/multicast/splice-stall.md +++ b/quest/m2/multicast/splice-stall.md @@ -20,7 +20,12 @@ Found by [the mesh replay](/test/mesh-replay/README.md). On main after [roll-echo-stall](https://github.com/moq-dev/moq/pull/9174) and [reopen-starve](https://github.com/moq-dev/moq/pull/9179), 19 of 20 `reopen.toml` runs (ring4, 32 broadcasts, debug build) passed. The failing run -starved 2 reader paths. +starved 2 reader paths. After +[resend](https://github.com/moq-dev/moq/pull/9155) landed, 2 of 3 passed. The +third shows the same strand: `a`'s reader got Unroutable for `c/5` as `a`'s +links reopened, then read nothing for 73s, across a re-follow of the path. +That run's 128 paths starved at the kill came while the host ran at 100% CPU, +so they are load, not this bug. What that run's logs show. `b`'s reader followed `d/0` and `d/3` through `a`. The `a`-`b` link paused, and its session idle-timed out 10s later. At that