From c220a7286e5f709b315df99bb6e52d6cbaba229c Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 18 Nov 2026 13:09:46 -0800 Subject: [PATCH 1/2] quest: plan rollout confidence for the lite-07 route layer Four quests from a survey of a downstream deployment: route counters in a relay's stats and metrics, a real-process mesh replay with a mixed roll, a threaded stress of the route layer, and the track replacement hang under Bun. Co-Authored-By: Claude Fable 5.2 --- .../m0/test-flakes-3/track-replacement-bun.md | 29 ++++++++ quest/m3/mesh-replay.md | 71 +++++++++++++++++++ quest/m3/route-counters.md | 49 +++++++++++++ quest/m3/route-race-stress.md | 44 ++++++++++++ 4 files changed, 193 insertions(+) create mode 100644 quest/m0/test-flakes-3/track-replacement-bun.md create mode 100644 quest/m3/mesh-replay.md create mode 100644 quest/m3/route-counters.md create mode 100644 quest/m3/route-race-stress.md diff --git a/quest/m0/test-flakes-3/track-replacement-bun.md b/quest/m0/test-flakes-3/track-replacement-bun.md new file mode 100644 index 0000000000..99261515ad --- /dev/null +++ b/quest/m0/test-flakes-3/track-replacement-bun.md @@ -0,0 +1,29 @@ +# [M] Track replacement under Bun finishes every run + +## Goal + +`just test interop --all` passes `track replacement js-native-bun` on +WebSocket and WebTransport on every run. + +## Plan + +Seen while validating two PRs on 2026-11-18, and not caused by either: + +- [moq-dev/moq#9053](https://github.com/moq-dev/moq/pull/9053): 1 of 4 runs + on a clean `origin/main`, 2 of 4 on the branch, on either transport. +- [moq-dev/moq#9084](https://github.com/moq-dev/moq/pull/9084): once on + WebSocket; the rerun passed. + +The client (`test/interop/clients/js-native/replace.ts`, run by +`run_replacement_case` in `test/interop/interop.sh`) hangs until the harness +timeout instead of printing `replacement automatic=64 immediate=2 resumed=2 +missing-group-fetch=2`. Every failure recorded was under Bun; whether Node +hangs too is not known. + +The case runs on moq-lite-07-wip, whose relay side +[moq-dev/moq#8987](https://github.com/moq-dev/moq/pull/8987) replaced. Find +first whether the hang predates it, then whether it sits in the relay, +`js/net`, or the native addon under Bun, and fix that cause rather than the +timeout. + +Public API: none. Wire: none expected. diff --git a/quest/m3/mesh-replay.md b/quest/m3/mesh-replay.md new file mode 100644 index 0000000000..3c246f1f87 --- /dev/null +++ b/quest/m3/mesh-replay.md @@ -0,0 +1,71 @@ +# [L] Real relays replay a mesh and its faults + +## Goal + +One recipe starts a `moq-relay` process for each relay of a mesh read from a +file, joins them over links that queue until they drop, and plays a script +of faults against it while publishers and subscribers read through: + +- a link reopening, and one relay reopening every link it has, twice; +- a relay killed and restarted; +- a roll that drains each relay with GOAWAY and replaces its binary, one + relay at a time, from the last published release to this checkout and back. + +It fails when a reader with a route to its broadcast stays unfed, a +broadcast stays shown after its source ended, or a relay does not go quiet +once the script does. For each event it reports the route and announce +messages and bytes, the time until the mesh is quiet, the broadcasts +rewritten into the origin, and each reader's interruption. + +Nightly runs it on a fixed mesh whose routes cross three links. A deployment +replays its own graph by passing its own file. + +## Plan + +Decided 2026-11-18 (maintainer, in the coordinating session, from a survey of +the moq.pro deployment): ✅ planned (rejected: leaving it out). What it adds +to the tests that exist: + +- The route simulator and its sweeps run generated shapes over mock + sessions. The drills inject one fault over real sockets into a relay or a + pair. `just test wire-compat` and the cut's two-relay smoke pair one + released relay with one from this checkout. +- moq.pro rolls one relay at a time, so its mesh spends hours with the + newest and previous release meshed, and lite-07 is chosen link by link: + only two new relays speak it to each other. Nothing runs a mesh in that + state, where one path crosses both versions. +- The same decision keeps the per-broadcast rewrite of + [route benches](/quest/m3/route-benches.md) in m3 on the condition that it + is measured on a deployed mesh's shape before that mesh's cluster links + move to lite-07. This quest's report is that measurement. + +Agent's choices, for the implementer to revisit: + +- m3, as work a downstream deployment waits on. It does not gate the lite-07 + cut. A flaw it finds in the wire is still fixed before the cut if the cut + has not happened. +- Reuse the drills' shaper for the links and `test/interop/compat.sh`'s + download of the released relay. +- The file names relays and links, each link with its cost and whether + either end marks it. The script is a list of timed events, so two runs + share one sequence and their reports compare. +- Sweep the report over both axes: relays in the mesh and broadcasts per + relay. +- The counts are a report, not a gate, as benchmarks are (2026-10-20, + maintainer). +- A pair of edges joined over lite-06 can still transit, a known gap of the + lite-06 path that ends when the mesh finishes its roll. Name it in the + report instead of failing on it. + +Public API: none. Wire: none. + +## Required + +- [A relay counts what its route layer does](/quest/m3/route-counters.md) - the counts the report reads from each relay + +## Related + +- [Route benches](/quest/m3/route-benches.md) - the costs this measures on a real mesh +- [The cases no test covers run on real sessions](/quest/m2/multicast/accept.md) - the same failures on mock sessions with mocked time +- [Random fault schedules](/quest/m4/cluster-fault-schedules.md) - searches fault timings on the simulated clock +- [A nightly soak](/quest/m4/relay-soak.md) - one real relay under churn diff --git a/quest/m3/route-counters.md b/quest/m3/route-counters.md new file mode 100644 index 0000000000..53edb3dd40 --- /dev/null +++ b/quest/m3/route-counters.md @@ -0,0 +1,49 @@ +# [M] A relay counts what its route layer does + +## Goal + +From a relay's `/metrics` and its stats, an operator rolling a mesh onto +moq-lite-07 can tell, per peer label and without reading logs: + +- how often its routes change, and how many broadcasts each change rewrote + into the origin; +- how many withdrawals it sent and received, DEAD and DRAIN apart; +- why it refused a route or a claim, by kind, where `routes_refused` gives + one number today; +- whether a peer's announce queue fell behind or overran; +- and how many sessions it closed for a route error, by kind. + +`doc/concept/stats.md` says what a rise in each one means during a rollout. + +## Plan + +Decided 2026-11-18 (maintainer, in the coordinating session, from a survey of +the moq.pro deployment): ✅ planned (rejected: leaving it out). The route +layer's only signal in a relay's stats is `routes_refused`, and `/metrics` +has announce and control counters and nothing for routes. moq.pro judges a +canary relay by its counters against its neighbours', so a route layer that +churns or disagrees between releases is invisible to the roll that would +catch it. + +Agent's choices, for the implementer to revisit: + +- m3, as work a downstream deployment waits on. It does not gate the lite-07 + cut: the fields are additive, and a deployment needs them in the release + it rolls onto lite-07 cluster links, which is no earlier than the cut. +- Count in `rs/moq-net/src/stats.rs` beside `routes_refused`, per label and + never per broadcast or per node, so the cost and the output stay flat as + the mesh grows. +- The count of broadcasts rewritten is the cost + [route benches](/quest/m3/route-benches.md) measures, read from a running + relay. +- Say what the announce counters mean on a lite-07 session, where a route + change is no longer one announce per broadcast, and fix the doc or the + counter if they now mislead. + +Public API: stats fields and metric names, additive; nest them under their +shared role. Wire: none. + +## Related + +- [Route benches](/quest/m3/route-benches.md) - measures the rewrite these count +- [Real relays replay a mesh and its faults](/quest/m3/mesh-replay.md) - reads these in its report diff --git a/quest/m3/route-race-stress.md b/quest/m3/route-race-stress.md new file mode 100644 index 0000000000..fcca3a8b4a --- /dev/null +++ b/quest/m3/route-race-stress.md @@ -0,0 +1,44 @@ +# [M] Nightly races the route layer across threads + +## Goal + +Nightly runs a seeded stress of a publisher, a relay, and readers over mock +moq-lite-07 sessions on a multi-threaded runtime, mixing the steps whose +order two tasks decide: a broadcast ending as another starts on the same +session, a request arriving as its route lands, a grant widening, and a +session closing during an edit. A reader that hangs, or a request refused +for a claim the reader was shown, fails the run with its seed, its round, +and the steps in flight. PRs run a short fixed count. + +## Plan + +Decided 2026-11-18 (maintainer, in the coordinating session): ✅ planned +(rejected: leaving it out). + +[moq-dev/moq#8987](https://github.com/moq-dev/moq/pull/8987) put a claim in +front of a reader before its route reached the origin, so a request could be +refused as unroutable. Every route test scripts its order on one thread, and +only the interop `finish` lane saw it, intermittently. +[moq-dev/moq#9052](https://github.com/moq-dev/moq/pull/9052) found it with a +scratch stress that was not kept: one publisher, one relay, and 8 readers, +ending one broadcast and announcing another on the same session. It failed +within 72 to 827 rounds before the fix and ran 18,000 clean after. + +Agent's choices, for the implementer to revisit: + +- m3: it tests code, not the wire, so a bug it finds after the cut is a + patch release. +- Show it has teeth: with #9052's fix reverted it fails inside the nightly + budget. Measure the rounds that takes and report it. +- Real threads do not replay from a seed. Try a controlled scheduler first, + such as `shuttle`, if the mock sessions and `kio` run under one; otherwise + log the steps in flight so a failure can be read without a replay. +- [Random fault schedules](/quest/m4/cluster-fault-schedules.md) searches + fault timings on the simulated clock and may come to cover this. Say in + the PR which orders it cannot reach. + +Public API: none. Wire: none. + +## Related + +- [Random fault schedules](/quest/m4/cluster-fault-schedules.md) - the simulated search this sits beside From 77274d7b04e8968757ad95d1868b7825324b94a3 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Wed, 18 Nov 2026 14:01:24 -0800 Subject: [PATCH 2/2] quest: the mesh replay and route counters block the lite-07 cut Maintainer decision on moq-dev/moq#9110: both move under the multicast epic in m2, since a wire flaw found on a real mesh after the cut would wait for lite-08. Co-Authored-By: Claude Fable 5.2 --- quest/m2/multicast/README.md | 4 +++- quest/{m3 => m2/multicast}/mesh-replay.md | 12 ++++++++---- quest/{m3 => m2/multicast}/route-counters.md | 12 ++++++++---- 3 files changed, 19 insertions(+), 9 deletions(-) rename quest/{m3 => m2/multicast}/mesh-replay.md (85%) rename quest/{m3 => m2/multicast}/route-counters.md (81%) diff --git a/quest/m2/multicast/README.md b/quest/m2/multicast/README.md index 0196142d0c..fa5620962e 100644 --- a/quest/m2/multicast/README.md +++ b/quest/m2/multicast/README.md @@ -32,7 +32,9 @@ chain name its announcing relay, then each grant's exporter and adopter. Its proof is `rs/moq-net/verify/route-algebra/`, and the quests written against the old wire were retargeted or deleted. The lite-07 cut still waits on this epic's -last child, [the uncovered cases](/quest/m2/multicast/accept.md), and on +children, [the uncovered cases](/quest/m2/multicast/accept.md), +[route counters](/quest/m2/multicast/route-counters.md), and +[the mesh replay](/quest/m2/multicast/mesh-replay.md), and on [route layer acceptance](/quest/m2/cluster-routes/acceptance.md) retiring the lite-06 mitigations. After the cut come warmth's wire ([cache-aware routing](/quest/m3/cache-routing/README.md)), diff --git a/quest/m3/mesh-replay.md b/quest/m2/multicast/mesh-replay.md similarity index 85% rename from quest/m3/mesh-replay.md rename to quest/m2/multicast/mesh-replay.md index 3c246f1f87..3411373d2a 100644 --- a/quest/m3/mesh-replay.md +++ b/quest/m2/multicast/mesh-replay.md @@ -39,11 +39,15 @@ to the tests that exist: is measured on a deployed mesh's shape before that mesh's cluster links move to lite-07. This quest's report is that measurement. +Decided 2026-11-18 (maintainer, on +[moq-dev/moq#9110](https://github.com/moq-dev/moq/pull/9110)): ✅ it blocks +the lite-07 cut, under this epic (rejected: m3, starting now without the cut +waiting on it). After the cut the lite-07 wire is frozen, so a wire flaw +found on a real mesh mid-roll would wait for lite-08. A flaw it finds is +fixed on lite-07-wip, in the quest that owns the code, before the cut. + Agent's choices, for the implementer to revisit: -- m3, as work a downstream deployment waits on. It does not gate the lite-07 - cut. A flaw it finds in the wire is still fixed before the cut if the cut - has not happened. - Reuse the drills' shaper for the links and `test/interop/compat.sh`'s download of the released relay. - The file names relays and links, each link with its cost and whether @@ -61,7 +65,7 @@ Public API: none. Wire: none. ## Required -- [A relay counts what its route layer does](/quest/m3/route-counters.md) - the counts the report reads from each relay +- [A relay counts what its route layer does](/quest/m2/multicast/route-counters.md) - the counts the report reads from each relay ## Related diff --git a/quest/m3/route-counters.md b/quest/m2/multicast/route-counters.md similarity index 81% rename from quest/m3/route-counters.md rename to quest/m2/multicast/route-counters.md index 53edb3dd40..dc6122356e 100644 --- a/quest/m3/route-counters.md +++ b/quest/m2/multicast/route-counters.md @@ -25,11 +25,15 @@ canary relay by its counters against its neighbours', so a route layer that churns or disagrees between releases is invisible to the roll that would catch it. +Decided 2026-11-18 (maintainer, on +[moq-dev/moq#9110](https://github.com/moq-dev/moq/pull/9110)): ✅ it blocks +the lite-07 cut, under this epic, with +[the mesh replay](/quest/m2/multicast/mesh-replay.md) that reads these +(rejected: m3). The release a deployment rolls onto lite-07 then carries +them. + Agent's choices, for the implementer to revisit: -- m3, as work a downstream deployment waits on. It does not gate the lite-07 - cut: the fields are additive, and a deployment needs them in the release - it rolls onto lite-07 cluster links, which is no earlier than the cut. - Count in `rs/moq-net/src/stats.rs` beside `routes_refused`, per label and never per broadcast or per node, so the cost and the output stay flat as the mesh grows. @@ -46,4 +50,4 @@ shared role. Wire: none. ## Related - [Route benches](/quest/m3/route-benches.md) - measures the rewrite these count -- [Real relays replay a mesh and its faults](/quest/m3/mesh-replay.md) - reads these in its report +- [Real relays replay a mesh and its faults](/quest/m2/multicast/mesh-replay.md) - reads these in its report