Skip to content

Real relays replay a mesh and its faults - #9153

Merged
kixelated merged 20 commits into
mainfrom
quest/m2/multicast/mesh-replay
Nov 19, 2026
Merged

kixelated merged 20 commits into
mainfrom
quest/m2/multicast/mesh-replay

Conversation

@kixelated

@kixelated kixelated commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Closes quest/m2/multicast/mesh-replay.md.

Problem

lite-07 sessions run a new route layer, and a CDN is about to roll its relays onto it one at a time. Nothing ran a mesh of real moq-relay processes through a link reopen, a relay crash, and above all a roll, where two new relays speak lite-07 to each other and lite-06 to an old one, so one route crosses both.

Approach

just test mesh-replay (README) meshes real relays over moq-shaper links (bufferbloat, never random loss) from a topology file, runs a publisher and a follower on every relay, and plays a script of events:

  • roll drains each relay with SIGTERM and starts the other build on its port: the newest release (downloaded and checksummed like compat.sh, now shared as test/lib/release.sh) or this checkout, or any relay binary with --new (the warmth port's branch build). A roll can stop partway (relays = [...]).
  • reopen-link pauses one link's shaper past the idle timeout, reopen-relay all of a relay's, kill is SIGKILL and a restart, end-source stops one relay's publisher and starts it again.

For each event the report gives how long until the route and announce counters went quiet and every routed reader read again, the announce-class messages and bytes, the moq_relay_route_* counters from #9126 (changed, rewritten, DEAD and DRAIN sent and received), readers hit and how many were bystanders, the longest pause, groups missed, and readers whose broadcast stepped through End, Start, End (#9112). A roll also lists, per link of the drained relay, whether the neighbour heard GOAWAY (DRAIN) or only saw it close (DEAD), read from either build's log and cross-checked against the new build's GOAWAY counter, starring the links the drained relay dialed, so the table stays comparable once drain-dialed-links lands.

A run fails only on a reader whose relay shows its live broadcast but that reads nothing after the event settles (stuck when its follow holds an ended track, starved otherwise), a relay still showing a stopped broadcast, or route and announce messages still moving once every publisher stopped. A violation a quest already owns is listed under [[known]] in the script, reported with its quest, and does not fail the run. Counts never gate anything.

moq-bench replay is the publisher and follower: one single-frame group per interval per broadcast, and a JSON line per gap, per online/offline/ended change, and per second of state.

Nightly runs a ring of four and a ring of six with a chord (diameter 3), each at 4 and 32 broadcasts per relay, and adds every report to the job summary.

Impact

  • New moq-bench replay subcommand. No library API or wire change.
  • test/interop/compat.sh fetches released binaries through test/lib/release.sh, which also picks aarch64 assets.
  • Two nightly jobs.
  • moq-relay's repository_configs_load test skips test/mesh-replay/, whose TOML files are topologies and event scripts, not relay configs.

Flaws found

  • New: roll-echo-stall. Mid-roll, a new relay accepts an old neighbour's lite-06 SUBSCRIBE that arrives the moment the neighbour withdraws its echo of the broadcast, then forwards no group until the broadcast ends, while its own upstream for that broadcast stays live. The old relay's viewers see the broadcast and read nothing. Reproduction: just test mesh-replay --mesh test/mesh-replay/ring6.toml --script test/mesh-replay/half-roll.toml --broadcasts 32 (rolling b starved old relay c's readers in every ring6 cell at 32 broadcasts, 33 to 41 paths; 1 of 3 half-roll runs at 4). The quest has the log trace. Not fixed here: it is in the publisher and model code other quests own.
  • detour-resume, as known, on real relays: rolls left readers ended (unroutable, internal error) and never handed out another, about 32 paths per roll in ring6 at 32 broadcasts. Noted in the quest.
  • Harness bugs, fixed here, that earlier reports carry: the follower ended its whole read on one group it could not finish (given up as old), which looked like detour-resume; and a pause still going at an event's end was measured past the reader's latest status line, which counted nearly every reader as hit, bystanders included.
  • New: reopen-starve. On lite-07 alone, at 32 broadcasts per relay, a reader two hops from a relay whose links reopened (or that crashed) shows the broadcast live and reads nothing for 75s or more: its relay subscribed through a neighbour whose own route had not returned yet, was refused as unroutable, and never asked again. The release (lite-06) settles the same faults in at most 4.1s. Reproduction: just test mesh-replay --mesh test/mesh-replay/ring4.toml --script test/mesh-replay/reopen.toml --broadcasts 32 (new reopen.toml; starved readers in all 3 runs). The quest has the log trace and a likely cause in model/front.rs's stranded tracks. Not fixed here, for the same reason as roll-echo-stall.
  • drain-dialed-links, as known: every link a rolled relay dialed shows DEAD, every link it accepted DRAIN.
  • resend and lite-06 edge-pair transit: not separately visible at this harness's resolution (it counts missed groups, not duplicates).

Nightly mesh report

The nightly sweep on 4c70fbadef's code (ring4 and ring6, 4 and 32 broadcasts per relay), on this host under other agents' load:

Cell Result Known New violations
ring4, 4 passed none none
ring4, 32 passed roll-echo-stall: starved in 2 rolls none
ring6, 4 passed roll-echo-stall: starved in 2 rolls none
ring6, 32 failed roll-echo-stall: starved in 5 rolls starved after the cuts and the kill on lite-07: reopen link a-b (186), kill b (269), end b's broadcasts (38), and one path carried through later events

The ring6-32 failures are reopen-starve's, whose [[known]] entry landed after this sweep, so nightly now reports them as known. No died, shown, or noisy violation in any cell, and no relay showed a stopped broadcast.

ring6 at 4 broadcasts, per event:

Event Quiet (s) Fed (s) Announce msgs / bytes Changed Rewritten DEAD sent/recv DRAIN sent/recv Readers hit (bystanders) Longest (s) Missed End, Start, End
start 0.0 1.0 0 / 21592 n/a n/a n/a/n/a n/a/n/a 0 (0) 0.0 0 0
roll a to new 3.2 3.6 94 / 26811 n/a n/a n/a/n/a n/a/n/a 40 (0) 3.4 10 17
roll b to new 4.0 - 554 / 47987 2 47 0/0 0/0 43 (3) 59.3 2 11
roll c to new 11.3 - 1042 / 54526 6 69 0/0 0/0 46 (6) 59.8 1829 6
roll d to new 2.3 2.6 1979 / 81849 10 80 0/0 0/0 40 (0) 73.1 94 16
roll e to new 3.4 0.0 3051 / 106273 28 197 0/0 0/0 32 (0) 3.8 8 4
roll f to new 3.7 4.2 3045 / 76912 43 122 0/0 0/0 40 (0) 4.5 0 3
reopen link a-b 3.2 0.0 279 / 10599 69 230 6/6 0/0 40 (8) 10.4 4038 0
reopen every link of b (1/2) 3.0 1.0 572 / 21101 121 350 8/8 0/0 48 (8) 15.8 6975 0
reopen every link of b (2/2) 3.2 1.9 635 / 23031 127 360 12/10 0/0 48 (8) 16.6 7182 0
kill and restart b 2.8 2.7 583 / 21644 104 320 12/12 0/0 48 (8) 17.9 7382 0
end b's broadcasts 0.8 0.0 112 / 900 0 0 0/0 0/0 20 (0) 3.6 13 0
restart b's broadcasts 0.5 0.0 93 / 3363 0 0 0/0 0/0 20 (0) 4.8 0 0
roll a to old 2.6 4.2 990 / 42874 56 143 10/10 5/5 40 (0) 14.7 0 0
roll b to old 2.7 3.6 1576 / 63613 20 284 6/6 2/2 42 (2) 13.4 109 0
roll c to old 2.2 2.6 659 / 28373 3 52 2/2 0/0 40 (0) 12.3 0 0
roll d to old 1.8 4.6 324 / 16876 2 26 1/1 0/0 40 (0) 14.8 0 0
roll e to old 1.7 4.1 181 / 17395 1 0 0/0 0/0 40 (0) 13.9 6 4
roll f to old 0.0 0.0 17 / 12637 n/a n/a n/a/n/a n/a/n/a 40 (0) 11.6 0 0
end every broadcast 0.0 0.0 0 / 1413 n/a n/a n/a/n/a n/a/n/a 120 (0) 3.2 0 0

Links of each rolled relay, as its neighbour saw the drain (* dialed by the rolled relay), the same in every cell of a mesh: a rolled relay sends GOAWAY only on the sessions it accepted (drain-dialed-links).

Roll GOAWAY heard (DRAIN) Closed only (DEAD)
roll a to new f-a a-b*
roll b to new a-b, e-b b-c*
roll c to new b-c c-d*
roll d to new c-d d-e*
roll e to new d-e e-f*, e-b*
roll f to new e-f f-a*
roll a to old f-a a-b*
roll b to old a-b, e-b b-c*
roll c to old b-c c-d*
roll d to old c-d d-e*
roll e to old d-e e-f*, e-b*
roll f to old e-f f-a*

What the counts say:

  • Drain: DRAIN moves only on the backward roll, where a new relay drains to new neighbours (5 sent in ring6 for a, 2 for b); the forward roll's drains cross lite-06 links and show as GOAWAY in the links table.
  • End, Start, End: only in rolls, where lite-06 is on the path (3 to 17 paths per roll at 4 broadcasts, up to 199 at 32); never in a lite-07 cut, kill, or source end. That matches fix(moq-net): lite-07 routes skip the lite-06 churn mitigations, and a let-go claim leaves every copy at once #9112's [End, Start, End] residual on lite-06.
  • Cuts and kills cost the readers behind the link 10 to 19s: a blackholed link is noticed only at the 10s idle timeout. In ring6 at 4, a cut of a-b also paused 8 bystander paths, whose relay and source were both elsewhere.
  • Rolling back to the release (to old) fed every reader within 4.7s at 4 broadcasts and 12s at 32.

ring6 at 32 broadcasts, per event:

Event Quiet (s) Fed (s) Announce msgs / bytes Changed Rewritten DEAD sent/recv DRAIN sent/recv Readers hit (bystanders) Longest (s) Missed End, Start, End
start 0.0 1.5 0 / 165157 n/a n/a n/a/n/a n/a/n/a 0 (0) 0.0 0 0
roll a to new 1.9 2.6 654 / 227830 n/a n/a n/a/n/a n/a/n/a 320 (0) 11.9 173 160
roll b to new 11.1 - 3583 / 334173 3 0 0/0 0/0 614 (294) 60.5 18908 133
roll c to new 11.0 - 6783 / 387374 5 0 0/0 0/0 713 (393) 130.5 48062 32
roll d to new 5.0 - 10724 / 468295 10 0 0/0 0/0 865 (545) 149.4 112938 199
roll e to new 13.8 - 32007 / 1086182 40 990 0/0 0/0 960 (640) 199.3 277835 54
roll f to new 5.8 - 61576 / 1721274 54 1206 0/0 0/0 949 (629) 205.3 221316 0
reopen link a-b 3.5 - 1349 / 55771 76 1419 6/6 0/0 959 (384) 136.3 172985 0
reopen every link of b (1/2) 32.7 - 2917 / 118053 138 1914 9/9 0/0 960 (640) 212.3 417159 0
reopen every link of b (2/2) 4.9 - 3188 / 127197 144 1914 10/10 0/0 954 (634) 288.3 168680 0
kill and restart b 16.7 - 2929 / 114484 105 1753 8/8 0/0 960 (640) 364.3 359980 0
end b's broadcasts 2.4 - 896 / 7456 0 0 0/0 0/0 960 (640) 425.3 339073 0
restart b's broadcasts 0.7 - 737 / 26971 0 0 0/0 0/0 959 (639) 486.3 272301 0
roll a to old 3.6 12.0 6844 / 315357 54 701 9/9 6/6 736 (416) 501.8 23962 0
roll b to old 2.0 3.3 10901 / 457534 31 2074 6/6 2/2 506 (186) 13.2 3108 0
roll c to old 2.1 8.2 5148 / 228232 6 166 2/2 0/0 864 (544) 18.6 12671 0
roll d to old 2.6 5.8 6181 / 265236 4 0 1/1 0/0 534 (214) 15.8 3452 32
roll e to old 3.2 4.3 1255 / 130246 2 0 0/0 0/0 455 (135) 14.5 917 32
roll f to old 0.0 3.6 73 / 95826 n/a n/a n/a/n/a n/a/n/a 320 (0) 13.7 108 0
end every broadcast 0.0 0.0 0 / 11257 n/a n/a n/a/n/a n/a/n/a 960 (0) 3.5 0 0

The full reports (report.md, report.json) for every cell are in the run directory, and nightly adds them to the job summary.

On 94e3edb7bb, telling reopen-starve apart from the host's load (about 36 on 32 cores from other agents):

  • No shaper dropped or overflowed a packet in any cell, and the follower's once-a-second status lines kept time while its paths starved.
  • ring4 at 32 with the faults only, no roll (start = "new"): reader c starved on all 32 of a's broadcasts for 75s after a's links reopened. The same faults with every relay on the release (start = "old"), under the same load: every event fed again within 4.1s, no violation.
  • A second full ring4-32 run starved readers in both reopens and the kill again.

Validation

  • python3 test/mesh-replay/replay.test.py (in just test harness): 16 pass. cargo test -p moq-relay --lib: 325 pass, and the moq-relay and moq-bench lints pass.

  • just check through quest gate: rustfmt and remark found formatting, fixed; sh/rs/select.sh lint for moq-bench and sh/markdown.sh check then pass; everything else passed on the first run.

  • reopen-starve's pieces are on the branch since f130d0a43f (the quest, reopen.toml, the catch-all known entry, the wait-list line): replay.test.py 16 pass, quest check 518 documents ok, and reopen.toml on ring4 at 32 broadcasts starved 52 paths after reopening every link of a, with the kill taking 57.5s to feed every reader.

  • The nightly sweep above, and the reproductions named under Flaws found.

  • Local review after the merge of the parallel session's commits (Codex, 2 sessions this iterate, 6 rounds, 6 findings, all fixed): a pause still going counts only to the reader's last report; upstream = true only in the new build's config; a link counts as lite-07 by the dialer's first version the acceptor lists (rs/moq-tokio/src/quic/backend.rs:905); the start needs every reader routed unless an applied mark may hide one; a roll repeats its times. replay.test.py: 19 pass; just check through quest gate passes; a real ring4 replay (start = "new", one link reopen) passes with the start routed and fed in 1.0s.

  • just check once failed js/net's a stale drop is recorded under the narrowest budget asked on lite-07 (subscriber.test.ts:2998), which this branch does not touch; it fails 3 of 3 in isolation on this checkout's unchanged js/, and moq-dev/moq#9164 fixes it.

  • Local review: Codex (quest review --harness codex), 13 rounds, 9 findings: 8 fixed, 1 refuted with evidence. The fixes: the run ignored a dead child or a path never read, double-counted a scrape across a restart, let a known entry excuse the start, and measured pauses past the latest status (twice), a stopped publisher dropped its tracks unfinished, and a link's flows each had their own shaper budget. Two checkpoint rounds came on narrowing report-arithmetic findings with the approach unchallenged, so I kept the approach.

  • CodeRabbit's 17 threads (this iterate): replay.test.py 20 pass (one new: a relay name with - is refused); cargo clippy -p moq-bench --all-targets and just check through quest gate pass. On the built binary, moq-bench replay --client-connect ... now fails naming --connect. release_binary moq-relay moq-relay 0.17.17 still fetches and checksums the asset, and a missing tag's SHA256SUMS answers 404, the only answer that builds instead. A real just test mesh-replay --mesh test/mesh-replay/ring4.toml --broadcasts 4 passes with 2 known (roll-echo-stall, roll b to new and roll c to new), every other event fed again in at most 4.7s after the fault.

Decisions

  • The follower and publisher are a moq-bench subcommand, not a new package; the orchestrator is a stdlib Python script with unit tests (replay.test.py, in just test harness), like the other harnesses' report code.
  • Peers are told apart by a per-peer token (peer.<relay>) the run's auth server grants, since the relay strips a peer URL's query.
  • A link cut is SIGSTOP on its shaper, so it is a real blackhole that outlives the idle timeout, not a shaped loss.
  • Events run in a fixed order, each after the previous settled, not on a wall clock, so counts compare night to night.
  • The harness sets drain_timeout = "10s" on both builds: 0.17.17 drains with a zero window by default and would send no GOAWAY.
  • A live broadcast its reader's relay does not show is reported per event (unrouted), not failed: a deployment's mesh with upstream marks may hide a route legitimately. The start is the exception: it fails on an unrouted reader unless a mark the starting build applies may hide one, so a run with no routes cannot pass having tested nothing.
  • upstream = true goes only into the new build's relay config: 0.17.17 denies unknown peer fields, and a deployment rolling off a release without the mark configures its old relays unmarked.
  • starved readers in a roll are [[known]] against roll-echo-stall in the nightly script, so nightly stays red only for something new; the reproduction script lists no known and still fails.
  • No [[known]] entry excuses the start event: it is the baseline before any fault.
  • A run also fails when a relay, publisher, reader, or shaper exits unasked (died), and checks every path each reader was told to follow, so a dead or silent reader cannot pass.
  • The follower reads on past a group it cannot finish, as a viewer does, and drops an unfinished group once a newer one arrives; only the track ending ends its read.
  • ✅ The lite-07 cut waits on roll-echo-stall (rejected: cutting without it), decided in this comment: every roll passes through the mixed state once per relay. It stays in the epic README's wait list, a follow-up of this PR.
  • ✅ The lite-07 cut also waits on reopen-starve (rejected: cutting without it, or deferring it to after the cut), taken by this session under the delegation for lite-07 routing calls: lite-06 settles the same faults in 4s, so it is a lite-07 regression on a link flap or crash, which a deployment sees daily. Listed in the epic README's wait list.
  • ✅ Nightly lists reopen-starve's starved readers as [[known]] for every event, after roll-echo-stall's roll entry so a roll's starved reader stays with that quest (rejected: keeping the 32-broadcast cells red every night until the fix, or one entry per event kind, which a starvation carried into a later event escapes). The same as detour-resume's entry; reopen.toml lists no known and still fails.
  • Nightly sweeps two rings rather than one larger mesh, so a run fits one runner (6 relays at 32 broadcasts is 192 broadcasts and 960 followed paths).
  • Relay names may no longer hold -, so a link's script-visible name dial-accept is unique (rejected: tuple keys plus an ambiguity check, more code for names no mesh uses).
  • Quiet stays the time of the last route or announce movement, confirmed by the quiet window after it, not the window's end (rejected: adding the window, which reports the configured constant instead of the mesh). The README and report note now say so.
  • Fed needs an arrival after the fault ended, so a short fault cannot pass on a group read before it.
  • The reopen-starve [[known]] entry stays unscoped, per the recorded decision above.

Follow-ups

  • roll-echo-stall, new. The cut waits on it (decided). No open decision.
  • reopen-starve, new. The cut waits on it (decided under the delegation). No open decision.

Models

Size: L
Implementer: Claude, claude-opus-5-5, unknown effort
Implementer: Claude Code, claude-opus-5-5, unknown effort (iterate, M)
Implementer: Claude Code, claude-opus-5-5, unknown effort (iterate, S)

(Written by Claude Opus 5.5)

🤖 Generated with Claude Code

Review

Rewritten by quest review after each round.

  • Head: 0823532

  • Base: origin/main at 334427b

  • Risk: high, it changes .github/workflows/nightly.yml, which is always high: workflows run with the repository's secrets and decide what merges and ships

  • Classified: high by the always-high paths

  • Reviewer: Codex, gpt-6.1-sol, xhigh effort

  • Session: 01a1f32d-1270-7db3-b36e-c04db5b20e3e, round 5

  • Score: 10/10, The merge preserves the reviewed implementation and updates the drain documentation correctly, with no new issues found in the branch’s changes.

  • Rounds: 8; first round 3 (medium 3); findings 6 (medium 6): fixed 6, refuted 0, open 0

  • Findings: none

kixelated and others added 6 commits November 18, 2026 20:17
… a roll

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… and reproduce a mid-roll stall

The roll table reads GOAWAY from both builds' logs, a roll can stop partway
(`relays`), and each event counts readers whose broadcast stepped through
End, Start, End. Nightly runs a ring of four and a ring of six.

chain3.toml with half-roll.toml reproduces a new relay accepting an old
neighbour's subscribe and forwarding nothing, now quest/m2/multicast/roll-echo-stall.md.
The quest file for this work is done and removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A group given up as old, reset mid-stream, or still open when a newer one
arrives no longer ends the follower's whole read, which looked like a reader
stuck on an ended track.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd count no scrape twice

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Maintainer decision (taken by the coordinating session, to which the maintainer delegated these calls while away):

✅ The lite-07 cut waits on roll-echo-stall (rejected: cutting without it).

Every roll passes through the mixed lite-06/lite-07 state once per relay, and a starved viewer sees a live broadcast that never plays. Letting a deployment roll lite-07 out safely is the reason for the cut, so the stall has to be fixed first. Keep it in quest/m2/multicast/README.md's wait list, and replace the ## Waiting item with this decision.

(Written by Claude Opus 5.5 with xhigh)

kixelated and others added 6 commits November 18, 2026 23:31
…nd restate the flaws from clean runs

The chain stopped reproducing once the follower read past unfinished groups;
the ring of six still starves. The stall's quest also covers a reader starving
after a lite-07 cut that follows a roll.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…est status

Measured to the end of the event, a status line up to a second old counted
nearly every reader as hit against the 0.5s threshold.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ll counts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A reader starving on lite-07 after a cut is reopen-starve's, which another
session is adding with its own catch-all entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

For the session holding this PR (quest:active, the reopen-starve work in the body): this branch moved on while you worked, and your commit isn't on any remote branch. The head is now 4c70fbadef. What changed since 94e3edb7bb, so you can merge rather than overwrite:

  • moq-bench replay's follower read on past a group it can't finish only from e1291c4d2d; before that, the stuck ... "old" readers were the follower's bug. Your runs on 94e3edb7bb have that fix.
  • replay.py: a pause still going is now measured to the reader's latest status line (faddfe9f60, 1d953bc023). Before that, nearly every reader counted as "hit" in every event, bystanders included, so Readers hit in older reports overstates. died and expected-path checks came in 94e3edb7bb; start is never excused by [[known]] (d37238dffd).
  • script.toml: roll-echo-stall's starved entry is back to event = "roll" (4c70fbadef), the same as on 94e3edb7bb, so your catch-all after it should merge cleanly. I'd briefly widened it to every event; that's reverted.
  • roll-echo-stall.md: its lite-07 "residue after a roll" paragraph is gone. Your reopen-starve evidence (starving on ring4 at 32 with no roll) is stronger than my ring6 at 4 test (10 cuts, none starved), so that case is yours. The coordinator's decision is recorded in its Plan.
  • chain3.toml is deleted: it stopped reproducing once the follower was fixed. roll-echo-stall now reproduces with --mesh ring6.toml --script half-roll.toml --broadcasts 32. Your body still names chain3 in its roll-echo-stall bullet.
  • detour-resume.md: the mesh-replay note now cites only the clean ring6-32 run (unroutable / internal error).

A full nightly sweep on this code is running now. I'll add its per-event tables under the body's Nightly mesh report and leave your sections as they are.

(Written by Claude Opus 5.5)

@coderabbitai

coderabbitai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

Walkthrough

The pull request adds a real-relay mesh replay harness with configurable mesh topologies and fault scripts, plus a benchmark command that publishes and follows broadcasts. The harness measures relay activity and reader interruptions, runs tests, and writes reports. Test recipes and nightly workflow entries invoke replay scenarios. Multicast quest documents record replay observations and update references and cut criteria.

Merge Risk: Moderate · up to e2bde

This change adds a real-relay mesh replay test and does not change relay behavior. Several harness checks can report a scenario as healthy when readers are unrouted, have not resumed, or have exited, and one known-issue exception is broader than intended. Because the lite-07 release decision relies on these results, fix the false-pass paths before treating replay output as a release gate.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title "Real relays replay a mesh and its faults" matches the main change: a new mesh-replay harness that runs real moq-relay processes through scripted mesh faults and rolls. It is specific and de…
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description is specific and related to the changeset. It describes the mesh-replay harness (just test mesh-replay), the moq-bench replay subcommand, the nightly ring4 and ring6 jobs, the share…
Full details: Title check

Explanation

The title "Real relays replay a mesh and its faults" matches the main change: a new mesh-replay harness that runs real moq-relay processes through scripted mesh faults and rolls. It is specific and descriptive, and the changeset (test/mesh-replay, moq-bench replay, nightly workflow) supports it.

Full details: Description check

Explanation

The description is specific and related to the changeset. It describes the mesh-replay harness (just test mesh-replay), the moq-bench replay subcommand, the nightly ring4 and ring6 jobs, the shared release downloader in test/lib/release.sh, the repository_configs_load exclusion, and the flaws found. Level of detail is high, and every part matches the raw summary.

Finishing Touches

Simplify code

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Important

The reopen-starve work the PR body describes is not on the branch, so the nightly 32-broadcast cells will fail every night on a violation no quest owns.

Approach: sound. Real relays over moq-shaper bufferbloat links, a scripted fault sequence, and reports with violations as the gate and counts never gating fit what the lite-07 cut needs to see mid-roll. Keeping the follower in moq-bench and the orchestrator in stdlib Python with unit tests also fits.

Reviewed changes: all 12 commits at 4c70fba. That covers the harness, the moq-bench replay endpoint, the shared release fetcher, the nightly entries, and the quest edits.

  • moq-bench replay: a new subcommand that publishes one single-frame group per interval and follows paths across instances through Following::available(). It writes a JSON line per gap, per online/offline/ended change, and per status tick.
  • test/mesh-replay/: the run.sh sweep over mesh × broadcasts and the replay.py orchestrator, which runs shapers, relays, the auth server, roll/reopen/kill/end-source events, /metrics scraping, and the unfed/shown/died/noisy checks with [[known]] excuses. Also adds unit tests and the ring4/ring6/script/half-roll inputs.
  • test/lib/release.sh: extracted from compat.sh. It now fetches the asset and SHA256SUMS with curl, picks aarch64 assets too, and builds the crate when the release has no checksummed asset for the host.
  • Nightly: two matrix entries, ring4 and ring6, each at 4 and 32 broadcasts, with reports added to the job summary.
  • Quests: closes mesh-replay.md, adds roll-echo-stall.md to the epic's wait list, and repoints references at test/mesh-replay/.

⚠️ The reopen-starve commit is missing from the branch

The PR body describes four things as part of this PR:

  • a new reopen-starve quest
  • a new test/mesh-replay/reopen.toml
  • a nightly [[known]] entry for reopen-starve's starved readers
  • an entry in the epic README's wait list

The Validation section also says "After the reopen-starve commit: replay.test.py 13 pass". None of this is at 4c70fba: git grep reopen-starve and git grep reopen.toml return nothing. The branch's latest commit is the roll-echo-stall known-entry change, and the test file has 15 tests.

What happens if this merges as is:

  • Nightly fails every night. The PR's own nightly table shows ring6 at 32 broadcasts failing with new starved violations after the reopens, the kill, and the source end. The body also reports that ring4 at 32 starved readers on repeat runs. script.toml excuses starved only for event = "roll", so both Mesh replay jobs are likely red every night with nothing that names an owner.
  • The cut doesn't wait on it. The decision recorded in the body ("the lite-07 cut also waits on reopen-starve") is not in quest/m2/multicast/README.md. Nothing in the quest tree stops the cut on a regression that, per the body, lite-06 settles in 4s and lite-07 does not settle in 75s.
  • Dead links. The body's links to reopen-starve.md and reopen.toml point at files that don't exist.
Technical details
# The reopen-starve commit was never pushed

## Affected sites
- `quest/m2/multicast/reopen-starve.md`: missing; the body describes its goal, the log trace, and the likely cause in `model/front.rs`'s stranded tracks
- `test/mesh-replay/reopen.toml`: missing; the reproduction runs it on ring4 with `--broadcasts 32` and lists no known
- `test/mesh-replay/script.toml`: no `[[known]]` with `check = "unfed"`, `kind = "starved"`, `quest = "/quest/m2/multicast/reopen-starve.md"`. Per the decision, it goes after roll-echo-stall's roll entry so `Run.violation`'s first match keeps roll starvation with roll-echo-stall
- `quest/m2/multicast/README.md` (wait list near line 41): no reopen-starve link
- `test/mesh-replay/replay.test.py`: the body's "13 pass" suggests that commit also changed the tests

## Required outcome
- The branch matches the PR body: the quest, the reproduction script, the known entry, and the wait-list entry are present, or the body drops the claims and the nightly 32-broadcast cells are expected to fail.

## Suggested approach
- Recover the local commit (it postdates `94e3edb7bb`) and push it, then rerun `python3 test/mesh-replay/replay.test.py` and `quest check`.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using claude-opus-5-5 | 𝕏

Comment thread rs/moq-bench/src/replay.rs Resolved
Comment thread test/mesh-replay/replay.py Resolved
…he follower, and a roll waits a bounded time

reopen-starve, its reopen.toml reproduction, its catch-all known entry, and
its place in the cut's wait list match what the PR body already decided. A
refusal of one followed path now ends only that path's read, and a drained
relay that never exits is killed and reported as hung.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Re the review's main finding (the reopen-starve work missing from the branch): fixed in f130d0a, which adds it to match the body.

  • quest/m2/multicast/reopen-starve.md: the quest, with the evidence and the likely cause. A track refused as unroutable is stranded on its source, and unstrand asks again only when the serving source changes, so a route returning behind the same source is never asked again.
  • test/mesh-replay/reopen.toml: the script's faults on the new build alone, no roll, and nothing known.
  • script.toml: a starved catch-all naming reopen-starve, after roll-echo-stall's roll entry.
  • quest/m2/multicast/README.md: reopen-starve is on the cut's wait list.
  • replay.test.py: a test that reopen.toml loads with nothing known (16 tests).

quest check (518 documents) and replay.test.py pass. A run of reopen.toml on ring4 at 32 broadcasts is going, to confirm the reproduction.

For the session that wrote that commit locally: this is my reconstruction from your body text. If yours differs, prefer yours and merge origin/quest/m2/multicast/mesh-replay rather than force-pushing.

(Written by Claude Opus 5.5)

@pullfrog

pullfrog Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Important

Addendum to the review above: this PR breaks a moq-relay unit test.

rs/moq-relay/src/config.rs's repository_configs_load loads every tracked *.toml under test/ as a relay config. That config uses deny_unknown_fields, and the only files it skips are Cargo.toml, rust-toolchain.toml, bunfig.toml, and cases.toml. The four new files in test/mesh-replay/ are mesh and script files, not relay configs, so the test fails. I ran it at 4c70fba:

$ cargo test -p moq-relay --lib repository_configs_load
test config::tests::repository_configs_load ... FAILED
panicked at rs/moq-relay/src/config.rs:1134:51:
.../test/mesh-replay/half-roll.toml: unknown field `events`, expected one of `limits`, `listen`, `server`, ...

just check runs tests only for the paths a branch changed. This PR doesn't touch moq-relay, so the local gate never ran this test, but any later change to moq-relay will fail on it.

Two ways to fix it:

  • Skip test/mesh-replay/ in that test. It already skips cases.toml for the same reason.
  • Rename the mesh and script inputs so they aren't .toml files the test picks up.

Pullfrog  | View workflow run | via Pullfrog | Using claude-opus-5-5 | 𝕏

pullfrog[bot]
pullfrog Bot previously approved these changes Nov 19, 2026

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found. The reopen-starve work the last review found missing is now on the branch, and both inline threads were fixed.

Reviewed changes: what changed since 4c70fba, all in f130d0a.

  • reopen-starve quest: added quest/m2/multicast/reopen-starve.md, which covers the reproduction, the relay-log trace, and the likely cause in Track::stranded/unstrand. It is linked from the epic README's wait list, so the lite-07 cut now waits on it.
  • reopen.toml: added the reproduction script. It runs the nightly faults on the new build with no roll and lists nothing known. A new unit test checks that it loads.
  • Nightly known entry: added a starved entry for reopen-starve to script.toml after roll-echo-stall's roll-only entry. Because the first match wins in Run.violation, a roll's starved reader stays with roll-echo-stall.
  • moq-bench replay refusals: both waits now go through next, which writes an ended line for a refusal and asks again. A refusal now ends only that path's read, not the whole reader. Following::poll_available waits for a route change after a refusal, so this can't spin. Closed can't spin either: run holds the consume origin until after tasks.shutdown() aborts the followers.
  • Bounded roll wait: a rolled relay now gets DRAIN (10s, which is also the value written to drain_timeout) plus a 20s margin to exit. After that the run sends SIGKILL, records a hung violation, and carries on, so the cell still writes its report. hung is documented in the README.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

…logies and scripts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated

Copy link
Copy Markdown
Collaborator Author

Re the addendum (repository_configs_load loading the mesh and script files as relay configs): fixed in 1a7c26d. The test now skips test/mesh-replay/, whose files are topologies and event scripts. The replay writes each relay's config at run time. cargo test -p moq-relay --lib: 325 pass, including repository_configs_load, and the moq-relay lint passes.

(Written by Claude Opus 5.5)

pullfrog[bot]
pullfrog Bot previously approved these changes Nov 19, 2026

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found. The repository_configs_load failure on the mesh replay's TOMLs is fixed.

Reviewed changes: the one commit since f130d0a.

  • repository_configs_load skips test/mesh-replay/: the relay config test no longer tries to parse the mesh topologies and event scripts as relay configs. That fixes the unknown field failure on half-roll.toml and the other three TOMLs. Other relay fixtures under test/ (drain/relay.toml, portrait/relay.toml, and others) still load, so the checked > 0 assertion for that area still holds.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

kixelated and others added 3 commits November 19, 2026 01:30
…ite-07 by the dialer's first version the acceptor lists

0.17.17 rejects an unknown peer field, so an upstream-marked mesh could not start
the old build. A link negotiates the first version in the dialer's offer that the
acceptor lists, so equal allowlists are neither needed nor enough.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eats its times

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…start from routing every reader

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pullfrog[bot]
pullfrog Bot previously approved these changes Nov 19, 2026

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes: the three commits since e2bde3e (cc71d6f, ff0cd87, f98af03). replay.test.py passes with 20 tests.

  • The upstream mark is on the new build only: Relay.config now writes upstream = true only when the dialer runs the new build. The release rejects that field, so this matches how a deployment rolling off the release would configure its old relays.
  • lite-07 is decided by negotiation: Run.lite07 now takes the first version in the dialer's offer that the acceptor also lists. Before, it needed the two version lists to be equal. A build with an empty (default) list counts as not lite-07.
  • A roll repeats its times: rolls now go through the same loop as the other events, with a (i/n) suffix on the name.
  • The start needs every reader routed: Run.baseline holds fed_at back in settle while a reader is unrouted at start, then records an unrouted violation. The check is skipped when a link the new build dials is marked upstream, since that mark may hide a route. This uses the same dialer-build test as Relay.config, so the two can't drift. No [[known]] entry excuses it, because known entries already skip start. The README documents the new violation.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

…lay endpoint

- Fed needs a group sent after the fault, and quiet is judged after reading the readers.
- A pause is classified before it is rounded.
- A reader or relay that dies after the last event fails the run.
- A torn shaper report is recorded, not raised.
- Relay names drop `-`, so link names cannot collide; sweep cells of one name are refused.
- `release_binary` builds the crate only on a 404 SHA256SUMS and retries transient errors.
- `moq-bench replay` refuses deprecated flags and aborts its tracks with the failure cause.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pullfrog[bot]
pullfrog Bot previously approved these changes Nov 19, 2026

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes: the one commit since f98af03, 81c437d, which answers CodeRabbit's threads. replay.test.py passes with 20 tests.

  • Deprecated flags refused: moq-bench replay now collects deprecated client and quic options and refuses to start if it finds any. It does this the same way the main moq-bench config does at config.rs:202.
  • A failed publisher aborts its tracks: on an error path, each held track gets abort with the run's error instead of being dropped unfinished. A clean stop still calls finish(). Both arms return Result<()>, and an already-ended track's error is still ignored.
  • Fed needs an arrival after the fault: hungry now counts a path as unfed while its last arrival is before restored. A short fault can no longer pass on a group read before it. settle now reads the route counters after reading the readers, so counters that moved during that read still delay quiet.
  • Pause threshold compared before rounding: per-path rows keep the unrounded duration, so the 0.5s cut is exact. Rounding happens only in max_s, p50_s, and worst.
  • Harness hardening:
    • The run checks for died processes once more after the final quiet window.
    • A torn shaper stats file is reported in that link's stats instead of crashing the report.
    • Relay names can no longer contain -, so a link's dial-accept name is unique.
    • run.sh refuses two --mesh files with the same basename.
    • The tests write their fixtures to a scratch directory that is removed when they exit.
  • release.sh tells a missing manifest from a failed fetch: it fetches SHA256SUMS with --retry 3. A 404 falls back to building, and any other non-200 answer fails, so a transient error can't quietly skip the published asset.
  • Docs: the README and report note now define Quiet and Fed the way the code measures them, and say how to keep a passing run's reports. The test index row and the route-wakes quest describe what the harness measures without overclaiming.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

@kixelated
kixelated added this pull request to the merge queue Nov 19, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Nov 19, 2026
@moq-bot

moq-bot Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

quest land watch: the merge queue removed this pull request at 81c437d because it conflicts with main (merge_conflict).

The land fixer works on it in the background, and the watcher re-enqueues it at the back of the queue once it is fixed or judged flaky, or hands it back here with what it needs.

(written by quest land watch)

Resolve against #9158 (drain-dialed-links landed) and close mesh-replay:
both quest files stay deleted, the epic's wait list keeps the two flaws
the replay found, and the replay README's starred-link note now names the
release that drains dialed links.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found. The merge of origin/main changes only docs, and its conflict resolution is consistent.

Reviewed changes: the one commit since 81c437d, 0823532. It merges origin/main, which landed drain-dialed-links in #9158.

  • Epic wait list: quest/m2/multicast/README.md no longer lists drain-dialed-links. The lite-07 cut still waits on both flaws the mesh replay found, roll-echo-stall and reopen-starve.
  • Starred-link note: test/mesh-replay/README.md now says that a starred link reads DEAD because the release predates #9158. Before, it pointed at the deleted quest. No file on the branch still links to drain-dialed-links.md.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

@kixelated
kixelated added this pull request to the merge queue Nov 19, 2026
@moq-bot

moq-bot Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

quest land watch: the land fixer's result for 81c437d was not applied: its head moved to 0823532 while the fixer ran, and a push by anyone but the land fixer ends the pick.

(written by quest land watch)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant