Repository navigation
Real relays replay a mesh and its faults - #9153
Conversation
Quest-Claim: 321afd2f932243a8
… a roll Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… and reproduce a mid-roll stall The roll table reads GOAWAY from both builds' logs, a roll can stop partway (`relays`), and each event counts readers whose broadcast stepped through End, Start, End. Nightly runs a ring of four and a ring of six. chain3.toml with half-roll.toml reproduces a new relay accepting an old neighbour's subscribe and forwarding nothing, now quest/m2/multicast/roll-echo-stall.md. The quest file for this work is done and removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A group given up as old, reset mid-stream, or still open when a newer one arrives no longer ends the follower's whole read, which looked like a reader stuck on an ended track. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nd count no scrape twice Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Maintainer decision (taken by the coordinating session, to which the maintainer delegated these calls while away): ✅ The lite-07 cut waits on roll-echo-stall (rejected: cutting without it). Every roll passes through the mixed lite-06/lite-07 state once per relay, and a starved viewer sees a live broadcast that never plays. Letting a deployment roll lite-07 out safely is the reason for the cut, so the stall has to be fixed first. Keep it in (Written by Claude Opus 5.5 with xhigh) |
…nd restate the flaws from clean runs The chain stopped reproducing once the follower read past unfinished groups; the ring of six still starves. The stall's quest also covers a reader starving after a lite-07 cut that follows a roll. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…est status Measured to the end of the event, a status line up to a second old counted nearly every reader as hit against the 0.5s threshold. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ll counts Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A reader starving on lite-07 after a cut is reopen-starve's, which another session is adding with its own catch-all entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
For the session holding this PR (
A full nightly sweep on this code is running now. I'll add its per-event tables under the body's Nightly mesh report and leave your sections as they are. (Written by Claude Opus 5.5) |
WalkthroughThe pull request adds a real-relay mesh replay harness with configurable mesh topologies and fault scripts, plus a benchmark command that publishes and follows broadcasts. The harness measures relay activity and reader interruptions, runs tests, and writes reports. Test recipes and nightly workflow entries invoke replay scenarios. Multicast quest documents record replay observations and update references and cut criteria. Merge Risk:
|
There was a problem hiding this comment.
Important
The reopen-starve work the PR body describes is not on the branch, so the nightly 32-broadcast cells will fail every night on a violation no quest owns.
Approach: sound. Real relays over moq-shaper bufferbloat links, a scripted fault sequence, and reports with violations as the gate and counts never gating fit what the lite-07 cut needs to see mid-roll. Keeping the follower in moq-bench and the orchestrator in stdlib Python with unit tests also fits.
Reviewed changes: all 12 commits at 4c70fba. That covers the harness, the moq-bench replay endpoint, the shared release fetcher, the nightly entries, and the quest edits.
moq-bench replay: a new subcommand that publishes one single-frame group per interval and follows paths across instances throughFollowing::available(). It writes a JSON line per gap, per online/offline/ended change, and per status tick.test/mesh-replay/: therun.shsweep over mesh × broadcasts and thereplay.pyorchestrator, which runs shapers, relays, the auth server, roll/reopen/kill/end-source events,/metricsscraping, and the unfed/shown/died/noisy checks with[[known]]excuses. Also adds unit tests and the ring4/ring6/script/half-roll inputs.test/lib/release.sh: extracted fromcompat.sh. It now fetches the asset andSHA256SUMSwithcurl, picks aarch64 assets too, and builds the crate when the release has no checksummed asset for the host.- Nightly: two matrix entries, ring4 and ring6, each at 4 and 32 broadcasts, with reports added to the job summary.
- Quests: closes
mesh-replay.md, addsroll-echo-stall.mdto the epic's wait list, and repoints references attest/mesh-replay/.
⚠️ The reopen-starve commit is missing from the branch
The PR body describes four things as part of this PR:
- a new reopen-starve quest
- a new
test/mesh-replay/reopen.toml - a nightly
[[known]]entry for reopen-starve's starved readers - an entry in the epic README's wait list
The Validation section also says "After the reopen-starve commit: replay.test.py 13 pass". None of this is at 4c70fba: git grep reopen-starve and git grep reopen.toml return nothing. The branch's latest commit is the roll-echo-stall known-entry change, and the test file has 15 tests.
What happens if this merges as is:
- Nightly fails every night. The PR's own nightly table shows ring6 at 32 broadcasts failing with new
starvedviolations after the reopens, the kill, and the source end. The body also reports that ring4 at 32 starved readers on repeat runs.script.tomlexcusesstarvedonly forevent = "roll", so both Mesh replay jobs are likely red every night with nothing that names an owner. - The cut doesn't wait on it. The decision recorded in the body ("the lite-07 cut also waits on reopen-starve") is not in
quest/m2/multicast/README.md. Nothing in the quest tree stops the cut on a regression that, per the body, lite-06 settles in 4s and lite-07 does not settle in 75s. - Dead links. The body's links to
reopen-starve.mdandreopen.tomlpoint at files that don't exist.
Technical details
# The reopen-starve commit was never pushed
## Affected sites
- `quest/m2/multicast/reopen-starve.md`: missing; the body describes its goal, the log trace, and the likely cause in `model/front.rs`'s stranded tracks
- `test/mesh-replay/reopen.toml`: missing; the reproduction runs it on ring4 with `--broadcasts 32` and lists no known
- `test/mesh-replay/script.toml`: no `[[known]]` with `check = "unfed"`, `kind = "starved"`, `quest = "/quest/m2/multicast/reopen-starve.md"`. Per the decision, it goes after roll-echo-stall's roll entry so `Run.violation`'s first match keeps roll starvation with roll-echo-stall
- `quest/m2/multicast/README.md` (wait list near line 41): no reopen-starve link
- `test/mesh-replay/replay.test.py`: the body's "13 pass" suggests that commit also changed the tests
## Required outcome
- The branch matches the PR body: the quest, the reproduction script, the known entry, and the wait-list entry are present, or the body drops the claims and the nightly 32-broadcast cells are expected to fail.
## Suggested approach
- Recover the local commit (it postdates `94e3edb7bb`) and push it, then rerun `python3 test/mesh-replay/replay.test.py` and `quest check`.claude-opus-5-5 | 𝕏
…he follower, and a roll waits a bounded time reopen-starve, its reopen.toml reproduction, its catch-all known entry, and its place in the cut's wait list match what the PR body already decided. A refusal of one followed path now ends only that path's read, and a drained relay that never exits is killed and reported as hung. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Re the review's main finding (the reopen-starve work missing from the branch): fixed in f130d0a, which adds it to match the body.
For the session that wrote that commit locally: this is my reconstruction from your body text. If yours differs, prefer yours and merge (Written by Claude Opus 5.5) |
|
Important Addendum to the review above: this PR breaks a
Two ways to fix it:
|
There was a problem hiding this comment.
✅ No new issues found. The reopen-starve work the last review found missing is now on the branch, and both inline threads were fixed.
Reviewed changes: what changed since 4c70fba, all in f130d0a.
- reopen-starve quest: added
quest/m2/multicast/reopen-starve.md, which covers the reproduction, the relay-log trace, and the likely cause inTrack::stranded/unstrand. It is linked from the epic README's wait list, so the lite-07 cut now waits on it. reopen.toml: added the reproduction script. It runs the nightly faults on the new build with no roll and lists nothing known. A new unit test checks that it loads.- Nightly known entry: added a
starvedentry for reopen-starve toscript.tomlafter roll-echo-stall's roll-only entry. Because the first match wins inRun.violation, a roll's starved reader stays with roll-echo-stall. moq-bench replayrefusals: both waits now go throughnext, which writes anendedline for a refusal and asks again. A refusal now ends only that path's read, not the whole reader.Following::poll_availablewaits for a route change after a refusal, so this can't spin.Closedcan't spin either:runholds theconsumeorigin until aftertasks.shutdown()aborts the followers.- Bounded roll wait: a rolled relay now gets
DRAIN(10s, which is also the value written todrain_timeout) plus a 20s margin to exit. After that the run sends SIGKILL, records ahungviolation, and carries on, so the cell still writes its report.hungis documented in the README.
claude-opus-5-5 | 𝕏
…logies and scripts Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Re the addendum ( (Written by Claude Opus 5.5) |
There was a problem hiding this comment.
✅ No new issues found. The
repository_configs_loadfailure on the mesh replay's TOMLs is fixed.
Reviewed changes: the one commit since f130d0a.
repository_configs_loadskipstest/mesh-replay/: the relay config test no longer tries to parse the mesh topologies and event scripts as relay configs. That fixes theunknown fieldfailure onhalf-roll.tomland the other three TOMLs. Other relay fixtures undertest/(drain/relay.toml,portrait/relay.toml, and others) still load, so thechecked > 0assertion for that area still holds.
claude-opus-5-5 | 𝕏
…ite-07 by the dialer's first version the acceptor lists 0.17.17 rejects an unknown peer field, so an upstream-marked mesh could not start the old build. A link negotiates the first version in the dialer's offer that the acceptor lists, so equal allowlists are neither needed nor enough. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eats its times Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…start from routing every reader Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes: the three commits since e2bde3e (cc71d6f, ff0cd87, f98af03). replay.test.py passes with 20 tests.
- The upstream mark is on the new build only:
Relay.confignow writesupstream = trueonly when the dialer runs the new build. The release rejects that field, so this matches how a deployment rolling off the release would configure its old relays. - lite-07 is decided by negotiation:
Run.lite07now takes the first version in the dialer's offer that the acceptor also lists. Before, it needed the two version lists to be equal. A build with an empty (default) list counts as not lite-07. - A roll repeats its
times: rolls now go through the same loop as the other events, with a(i/n)suffix on the name. - The start needs every reader routed:
Run.baselineholdsfed_atback insettlewhile a reader is unrouted atstart, then records anunroutedviolation. The check is skipped when a link the new build dials is markedupstream, since that mark may hide a route. This uses the same dialer-build test asRelay.config, so the two can't drift. No[[known]]entry excuses it, because known entries already skipstart. The README documents the new violation.
claude-opus-5-5 | 𝕏
…lay endpoint - Fed needs a group sent after the fault, and quiet is judged after reading the readers. - A pause is classified before it is rounded. - A reader or relay that dies after the last event fails the run. - A torn shaper report is recorded, not raised. - Relay names drop `-`, so link names cannot collide; sweep cells of one name are refused. - `release_binary` builds the crate only on a 404 SHA256SUMS and retries transient errors. - `moq-bench replay` refuses deprecated flags and aborts its tracks with the failure cause. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes: the one commit since f98af03, 81c437d, which answers CodeRabbit's threads. replay.test.py passes with 20 tests.
- Deprecated flags refused:
moq-bench replaynow collects deprecatedclientandquicoptions and refuses to start if it finds any. It does this the same way the mainmoq-benchconfig does atconfig.rs:202. - A failed publisher aborts its tracks: on an error path, each held track gets
abortwith the run's error instead of being dropped unfinished. A clean stop still callsfinish(). Both arms returnResult<()>, and an already-ended track's error is still ignored. - Fed needs an arrival after the fault:
hungrynow counts a path as unfed while its last arrival is beforerestored. A short fault can no longer pass on a group read before it.settlenow reads the route counters after reading the readers, so counters that moved during that read still delay quiet. - Pause threshold compared before rounding: per-path rows keep the unrounded duration, so the 0.5s cut is exact. Rounding happens only in
max_s,p50_s, andworst. - Harness hardening:
- The run checks for
diedprocesses once more after the final quiet window. - A torn shaper stats file is reported in that link's stats instead of crashing the report.
- Relay names can no longer contain
-, so a link'sdial-acceptname is unique. run.shrefuses two--meshfiles with the same basename.- The tests write their fixtures to a scratch directory that is removed when they exit.
- The run checks for
release.shtells a missing manifest from a failed fetch: it fetchesSHA256SUMSwith--retry 3. A 404 falls back to building, and any other non-200 answer fails, so a transient error can't quietly skip the published asset.- Docs: the README and report note now define Quiet and Fed the way the code measures them, and say how to keep a passing run's reports. The test index row and the
route-wakesquest describe what the harness measures without overclaiming.
claude-opus-5-5 | 𝕏
|
quest land watch: the merge queue removed this pull request at 81c437d because it conflicts with The land fixer works on it in the background, and the watcher re-enqueues it at the back of the queue once it is fixed or judged flaky, or hands it back here with what it needs. (written by quest land watch) |
Resolve against #9158 (drain-dialed-links landed) and close mesh-replay: both quest files stay deleted, the epic's wait list keeps the two flaws the replay found, and the replay README's starred-link note now names the release that drains dialed links. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
✅ No new issues found. The merge of
origin/mainchanges only docs, and its conflict resolution is consistent.
Reviewed changes: the one commit since 81c437d, 0823532. It merges origin/main, which landed drain-dialed-links in #9158.
- Epic wait list:
quest/m2/multicast/README.mdno longer lists drain-dialed-links. The lite-07 cut still waits on both flaws the mesh replay found, roll-echo-stall and reopen-starve. - Starred-link note:
test/mesh-replay/README.mdnow says that a starred link reads DEAD because the release predates #9158. Before, it pointed at the deleted quest. No file on the branch still links todrain-dialed-links.md.
claude-opus-5-5 | 𝕏

Closes quest/m2/multicast/mesh-replay.md.
Problem
lite-07 sessions run a new route layer, and a CDN is about to roll its relays onto it one at a time. Nothing ran a mesh of real
moq-relayprocesses through a link reopen, a relay crash, and above all a roll, where two new relays speak lite-07 to each other and lite-06 to an old one, so one route crosses both.Approach
just test mesh-replay(README) meshes real relays overmoq-shaperlinks (bufferbloat, never random loss) from a topology file, runs a publisher and a follower on every relay, and plays a script of events:rolldrains each relay with SIGTERM and starts the other build on its port: the newest release (downloaded and checksummed likecompat.sh, now shared astest/lib/release.sh) or this checkout, or any relay binary with--new(the warmth port's branch build). A roll can stop partway (relays = [...]).reopen-linkpauses one link's shaper past the idle timeout,reopen-relayall of a relay's,killis SIGKILL and a restart,end-sourcestops one relay's publisher and starts it again.For each event the report gives how long until the route and announce counters went quiet and every routed reader read again, the announce-class messages and bytes, the
moq_relay_route_*counters from #9126 (changed, rewritten, DEAD and DRAIN sent and received), readers hit and how many were bystanders, the longest pause, groups missed, and readers whose broadcast stepped through End, Start, End (#9112). A roll also lists, per link of the drained relay, whether the neighbour heard GOAWAY (DRAIN) or only saw it close (DEAD), read from either build's log and cross-checked against the new build's GOAWAY counter, starring the links the drained relay dialed, so the table stays comparable once drain-dialed-links lands.A run fails only on a reader whose relay shows its live broadcast but that reads nothing after the event settles (
stuckwhen its follow holds an ended track,starvedotherwise), a relay still showing a stopped broadcast, or route and announce messages still moving once every publisher stopped. A violation a quest already owns is listed under[[known]]in the script, reported with its quest, and does not fail the run. Counts never gate anything.moq-bench replayis the publisher and follower: one single-frame group per interval per broadcast, and a JSON line per gap, per online/offline/ended change, and per second of state.Nightly runs a ring of four and a ring of six with a chord (diameter 3), each at 4 and 32 broadcasts per relay, and adds every report to the job summary.
Impact
moq-bench replaysubcommand. No library API or wire change.test/interop/compat.shfetches released binaries throughtest/lib/release.sh, which also picks aarch64 assets.moq-relay'srepository_configs_loadtest skipstest/mesh-replay/, whose TOML files are topologies and event scripts, not relay configs.Flaws found
just test mesh-replay --mesh test/mesh-replay/ring6.toml --script test/mesh-replay/half-roll.toml --broadcasts 32(rollingbstarved old relayc's readers in every ring6 cell at 32 broadcasts, 33 to 41 paths; 1 of 3 half-roll runs at 4). The quest has the log trace. Not fixed here: it is in the publisher and model code other quests own.unroutable,internal error) and never handed out another, about 32 paths per roll in ring6 at 32 broadcasts. Noted in the quest.old), which looked like detour-resume; and a pause still going at an event's end was measured past the reader's latest status line, which counted nearly every reader as hit, bystanders included.just test mesh-replay --mesh test/mesh-replay/ring4.toml --script test/mesh-replay/reopen.toml --broadcasts 32(newreopen.toml; starved readers in all 3 runs). The quest has the log trace and a likely cause inmodel/front.rs's stranded tracks. Not fixed here, for the same reason as roll-echo-stall.Nightly mesh report
The nightly sweep on
4c70fbadef's code (ring4andring6, 4 and 32 broadcasts per relay), on this host under other agents' load:The ring6-32 failures are reopen-starve's, whose
[[known]]entry landed after this sweep, so nightly now reports them as known. Nodied,shown, ornoisyviolation in any cell, and no relay showed a stopped broadcast.ring6 at 4 broadcasts, per event:
Links of each rolled relay, as its neighbour saw the drain (
*dialed by the rolled relay), the same in every cell of a mesh: a rolled relay sends GOAWAY only on the sessions it accepted (drain-dialed-links).What the counts say:
a, 2 forb); the forward roll's drains cross lite-06 links and show as GOAWAY in the links table.[End, Start, End]residual on lite-06.a-balso paused 8 bystander paths, whose relay and source were both elsewhere.to old) fed every reader within 4.7s at 4 broadcasts and 12s at 32.ring6 at 32 broadcasts, per event:
The full reports (
report.md,report.json) for every cell are in the run directory, and nightly adds them to the job summary.On
94e3edb7bb, telling reopen-starve apart from the host's load (about 36 on 32 cores from other agents):start = "new"): readercstarved on all 32 ofa's broadcasts for 75s aftera's links reopened. The same faults with every relay on the release (start = "old"), under the same load: every event fed again within 4.1s, no violation.Validation
python3 test/mesh-replay/replay.test.py(injust test harness): 16 pass.cargo test -p moq-relay --lib: 325 pass, and the moq-relay and moq-bench lints pass.just checkthroughquest gate: rustfmt and remark found formatting, fixed;sh/rs/select.sh lintfor moq-bench andsh/markdown.sh checkthen pass; everything else passed on the first run.reopen-starve's pieces are on the branch since
f130d0a43f(the quest,reopen.toml, the catch-all known entry, the wait-list line):replay.test.py16 pass,quest check518 documents ok, andreopen.tomlon ring4 at 32 broadcasts starved 52 paths after reopening every link ofa, with the kill taking 57.5s to feed every reader.The nightly sweep above, and the reproductions named under Flaws found.
Local review after the merge of the parallel session's commits (Codex, 2 sessions this iterate, 6 rounds, 6 findings, all fixed): a pause still going counts only to the reader's last report;
upstream = trueonly in the new build's config; a link counts as lite-07 by the dialer's first version the acceptor lists (rs/moq-tokio/src/quic/backend.rs:905); the start needs every reader routed unless an applied mark may hide one; a roll repeats itstimes.replay.test.py: 19 pass;just checkthroughquest gatepasses; a real ring4 replay (start = "new", one link reopen) passes with the start routed and fed in 1.0s.just checkonce failedjs/net'sa stale drop is recorded under the narrowest budget asked on lite-07(subscriber.test.ts:2998), which this branch does not touch; it fails 3 of 3 in isolation on this checkout's unchangedjs/, and moq-dev/moq#9164 fixes it.Local review: Codex (
quest review --harness codex), 13 rounds, 9 findings: 8 fixed, 1 refuted with evidence. The fixes: the run ignored a dead child or a path never read, double-counted a scrape across a restart, let a known entry excuse the start, and measured pauses past the latest status (twice), a stopped publisher dropped its tracks unfinished, and a link's flows each had their own shaper budget. Two checkpoint rounds came on narrowing report-arithmetic findings with the approach unchallenged, so I kept the approach.CodeRabbit's 17 threads (this iterate):
replay.test.py20 pass (one new: a relay name with-is refused);cargo clippy -p moq-bench --all-targetsandjust checkthroughquest gatepass. On the built binary,moq-bench replay --client-connect ...now fails naming--connect.release_binary moq-relay moq-relay 0.17.17still fetches and checksums the asset, and a missing tag's SHA256SUMS answers 404, the only answer that builds instead. A realjust test mesh-replay --mesh test/mesh-replay/ring4.toml --broadcasts 4passes with 2 known (roll-echo-stall,roll b to newandroll c to new), every other event fed again in at most 4.7s after the fault.Decisions
moq-benchsubcommand, not a new package; the orchestrator is a stdlib Python script with unit tests (replay.test.py, injust test harness), like the other harnesses' report code.peer.<relay>) the run's auth server grants, since the relay strips a peer URL's query.drain_timeout = "10s"on both builds: 0.17.17 drains with a zero window by default and would send no GOAWAY.unrouted), not failed: a deployment's mesh withupstreammarks may hide a route legitimately. The start is the exception: it fails on an unrouted reader unless a mark the starting build applies may hide one, so a run with no routes cannot pass having tested nothing.upstream = truegoes only into the new build's relay config: 0.17.17 denies unknown peer fields, and a deployment rolling off a release without the mark configures its old relays unmarked.starvedreaders in a roll are[[known]]against roll-echo-stall in the nightly script, so nightly stays red only for something new; the reproduction script lists no known and still fails.[[known]]entry excuses thestartevent: it is the baseline before any fault.died), and checks every path each reader was told to follow, so a dead or silent reader cannot pass.[[known]]for every event, after roll-echo-stall's roll entry so a roll's starved reader stays with that quest (rejected: keeping the 32-broadcast cells red every night until the fix, or one entry per event kind, which a starvation carried into a later event escapes). The same as detour-resume's entry;reopen.tomllists no known and still fails.-, so a link's script-visible namedial-acceptis unique (rejected: tuple keys plus an ambiguity check, more code for names no mesh uses).[[known]]entry stays unscoped, per the recorded decision above.Follow-ups
Models
Size: L
Implementer: Claude, claude-opus-5-5, unknown effort
Implementer: Claude Code, claude-opus-5-5, unknown effort (iterate, M)
Implementer: Claude Code, claude-opus-5-5, unknown effort (iterate, S)
(Written by Claude Opus 5.5)
🤖 Generated with Claude Code
Review
Rewritten by
quest reviewafter each round.Head: 0823532
Base: origin/main at 334427b
Risk: high,
it changes .github/workflows/nightly.yml, which is always high: workflows run with the repository's secrets and decide what merges and shipsClassified: high by the always-high paths
Reviewer: Codex, gpt-6.1-sol, xhigh effort
Session:
01a1f32d-1270-7db3-b36e-c04db5b20e3e, round 5Score: 10/10,
The merge preserves the reviewed implementation and updates the drain documentation correctly, with no new issues found in the branch’s changes.Rounds: 8; first round 3 (medium 3); findings 6 (medium 6): fixed 6, refuted 0, open 0
Findings: none