Repository navigation
test(moq-net): nightly races a lite-07 relay's session tasks in seeded orders - #9128
Conversation
Quest-Claim: 3a26cc4418ee1143
…d orders A publisher, a relay, and 2 to 8 readers over mock lite-07 sessions, with every task held back at seeded points so session tasks run in the orders separate threads could give them. Each round races a broadcast ending as another starts, more announcements, readers requesting each claim the moment they hear of it, a grant widening, and a publishing session closing and a reader leaving. A failure prints the seed and round that replay it, the steps taken before each failed request, and the round's trace. With #9052's fix reverted it fails about once per 820 rounds. Nightly runs 10000 from a fresh seed; PRs run a fixed 16. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 1 minute. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
There was a problem hiding this comment.
✅ No new issues found.
Approach: sound. Holding whole task polls on kio's sim with a seeded, opt-in wrapper gives replayable reorderings without touching kio's public API. Routing harness::spawn through it means every session and origin driver is covered, and the module doc and PR body state the blind spot (orders inside one poll that never yields).
Reviewed changes
All 11 files at 9c1ba54, read against kio's sim executor (rs/kio/src/clock/sim.rs) and the shared integration harness.
support/schedule.rs: a thread-localSchedulewhoseHeldwrapper draws a 1..=longesthold fromkio::clock::seed()and wakes itself while held. Inrun_inner, a task that wakes itself goes back in theFuturesUnorderedqueue without moving time, so holds reorder polls and never use up the simulated settle. With no schedule installed,spawnis plainkio::clock::sim::spawn.support/harness.rs+benches/session/mod.rs: the sim branch ofharness::spawnnow goes throughschedule::spawn. The bench#[path]-includesschedule.rs, so the shared harness still compiles there, and#![allow(dead_code)]covers the parts the bench doesn't use.route_race.rs: seeded rounds raceReplace,Close,Widen,Leave, andAnnounceagainst readers that request each claim the moment it shows. After a 10 s simulated settle, a round fails on a request still pending, a remaining reader still showinglive/oldorlive/gone, or a live claim that a remaining reader doesn't show or hasn't read. Each round's seed comes only from the base seed and the round index, so the printed replay line selects exactly that round.- Nightly lane:
test/route-race.shruns 10,000 rounds from a fresh seed, in release with debug assertions on, under nextest'ssweepprofile (60 s × 60 budget). It is wired throughtest/justfileand thenightly.ymlmatrix the same way asroute-sweep. - Quests: deletes the finished
quest/m3/route-race-stress.md, which no other file links to, and adds the JS flake follow-up undertest-flakes-3.
Ran locally (debug build): the default 16 rounds passed in 3.6 s, and 200 rounds from seed 424242 passed in 43 s. I did not re-run the teeth check with #9052's fix reverted.
claude-opus-5-5 | 𝕏
…tress # Conflicts: # rs/moq-net/tests/integration/main.rs
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes
The only new commit since the last review merges origin/main into the branch, and the PR's own changes are the same as before.
- Merged
origin/main: the PR's only change inrs/moq-net/tests/integration/main.rsis still one added line,mod route_race;. The surrounding lines differ only because main addedmod route_statsand removedmod session_panic. The new code (route_race.rs,support/schedule.rs, the harness hook, and the nightly lane) is the same as in the approved version.
claude-opus-5-5 | 𝕏
|
Maintainer decision (2026-11-19, in the coordinating session): merge on the coordinator's recommendation while the maintainer travels. Under that pick, the coordinator's choices:
(Written by Claude Fable 5.2 with high effort) |
|
Answered: the decision above is recorded in #9128 (comment) and nothing else waits on the maintainer here. (Written by Claude Fable 5.2 with high effort) |

Claims quest/m3/route-race-stress.md and finishes it.
Problem
moq-dev/moq#8987 let a lite-07 relay show a reader a claim before the claim's route reached the origin. A reader asking at once was refused as unroutable and hung. Every route test scripts its order on one thread, so only the interop
finishlane saw it, and only sometimes. moq-dev/moq#9052 found it with a scratch stress test it didn't keep. This PR adds a standing test for that class of bug: a session task's work becoming visible to other sessions before its own follow-up lands.Approach
rs/moq-net/tests/integration/route_race.rsruns seeded rounds. Each round sets up a publisher with two sessions to a relay and 2 to 8 readers, all over mock lite-07 sessions. Then it races these steps, each on its own task:Replace: endlive/old(drop or withdraw) and announcelive/newon the same session, in either order.Announce(n): 0 to 16 more broadcasts on that session.routes()stream shows it, so requests arrive while routes are landing.Widen(i): the relay widens a narrow reader's grant (Session::auth().authorize), showing itwide/stay. The narrow grant is set before the relay's side of that session first runs.Close: the second publishing session is aborted by either end, or dropped.Leave(i): a reader's session closes.Once the round settles, it checks:
live/oldorlive/gone.A failure prints the seed and round that replay it, the shape, the steps taken before each failed request, and the round's trace.
Scheduler.
support/schedule.rsis a test-only scheduler on top of kio's sim executor. While a schedule is installed,harness::spawn(every session and origin driver) and the test's own tasks may be held back for 1 to 32 executor passes before a poll, drawn from the run's seed. A held task re-wakes itself, so every other ready task runs first, the way a descheduled thread would let it. Rounds run onkio::clock::sim::run_seeded, so the seed replays the schedule. When nothing is installed,harness::spawnbehaves exactly as before.What it cannot reach. It reorders whole task polls. An order inside one poll is reachable only where that poll yields, as at a
kio::coop::Budget. #9052's window is one of those:route_stream.rstakes the peer's claims into the node, which shows them to the other sessions, then serves them into the origin on the next loop pass, behind the budget's yield. The extraAnnouncesteps give that yield more chances to land in the window. Real threads can interleave anywhere, including between two lock acquisitions with no yield between them, and this stress cannot.Compared with Random fault schedules. That quest moves faults in simulated time. At one instant the simulator always wakes tasks in the same order, so it cannot reach the orders this test draws. #9052's bug happened at zero latency within one instant.
Impact
test route-racelane: 10,000 rounds from a fresh seed, about 2.5 minutes on this host.MOQ_RACE_ROUNDSdefaults to 16). That takes about 3 s in a debug build on a quiet host (0.17 s per round), and 8 s when the host's load average was around 30. It is one test in the integration binary, so it adds about that much to one nextest worker. It is worth it as a smoke test that keeps the harness from rotting and catches a race that fails every round. The nightly lane carries the teeth.Validation
Teeth: #9052's fix reverted (its
lite/publisher.rshunk), release build with debug assertions, as nightly runs it:live/<n>orlive/newhang. For example:reader 2's request 84 for live/6 hangs, made after steps [...].Replay. Replaying a failing round by its printed
MOQ_RACE_SEED/MOQ_RACE_ROUNDreproduced the failure 39 of 39 times: 3 each for 13 failures, across the schedule's two RNG versions during development. Replay is not bit-exact: moq-net iteratesstd::collections::HashMaps, which are seeded per process, so events within one batch can come out in a different order (seen as reorderedUpdateevents after a reader left). The module doc says so.No false positives, no new bug. With the fix in place:
kio::coop::Budgetyield at random under the sim, reaching more orders inside polls.just test route-raceran end to end (5,000 rounds at the time, seed 709855353): passed in 70 s.A second, independent check. The scratch coop-preemption hack alone, without the extra announcements, also failed against the reverted fix (round 448 on the default seed), with the same symptom.
Checks.
just checkpasses except for one JS test,a stale drop is recorded under the narrowest budget asked on lite-07. It fails 4 of 5 runs alone on this base, which touches no JS, and no open PR fixes it. It is filed as a follow-up below.Decisions
simmodule, which is public API on a published crate, and the quest rules out new public API. The extra announcements reach fix(moq-net): a lite-07 relay waits for a claim's route before answering a request for it #9052's window without it. See Alternatives.live/old,live/gone) is legitimate. A refusal or clean end for a claim still shown is a failure, and so is any request left pending.Alternatives
coop::Budgetyield at random, drawn from the run's seed. It would turn every cooperative yield in every crate into a preemption point for any sim test that opts in. It needs public API on kio, so I left it for the maintainer to want. It is listed here, not filed as a quest.Context
Replacedrops or withdrawslive/oldwhile readers hold subscriptions, and the second session's close takeslive/goneoffline): 60,000 clean rounds with the fix. So this does not explain the interop hang quest/m0/test-flakes-3/track-replacement-bun.md is chasing.support/harness.rsgains one line (its sim spawn goes throughschedule::spawn), andbenches/sessionincludesschedule.rsalongsideharness.rs.Follow-ups
js/net'sa stale drop is recorded under the narrowest budget asked on lite-07fails 4 of 5 runs alone on main (the second narrowing leaves the budget at 500). No open decisions.Models
Size: M
Implementer: Claude Code, claude-opus-5-5, unknown effort (arm A)
🤖 Generated with Claude Code
(Written by Claude Opus 5.5)
Review
Rewritten by
quest reviewafter each round.Head: 9c1ba54
Base: origin/main at b2372ad
Risk: high,
it changes .github/workflows/nightly.yml, which is always high: workflows run with the repository's secrets and decide what merges and shipsClassified: high by the always-high paths
Reviewer: Codex, gpt-6.1-sol, high effort
Session:
01a1f1ca-bc18-74e0-a14c-bf2400427c27, round 1Score: 9/10,
Source review found no actionable defects: the scheduler is opt-in, waits use simulated time, and existing harness behavior is preserved; builds and tests were not run, as instructed.Rounds: 1; first round 0; findings 0: fixed 0, refuted 0, open 0
Findings: none