Skip to content

test(moq-net): nightly races a lite-07 relay's session tasks in seeded orders - #9128

Merged
kixelated merged 3 commits into
mainfrom
quest/m3/route-race-stress
Nov 19, 2026
Merged

kixelated merged 3 commits into
mainfrom
quest/m3/route-race-stress

Conversation

@kixelated

@kixelated kixelated commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Claims quest/m3/route-race-stress.md and finishes it.

Problem

moq-dev/moq#8987 let a lite-07 relay show a reader a claim before the claim's route reached the origin. A reader asking at once was refused as unroutable and hung. Every route test scripts its order on one thread, so only the interop finish lane saw it, and only sometimes. moq-dev/moq#9052 found it with a scratch stress test it didn't keep. This PR adds a standing test for that class of bug: a session task's work becoming visible to other sessions before its own follow-up lands.

Approach

rs/moq-net/tests/integration/route_race.rs runs seeded rounds. Each round sets up a publisher with two sessions to a relay and 2 to 8 readers, all over mock lite-07 sessions. Then it races these steps, each on its own task:

  • Replace: end live/old (drop or withdraw) and announce live/new on the same session, in either order.
  • Announce(n): 0 to 16 more broadcasts on that session.
  • Every reader requests each claim the moment its routes() stream shows it, so requests arrive while routes are landing.
  • Widen(i): the relay widens a narrow reader's grant (Session::auth().authorize), showing it wide/stay. The narrow grant is set before the relay's side of that session first runs.
  • Close: the second publishing session is aborted by either end, or dropped. Leave(i): a reader's session closes.

Once the round settles, it checks:

  • no request is still pending (a hang);
  • every remaining reader shows, and has read, each broadcast still live;
  • no reader still shows live/old or live/gone.

A failure prints the seed and round that replay it, the shape, the steps taken before each failed request, and the round's trace.

Scheduler. support/schedule.rs is a test-only scheduler on top of kio's sim executor. While a schedule is installed, harness::spawn (every session and origin driver) and the test's own tasks may be held back for 1 to 32 executor passes before a poll, drawn from the run's seed. A held task re-wakes itself, so every other ready task runs first, the way a descheduled thread would let it. Rounds run on kio::clock::sim::run_seeded, so the seed replays the schedule. When nothing is installed, harness::spawn behaves exactly as before.

What it cannot reach. It reorders whole task polls. An order inside one poll is reachable only where that poll yields, as at a kio::coop::Budget. #9052's window is one of those: route_stream.rs takes the peer's claims into the node, which shows them to the other sessions, then serves them into the origin on the next loop pass, behind the budget's yield. The extra Announce steps give that yield more chances to land in the window. Real threads can interleave anywhere, including between two lock acquisitions with no yield between them, and this stress cannot.

Compared with Random fault schedules. That quest moves faults in simulated time. At one instant the simulator always wakes tasks in the same order, so it cannot reach the orders this test draws. #9052's bug happened at zero latency within one instant.

Impact

  • Public API: none. Wire: none. Test and CI only.
  • Nightly gains a test route-race lane: 10,000 rounds from a fresh seed, about 2.5 minutes on this host.
  • The merge gate runs 16 rounds from a fixed seed (MOQ_RACE_ROUNDS defaults to 16). That takes about 3 s in a debug build on a quiet host (0.17 s per round), and 8 s when the host's load average was around 30. It is one test in the integration binary, so it adds about that much to one nextest worker. It is worth it as a smoke test that keeps the harness from rotting and catches a race that fails every round. The nightly lane carries the teeth.

Validation

Teeth: #9052's fix reverted (its lite/publisher.rs hunk), release build with debug assertions, as nightly runs it:

  • 10 seeds (101, 202, ... 1010) failed after 40, 289, 527, 612, 651, 981, 992, 1262, 1415 and 1464 rounds, about once per 820 rounds.
  • Every failure was fix(moq-net): a lite-07 relay waits for a claim's route before answering a request for it #9052's symptom: readers' requests for a newly announced live/<n> or live/new hang. For example: reader 2's request 84 for live/6 hangs, made after steps [...].
  • That rate sets the nightly count. 10,000 rounds miss a bug that common less than once in 100,000 nights (e^-12). 5,000 would miss about once in 440.

Replay. Replaying a failing round by its printed MOQ_RACE_SEED/MOQ_RACE_ROUND reproduced the failure 39 of 39 times: 3 each for 13 failures, across the schedule's two RNG versions during development. Replay is not bit-exact: moq-net iterates std::collections::HashMaps, which are seeded per process, so events within one batch can come out in a different order (seen as reordered Update events after a reader left). The module doc says so.

No false positives, no new bug. With the fix in place:

  • 6 × 10,000 rounds were clean, on fresh seeds 11 to 66.
  • Half of those ran with a scratch hack (not committed) that made every kio::coop::Budget yield at random under the sim, reaching more orders inside polls.
  • just test route-race ran end to end (5,000 rounds at the time, seed 709855353): passed in 70 s.

A second, independent check. The scratch coop-preemption hack alone, without the extra announcements, also failed against the reverted fix (round 448 on the default seed), with the same symptom.

Checks. just check passes except for one JS test, a stale drop is recorded under the narrowest budget asked on lite-07. It fails 4 of 5 runs alone on this base, which touches no JS, and no open PR fixes it. It is filed as a follow-up below.

Decisions

  • Scheduler: a controlled scheduler (seeded task holds on the sim), not real threads. Shuttle can't run kio without swapping its locks and crossbeam-epoch for shuttle's, so I built the scheduler at the task level instead. It has teeth, and a failure replays from its seed, which threads can't give. Its blind spot is stated above.
  • No kio change. Random yields at every coop budget would reach more orders inside polls, and they also caught the bug. But turning them on needs a knob on kio's sim module, which is public API on a published crate, and the quest rules out new public API. The extra announcements reach fix(moq-net): a lite-07 relay waits for a claim's route before answering a request for it #9052's window without it. See Alternatives.
  • The goal's "multi-threaded runtime" is met by the seeded schedule standing in for threads. moq-net's AGENTS.md keeps its tests off tokio and the wall clock, and a wall-clock hang detector on real threads would be the kind of timing test the rules reject.
  • Hang detection is a 10 s settle in simulated time, which costs no wall time. Everything here is zero-latency, so a request still pending after it is a hang, not a slow answer.
  • A refused request counts against the claim the reader shows at the end. A refusal for a claim that later ended (live/old, live/gone) is legitimate. A refusal or clean end for a claim still shown is a failure, and so is any request left pending.

Alternatives

Context

  • Nothing here reproduced a hang in track replacement or route switching (Replace drops or withdraws live/old while readers hold subscriptions, and the second session's close takes live/gone offline): 60,000 clean rounds with the fix. So this does not explain the interop hang quest/m0/test-flakes-3/track-replacement-bun.md is chasing.
  • The new files touch no one else's route tests. support/harness.rs gains one line (its sim spawn goes through schedule::spawn), and benches/session includes schedule.rs alongside harness.rs.

Follow-ups

  • quest/m0/test-flakes-3/js-stale-budget.md: js/net's a stale drop is recorded under the narrowest budget asked on lite-07 fails 4 of 5 runs alone on main (the second narrowing leaves the budget at 500). No open decisions.

Models

Size: M
Implementer: Claude Code, claude-opus-5-5, unknown effort (arm A)

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

Review

Rewritten by quest review after each round.

  • Head: 9c1ba54

  • Base: origin/main at b2372ad

  • Risk: high, it changes .github/workflows/nightly.yml, which is always high: workflows run with the repository's secrets and decide what merges and ships

  • Classified: high by the always-high paths

  • Reviewer: Codex, gpt-6.1-sol, high effort

  • Session: 01a1f1ca-bc18-74e0-a14c-bf2400427c27, round 1

  • Score: 9/10, Source review found no actionable defects: the scheduler is opt-in, waits use simulated time, and existing harness behavior is preserved; builds and tests were not run, as instructed.

  • Rounds: 1; first round 0; findings 0: fixed 0, refuted 0, open 0

  • Findings: none

kixelated and others added 2 commits November 18, 2026 14:43
…d orders

A publisher, a relay, and 2 to 8 readers over mock lite-07 sessions, with
every task held back at seeded points so session tasks run in the orders
separate threads could give them. Each round races a broadcast ending as
another starts, more announcements, readers requesting each claim the
moment they hear of it, a grant widening, and a publishing session closing
and a reader leaving. A failure prints the seed and round that replay it,
the steps taken before each failed request, and the round's trace.

With #9052's fix reverted it fails about once per 820 rounds.
Nightly runs 10000 from a fresh seed; PRs run a fixed 16.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title Nightly races the route layer across threads test(moq-net): nightly races a lite-07 relay's session tasks in seeded orders Nov 19, 2026
@kixelated
kixelated marked this pull request as ready for review November 19, 2026 03:14
@coderabbitai

coderabbitai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 1 minute.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 7111f153-139f-4e10-a300-03d6735cb9df
Commits

Reviewing files that changed from the base of the PR and between b2372ad and 6b3cb08.

Files selected for processing: 11

  • .github/workflows/nightly.yml
  • quest/m0/test-flakes-3/js-stale-budget.md
  • quest/m3/route-race-stress.md
  • rs/moq-net/benches/session/mod.rs
  • rs/moq-net/tests/integration/main.rs
  • rs/moq-net/tests/integration/route_race.rs
  • rs/moq-net/tests/integration/support/harness.rs
  • rs/moq-net/tests/integration/support/mod.rs
  • rs/moq-net/tests/integration/support/schedule.rs
  • test/justfile
  • test/route-race.sh
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

pullfrog[bot]
pullfrog Bot previously approved these changes Nov 19, 2026

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Approach: sound. Holding whole task polls on kio's sim with a seeded, opt-in wrapper gives replayable reorderings without touching kio's public API. Routing harness::spawn through it means every session and origin driver is covered, and the module doc and PR body state the blind spot (orders inside one poll that never yields).

Reviewed changes

All 11 files at 9c1ba54, read against kio's sim executor (rs/kio/src/clock/sim.rs) and the shared integration harness.

  • support/schedule.rs: a thread-local Schedule whose Held wrapper draws a 1..=longest hold from kio::clock::seed() and wakes itself while held. In run_inner, a task that wakes itself goes back in the FuturesUnordered queue without moving time, so holds reorder polls and never use up the simulated settle. With no schedule installed, spawn is plain kio::clock::sim::spawn.
  • support/harness.rs + benches/session/mod.rs: the sim branch of harness::spawn now goes through schedule::spawn. The bench #[path]-includes schedule.rs, so the shared harness still compiles there, and #![allow(dead_code)] covers the parts the bench doesn't use.
  • route_race.rs: seeded rounds race Replace, Close, Widen, Leave, and Announce against readers that request each claim the moment it shows. After a 10 s simulated settle, a round fails on a request still pending, a remaining reader still showing live/old or live/gone, or a live claim that a remaining reader doesn't show or hasn't read. Each round's seed comes only from the base seed and the round index, so the printed replay line selects exactly that round.
  • Nightly lane: test/route-race.sh runs 10,000 rounds from a fresh seed, in release with debug assertions on, under nextest's sweep profile (60 s × 60 budget). It is wired through test/justfile and the nightly.yml matrix the same way as route-sweep.
  • Quests: deletes the finished quest/m3/route-race-stress.md, which no other file links to, and adds the JS flake follow-up under test-flakes-3.

Ran locally (debug build): the default 16 rounds passed in 3.6 s, and 200 rounds from seed 424242 passed in 43 s. I did not re-run the teeth check with #9052's fix reverted.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

…tress

# Conflicts:
#	rs/moq-net/tests/integration/main.rs

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

The only new commit since the last review merges origin/main into the branch, and the PR's own changes are the same as before.

  • Merged origin/main: the PR's only change in rs/moq-net/tests/integration/main.rs is still one added line, mod route_race;. The surrounding lines differ only because main added mod route_stats and removed mod session_panic. The new code (route_race.rs, support/schedule.rs, the harness hook, and the nightly lane) is the same as in the approved version.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

@kixelated

Copy link
Copy Markdown
Collaborator Author

Maintainer decision (2026-11-19, in the coordinating session): merge on the coordinator's recommendation while the maintainer travels.

Under that pick, the coordinator's choices:

  • ✅ Merge, with the controlled scheduler in place of real threads, as the PR's Decisions section records. Its blind spot (it reorders only where a task yields) is stated in the PR and accepted.
  • ✅ Keep the follow-up, quest/m0/test-flakes-3/js-stale-budget.md (rejected: dropping it): a test that fails 4 of 5 runs on main needs an owner.
  • The kio knob for preemptible yields stays an alternative, not a quest: it needs public API on kio's sim.

(Written by Claude Fable 5.2 with high effort)

@kixelated

Copy link
Copy Markdown
Collaborator Author

Answered: the decision above is recorded in #9128 (comment) and nothing else waits on the maintainer here.

(Written by Claude Fable 5.2 with high effort)

@kixelated
kixelated added this pull request to the merge queue Nov 19, 2026
Merged via the queue into main with commit 69aa60e Nov 19, 2026
6 checks passed
@kixelated
kixelated deleted the quest/m3/route-race-stress branch November 19, 2026 03:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant