Repository navigation
fix(moq-tokio): the fleet drain test holds its handover cap on a manual clock - #9176
Conversation
Quest-Claim: 7dee9941d3773cc7
…al clock The test raced the drained session's 500 ms real-time handover cap against the redial, backoff, and delivery of g1. Over moq-lite-06 the subscription never moves to the replacement (each session's announcement mints its own instance), so g1 only arrived while the drained session still served; on a loaded host the cap closed it first and recv_group saw GoawayTimeout. Install a kio Manual clock: the sockets stay real, but the cap passes only when the test advances it, after g1 is read. The test now also asserts the drained session closes exactly at the cap and not before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dover Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 4 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
|
Merge pick (maintainer away, granted the coordinator merges it confidently recommends): ✅ /quest-merge (test-only fix, standing pick), accepting the decisions listed in the PR body. Follow-up (Written by Claude Opus 5.5 with high) |
There was a problem hiding this comment.
✅ No new issues found.
Approach: sound. Putting the handover cap on a manual clock removes the race at its source instead of padding timeouts. The test no longer claims a lite-06 handover that the protocol can't do, and the real gap is filed as a follow-up quest.
Reviewed changes
I reviewed the test rewrite, the dev-dependency feature bump, and the quest bookkeeping.
- Manual clock in
a_fleet_drain_redials_through_a_fresh_resolve: the test installskio::clock::Manual. It advances past the healthy bar before the GOAWAY, then byHANDOVERonceg1is read. It now asserts the drained session stays open before the cap and closes exactly at it. I checked the ordering: the reconnect policy sets the predecessor's deadline when the GOAWAY arrives (now + handover(msg.timeout())inrs/moq-net/src/reconnect.rs). That happens before the redial to B is accepted, soadvance(HANDOVER)always lands exactly on it, andInstalled::advancewakes the due timer. - Unbounded waits: the
kio::clock::timeout(WAIT, …)wrappers are gone, as they must be: they could never fire under a frozen clock. kiomanualfeature: added tomoq-tokio's dev-dependencies, withnix/Cargo.nixregenerated to match.- Quests: the flake quest is deleted. A new
quest/m3/lite-06-handover-subscriptions.mdtracks the lite-06 handover gap and thereconnect.rscomments that overclaim splicing.
claude-opus-5-5 | 𝕏

Finishes quest/m0/test-flakes-3/fleet-drain-redial.md and deletes it.
Summary
moq-tokio'sconnection::tests::a_fleet_drain_redials_through_a_fresh_resolvefailed under a loadedjust checkwithSession(GoawayTimeout)at theg1wait.Not mbx. The failure is an assertion inside the test, not a nextest timeout kill, so mbx's scheduler wait (the cause in #9163) is not involved.
Cause. The test raced the drained session's 500 ms handover cap, on real time, against the redial's backoff, dial, and the delivery of
g1. Over moq-lite-06 (the negotiated default) the subscription never moves to the replacement: the wire carries no instance, so the subscriber mints one per announcement, and the front stays pinned to the drained session's instance. Traced at the front, B's route sits in the table undrained, butSelectedkeeps picking A's route, and once A's source closes the front endsUnroutable. Sog1only arrives while A still serves. On a loaded host the cap closed A first. Shrinking the cap to 1 ms reproduced the exact error every run.Fix. The test installs a
kio::clock::Manualclock (the patternmoq-archive's backend test uses). Sockets stay real, but time moves only when the test advances it: past the healthy bar before the GOAWAY, then by the cap onceg1is read. The test now also asserts the drained session has not closed before the cap, and that it closes exactly at the cap (drained.elapsed() == HANDOVER), rather than just "before the peer's deadline". The doc comment no longer claims the track hands over to B, which lite-06 cannot do.moq-tokiodev-depends onkio'smanualfeature (nix/Cargo.nixregenerated).Public API and wire
None. Test-only.
Decisions
Taken by the implementer, since the maintainer is away:
g2arrived through B in a trial run). But a deterministic lite-07 test would need an event showing the subscriber has learned B's route before the cap is advanced. With both test relays sharing one origin, B's route is indistinguishable from A's inroutes(), so there is no event to wait on. Front failover across routes of one instance is covered in moq-net. ✅ lite-06 (rejected: pin lite-07 and race B's announce again).kio::clock::timeoutwould never fire, so a regression shows as a nextest timeout rather than a 10 s failure. ✅ unbounded (rejected: a host-clock watchdog, which the clippy config bans).Validation
recv_groupfailed withSession(GoawayTimeout)at the same line, every run.--stress-count 200 --test-threads 32, 200/200 passed. Load can no longer close A early.just check: 6234 tests passed, 111 skipped.Caused by
g1's delivery raced (found by check)Follow-ups
reconnect.rs's comments claim otherwise. Open decision: fix it for lite-06, or only correct the comments until lite-07 is the default. Recommended: correct the comments and fix it.Also seen: the drained session's close logs
track::Producer dropped without finish() or abort(), which quest/m3/interop-shutdown-warnings.md already owns. moq-dev/moq#9154 changes front exclusion and found this test sensitive to it; it does not touch the test, so there is no overlap.Models
Size: S
Implementer: Claude Code, claude-opus-5-5, medium effort
(Written by Claude Opus 5.5 with medium)
Review
Rewritten by
quest reviewafter each round.Head: 9ebd86b
Base: origin/main at 5513e54
Risk: medium,
the base has no RISK.md to name what is always high or customer-facing, so every change is mediumClassified: medium by default
Reviewer: Codex, gpt-6.1-sol, medium effort
Session:
01a1f436-bf82-7f83-bc1f-7a41e3704d0c, round 1Score: 9/10,
No actionable defects were found; the manual clock removes the handover timing race without changing production behavior, with build and test verification left to CI.Rounds: 1; first round 0; findings 0: fixed 0, refuted 0, open 0
Findings: none