Skip to content

fix(moq-tokio): the fleet drain test holds its handover cap on a manual clock - #9176

Merged
kixelated merged 3 commits into
mainfrom
quest/m0/test-flakes-3/fleet-drain-redial
Nov 19, 2026
Merged

kixelated merged 3 commits into
mainfrom
quest/m0/test-flakes-3/fleet-drain-redial

Conversation

@kixelated

@kixelated kixelated commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Finishes quest/m0/test-flakes-3/fleet-drain-redial.md and deletes it.

Summary

moq-tokio's connection::tests::a_fleet_drain_redials_through_a_fresh_resolve failed under a loaded just check with Session(GoawayTimeout) at the g1 wait.

Not mbx. The failure is an assertion inside the test, not a nextest timeout kill, so mbx's scheduler wait (the cause in #9163) is not involved.

Cause. The test raced the drained session's 500 ms handover cap, on real time, against the redial's backoff, dial, and the delivery of g1. Over moq-lite-06 (the negotiated default) the subscription never moves to the replacement: the wire carries no instance, so the subscriber mints one per announcement, and the front stays pinned to the drained session's instance. Traced at the front, B's route sits in the table undrained, but Selected keeps picking A's route, and once A's source closes the front ends Unroutable. So g1 only arrives while A still serves. On a loaded host the cap closed A first. Shrinking the cap to 1 ms reproduced the exact error every run.

Fix. The test installs a kio::clock::Manual clock (the pattern moq-archive's backend test uses). Sockets stay real, but time moves only when the test advances it: past the healthy bar before the GOAWAY, then by the cap once g1 is read. The test now also asserts the drained session has not closed before the cap, and that it closes exactly at the cap (drained.elapsed() == HANDOVER), rather than just "before the peer's deadline". The doc comment no longer claims the track hands over to B, which lite-06 cannot do.

moq-tokio dev-depends on kio's manual feature (nix/Cargo.nix regenerated).

Public API and wire

None. Test-only.

Decisions

Taken by the implementer, since the maintainer is away:

  • Stay on the default lite-06; don't pin lite-07. Over lite-07 the track does fail over to B after A closes (g2 arrived through B in a trial run). But a deterministic lite-07 test would need an event showing the subscriber has learned B's route before the cap is advanced. With both test relays sharing one origin, B's route is indistinguishable from A's in routes(), so there is no event to wait on. Front failover across routes of one instance is covered in moq-net. ✅ lite-06 (rejected: pin lite-07 and race B's announce again).
  • Waits are unbounded. Under the manual clock, kio::clock::timeout would never fire, so a regression shows as a nextest timeout rather than a 10 s failure. ✅ unbounded (rejected: a host-clock watchdog, which the clippy config bans).

Validation

  • Reproduced first: with the old test and a 1 ms cap, recv_group failed with Session(GoawayTimeout) at the same line, every run.
  • The same 1 ms cap on the new test: --stress-count 200 --test-threads 32, 200/200 passed. Load can no longer close A early.
  • The new test at 500 ms passes (0.04 s, down from 0.5 to 3 s on real time).
  • just check: 6234 tests passed, 111 skipped.

Caused by

  • moq-dev/moq#4132 - added the test with a real-time 500 ms handover cap that g1's delivery raced (found by check)

Follow-ups

  • A moq-lite-06 subscription survives a GOAWAY handover - over lite-06 a live subscription ends when the drained session closes, even with the replacement up, and reconnect.rs's comments claim otherwise. Open decision: fix it for lite-06, or only correct the comments until lite-07 is the default. Recommended: correct the comments and fix it.

Also seen: the drained session's close logs track::Producer dropped without finish() or abort(), which quest/m3/interop-shutdown-warnings.md already owns. moq-dev/moq#9154 changes front exclusion and found this test sensitive to it; it does not touch the test, so there is no overlap.

Models

Size: S
Implementer: Claude Code, claude-opus-5-5, medium effort

(Written by Claude Opus 5.5 with medium)

Review

Rewritten by quest review after each round.

  • Head: 9ebd86b

  • Base: origin/main at 5513e54

  • Risk: medium, the base has no RISK.md to name what is always high or customer-facing, so every change is medium

  • Classified: medium by default

  • Reviewer: Codex, gpt-6.1-sol, medium effort

  • Session: 01a1f436-bf82-7f83-bc1f-7a41e3704d0c, round 1

  • Score: 9/10, No actionable defects were found; the manual clock removes the handover timing race without changing production behavior, with build and test verification left to CI.

  • Rounds: 1; first round 0; findings 0: fixed 0, refuted 0, open 0

  • Findings: none

kixelated and others added 3 commits November 19, 2026 06:07
…al clock

The test raced the drained session's 500 ms real-time handover cap against
the redial, backoff, and delivery of g1. Over moq-lite-06 the subscription
never moves to the replacement (each session's announcement mints its own
instance), so g1 only arrived while the drained session still served; on a
loaded host the cap closed it first and recv_group saw GoawayTimeout.

Install a kio Manual clock: the sockets stay real, but the cap passes only
when the test advances it, after g1 is read. The test now also asserts the
drained session closes exactly at the cap and not before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dover

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title A fleet drain's redial delivers the next group under load fix(moq-tokio): the fleet drain test holds its handover cap on a manual clock Nov 19, 2026
@kixelated
kixelated marked this pull request as ready for review November 19, 2026 14:30
@coderabbitai

coderabbitai Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 4 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: c3fa38bd-347a-4c93-9dcf-38848f5ac77d
Commits

Reviewing files that changed from the base of the PR and between 03bfeea and 9ebd86b.

Files selected for processing: 5

  • nix/Cargo.nix
  • quest/m0/test-flakes-3/fleet-drain-redial.md
  • quest/m3/lite-06-handover-subscriptions.md
  • rs/moq-tokio/Cargo.toml
  • rs/moq-tokio/src/connection.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@kixelated

Copy link
Copy Markdown
Collaborator Author

Merge pick (maintainer away, granted the coordinator merges it confidently recommends): ✅ /quest-merge (test-only fix, standing pick), accepting the decisions listed in the PR body.

Follow-up quest/m3/lite-06-handover-subscriptions.md: ✅ keep, with its open lite-06 fix-or-document question left for the maintainer.

(Written by Claude Opus 5.5 with high)

@kixelated
kixelated enabled auto-merge November 19, 2026 14:31

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Approach: sound. Putting the handover cap on a manual clock removes the race at its source instead of padding timeouts. The test no longer claims a lite-06 handover that the protocol can't do, and the real gap is filed as a follow-up quest.

Reviewed changes

I reviewed the test rewrite, the dev-dependency feature bump, and the quest bookkeeping.

  • Manual clock in a_fleet_drain_redials_through_a_fresh_resolve: the test installs kio::clock::Manual. It advances past the healthy bar before the GOAWAY, then by HANDOVER once g1 is read. It now asserts the drained session stays open before the cap and closes exactly at it. I checked the ordering: the reconnect policy sets the predecessor's deadline when the GOAWAY arrives (now + handover(msg.timeout()) in rs/moq-net/src/reconnect.rs). That happens before the redial to B is accepted, so advance(HANDOVER) always lands exactly on it, and Installed::advance wakes the due timer.
  • Unbounded waits: the kio::clock::timeout(WAIT, …) wrappers are gone, as they must be: they could never fire under a frozen clock.
  • kio manual feature: added to moq-tokio's dev-dependencies, with nix/Cargo.nix regenerated to match.
  • Quests: the flake quest is deleted. A new quest/m3/lite-06-handover-subscriptions.md tracks the lite-06 handover gap and the reconnect.rs comments that overclaim splicing.

Pullfrog  | View workflow run | Using claude-opus-5-5 | 𝕏

@kixelated
kixelated added this pull request to the merge queue Nov 19, 2026
Merged via the queue into main with commit d2e79d3 Nov 19, 2026
10 checks passed
@kixelated
kixelated deleted the quest/m0/test-flakes-3/fleet-drain-redial branch November 19, 2026 14:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant