Repository navigation
test(moq-relay): handoff tests wait on events, not a 10s wall clock - #9239
Conversation
Quest-Claim: a3d01d41ff7605ed
A loaded host can stall a relay past any deadline, so each wait follows its event: READY or the process exiting, a session, a drain, an exit. Timers the tests don't exercise (QUIC idle, drain) are set out of reach of a stall. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 36 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
There was a problem hiding this comment.
ℹ️ No critical issues — one minor suggestion inline.
Approach: sound. It fixes the cause, not the symptom. The tests now wait for events instead of a wall clock, and every timer they don't exercise moves out of reach (NEVER). That matters because the relay's default drain_timeout is 10s (crate::DEFAULT_DRAIN_TIMEOUT), so under a stall the old takes_over and stand-in drains could fire on their own. nextest's slow-timeout (60s on the default profile, 120s on ci) still bounds a real hang, and every spawned relay stays in the test's process group, so nextest's kill reaches them.
Reviewed changes
I reviewed the whole diff of handoff.rs and the removed quest file.
ready()races READY against the child's exit: a relay or stand-in that dies before READY now fails at once with its exit status. BothUnixDatagram::recvandChild::waitare cancel-safe inside theselect!loop.- Unbounded waits: the
kio::clock::timeoutwrappers are gone from connects, GOAWAY, relay task ends, child exits, and theDisconnectedwait. Theexitedhelper and thewhatlabels are removed. NEVERandquic(): one 600s value now sets the QUIC idle timeout on clients and relays, the drain timeout on in-process relays, and the stand-ins' drain window. Two tests already set 600s by hand and now share these helpers.a_relay_no_successor_takes_over_from_drainskeeps its 1s drain.stays()builds its own QUIC config: in the killed-relay test,_keptno longer gets the 200ms keep-alive. Only the orphan needed it.
claude-opus-5-5 | 𝕏
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
✅ No new issues found.
Reviewed changes
I reviewed the one commit since the prior pullfrog review (79bfc34..4cdcd57).
serving()polls on a 5ms host-clock sleep: it replaces theyield_nowspin, so the current-thread runtime can park between polls. This addresses the earlier thread. The doc comment explains why polling stays: the registry has no change notification.
claude-sonnet-5-5 | 𝕏

Claims quest/m0/test-flakes-3/handoff-ready-load.md, and finishes it.
Problem
Three
moq-relayhandoff::integration tests failed under a fulljust checkon 2026-11-19 withno READY reached the service manager: Elapsed. Every wait in these tests had a 10s wall-clock deadline, and on a loaded host a real relay process can be stalled past it. The original run's logs show the cause: one relay took 3s to bind its QUIC worker and was still short of READY at 10s, and another logged one line in 10s. Neither one hung; both were starved of CPU.Approach
Each wait now follows its event instead of a deadline (
rs/moq-relay/tests/integration/handoff.rs):ready()races READY against the process exiting, and fails at once with the exit status if the process exits first. Before, it waited out 10s and blamed the service manager.Disconnectedare awaited with no deadline. nextest's slow-timeout (60s) bounds a real hang.NEVER(600s) sets the QUIC idle timeout on every client and relay, the drain timeout on the in-process relays, and the stand-ins' drain window.a_relay_no_successor_takes_over_from_drainskeeps the 1s drain it tests. Before this change, a 10s idle or drain timer could end a session that a test expected to stay open. A stand-in whose successor started after 10s would also drain alone, soa_restart_starts_the_new_relay_beside_the_oldpassed without testing the handoff.The two tests that already set 600s timers now share the helpers, so the file shrinks by 31 lines.
Impact
None: no public API and no wire. Test code only.
Validation
All three stall models ran against the test binary directly, so the
mbxrunner's machine-wide scheduler wait doesn't count toward a test's time.main)CARGO_BIN_EXE_moq-relay), on the three process testsno READY reached the service manager: Elapsed(the original failure)connect timeout,kept serving,outlived its last session,never exiteda_relay_no_successor_takes_over_from_drainsanda_killed_relay_s_clients_are_reset_by_the_one_it_replacedfailexit 3) now fails in 8ms withexited before READY: exit status: 3.stress-nghogs against the tests atnice 19on two cores. It also showed that under thembxrunner, its scheduler wait for the stressed machine counts toward nextest's 60s budget, so that budget, not these tests, is the limit at extreme load.just checkpasses.Alternatives
TIMEOUT: ruled out by the epic's rules. It only moves where load starts failing the tests.Choices made without asking
handoff.rs, not only the READY wait the quest names: the stall models fail the other waits too.serving()still polls the stats registry, since the registry has no change notification. It now polls every 5ms on the host clock, with no bound beyond nextest's timeout. Pullfrog found the earlieryield_nowloop never let the current-thread runtime park. It still ran the in-process relays' tasks, but it pinned a core the relays need on a loaded host.Caused by
Follow-ups
None.
Models
Size: S
Implementer: Claude Code, claude-opus-5-5, unknown effort (arm A)
Implementer: Claude Code, claude-opus-5-5, low effort (iterate, XS, arm B)
(Written by Opus 5.5)
Review
Rewritten by
quest reviewafter each round.Head: 4cdcd57
Base: origin/main at d02174a
Risk: low,
Only integration tests and a quest plan change, so any breakage affects contributors running the handoff tests, and a revert fully undoes it without affecting consumers or deployed peers.Classified: low by Codex, gpt-6.1-sol, low effort, session
01a1fb8a-ae18-7b71-bbb7-2ff55168a039Reviewer: Codex, gpt-6.1-sol, medium effort
Session:
01a1fb8a-bd49-7e61-b2c6-0f5b987a1a97, round 1Score: 9/10,
Source review found no actionable regressions, but runtime confirmation under load remains pending because tests were not run as instructed.Rounds: 2; first round 0; findings 0: fixed 0, refuted 0, open 0
Findings: none