Skip to content

test(moq-relay): handoff tests wait on events, not a 10s wall clock - #9239

Merged
kixelated merged 4 commits into
mainfrom
quest/m0/test-flakes-3/handoff-ready-load
Nov 21, 2026
Merged

kixelated merged 4 commits into
mainfrom
quest/m0/test-flakes-3/handoff-ready-load

Conversation

@kixelated

@kixelated kixelated commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Claims quest/m0/test-flakes-3/handoff-ready-load.md, and finishes it.

Problem

Three moq-relay handoff:: integration tests failed under a full just check on 2026-11-19 with no READY reached the service manager: Elapsed. Every wait in these tests had a 10s wall-clock deadline, and on a loaded host a real relay process can be stalled past it. The original run's logs show the cause: one relay took 3s to bind its QUIC worker and was still short of READY at 10s, and another logged one line in 10s. Neither one hung; both were starved of CPU.

Approach

Each wait now follows its event instead of a deadline (rs/moq-relay/tests/integration/handoff.rs):

  • ready() races READY against the process exiting, and fails at once with the exit status if the process exits first. Before, it waited out 10s and blamed the service manager.
  • Sessions, GOAWAYs, relay task ends, process exits, and the reset client's Disconnected are awaited with no deadline. nextest's slow-timeout (60s) bounds a real hang.
  • Timers the tests don't exercise can no longer fire during a stall. One NEVER (600s) sets the QUIC idle timeout on every client and relay, the drain timeout on the in-process relays, and the stand-ins' drain window. a_relay_no_successor_takes_over_from_drains keeps the 1s drain it tests. Before this change, a 10s idle or drain timer could end a session that a test expected to stay open. A stand-in whose successor started after 10s would also drain alone, so a_restart_starts_the_new_relay_beside_the_old passed without testing the handoff.

The two tests that already set 600s timers now share the helpers, so the file shrinks by 31 lines.

Impact

None: no public API and no wire. Test code only.

Validation

All three stall models ran against the test binary directly, so the mbx runner's machine-wide scheduler wait doesn't count toward a test's time.

Stall model Before (main) After
Every relay process start delayed 15s (wrapper at CARGO_BIN_EXE_moq-relay), on the three process tests 3/3 fail, each no READY reached the service manager: Elapsed (the original failure) 3/3 pass (16 to 45s)
The whole test process (client and in-process relays) SIGSTOPped for 15s at 5, 20, or 40ms in, on all 7 tests 14/21 fail: READY, connect timeout, kept serving, outlived its last session, never exited 21/21 pass
Each spawned relay SIGSTOPped for 11 to 15s on first sight a_relay_no_successor_takes_over_from_drains and a_killed_relay_s_clients_are_reset_by_the_one_it_replaced fail all pass
  • A relay that exits before READY (wrapper exit 3) now fails in 8ms with exited before READY: exit status: 3.
  • CPU starvation reproduced the failure class only intermittently: 8 nice-0 stress-ng hogs against the tests at nice 19 on two cores. It also showed that under the mbx runner, its scheduler wait for the stressed machine counts toward nextest's 60s budget, so that budget, not these tests, is the limit at extreme load.
  • Unloaded, all 7 still pass in 1.2s. just check passes.

Alternatives

  • Raising TIMEOUT: ruled out by the epic's rules. It only moves where load starts failing the tests.
  • Keeping deadlines for their descriptive panics: a hang now shows as nextest's TIMEOUT, with the relays' logs, instead of a named step. A hang kills no more cleanly than before: nextest's SIGTERM reaches the stand-ins, which forward it to their relays, and the directly spawned relays share the test's process group.

Choices made without asking

  • Removed every deadline in handoff.rs, not only the READY wait the quest names: the stall models fail the other waits too.
  • serving() still polls the stats registry, since the registry has no change notification. It now polls every 5ms on the host clock, with no bound beyond nextest's timeout. Pullfrog found the earlier yield_now loop never let the current-thread runtime park. It still ran the in-process relays' tasks, but it pinned a core the relays need on a loaded host.

Caused by

  • moq-dev/moq#8596 - added the handoff tests with a 10s wall-clock deadline on each wait on a real relay (found by check)

Follow-ups

None.

Models

Size: S
Implementer: Claude Code, claude-opus-5-5, unknown effort (arm A)
Implementer: Claude Code, claude-opus-5-5, low effort (iterate, XS, arm B)

(Written by Opus 5.5)

Review

Rewritten by quest review after each round.

  • Head: 4cdcd57

  • Base: origin/main at d02174a

  • Risk: low, Only integration tests and a quest plan change, so any breakage affects contributors running the handoff tests, and a revert fully undoes it without affecting consumers or deployed peers.

  • Classified: low by Codex, gpt-6.1-sol, low effort, session 01a1fb8a-ae18-7b71-bbb7-2ff55168a039

  • Reviewer: Codex, gpt-6.1-sol, medium effort

  • Session: 01a1fb8a-bd49-7e61-b2c6-0f5b987a1a97, round 1

  • Score: 9/10, Source review found no actionable regressions, but runtime confirmation under load remains pending because tests were not run as instructed.

  • Rounds: 2; first round 0; findings 0: fixed 0, refuted 0, open 0

  • Findings: none

kixelated and others added 3 commits November 20, 2026 10:19
A loaded host can stall a relay past any deadline, so each wait follows its
event: READY or the process exiting, a session, a drain, an exit. Timers the
tests don't exercise (QUIC idle, drain) are set out of reach of a stall.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title moq-relay's handoff tests hold under load test(moq-relay): handoff tests wait on events, not a 10s wall clock Nov 20, 2026
@kixelated
kixelated marked this pull request as ready for review November 20, 2026 20:22
@coderabbitai

coderabbitai Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 36 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: a8794194-1eaf-4408-80c8-76618f98a513
Commits

Reviewing files that changed from the base of the PR and between 9828a20 and 4cdcd57.

Files selected for processing: 2

  • quest/m0/test-flakes-3/handoff-ready-load.md
  • rs/moq-relay/tests/integration/handoff.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ No critical issues — one minor suggestion inline.

Approach: sound. It fixes the cause, not the symptom. The tests now wait for events instead of a wall clock, and every timer they don't exercise moves out of reach (NEVER). That matters because the relay's default drain_timeout is 10s (crate::DEFAULT_DRAIN_TIMEOUT), so under a stall the old takes_over and stand-in drains could fire on their own. nextest's slow-timeout (60s on the default profile, 120s on ci) still bounds a real hang, and every spawned relay stays in the test's process group, so nextest's kill reaches them.

Reviewed changes

I reviewed the whole diff of handoff.rs and the removed quest file.

  • ready() races READY against the child's exit: a relay or stand-in that dies before READY now fails at once with its exit status. Both UnixDatagram::recv and Child::wait are cancel-safe inside the select! loop.
  • Unbounded waits: the kio::clock::timeout wrappers are gone from connects, GOAWAY, relay task ends, child exits, and the Disconnected wait. The exited helper and the what labels are removed.
  • NEVER and quic(): one 600s value now sets the QUIC idle timeout on clients and relays, the drain timeout on in-process relays, and the stand-ins' drain window. Two tests already set 600s by hand and now share these helpers. a_relay_no_successor_takes_over_from_drains keeps its 1s drain.
  • stays() builds its own QUIC config: in the killed-relay test, _kept no longer gets the 200ms keep-alive. Only the orphan needed it.

Pullfrog  | Fix all ➔ | Fix 👍s ➔ | View workflow run | Using claude-opus-5-5 | 𝕏

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Reviewed changes

I reviewed the one commit since the prior pullfrog review (79bfc34..4cdcd57).

  • serving() polls on a 5ms host-clock sleep: it replaces the yield_now spin, so the current-thread runtime can park between polls. This addresses the earlier thread. The doc comment explains why polling stays: the registry has no change notification.

Pullfrog  | View workflow run | Using claude-sonnet-5-5 | 𝕏

@kixelated
kixelated added this pull request to the merge queue Nov 21, 2026
Merged via the queue into main with commit 9f3c781 Nov 21, 2026
5 checks passed
@kixelated
kixelated deleted the quest/m0/test-flakes-3/handoff-ready-load branch November 21, 2026 01:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant