Skip to content

fix(moq-ffi): the in-place shutdown test's timer is never due under load - #9162

Merged
kixelated merged 3 commits into
mainfrom
quest/m0/test-flakes-3/ffi-shutdown-in-place-poll
Nov 19, 2026
Merged

kixelated merged 3 commits into
mainfrom
quest/m0/test-flakes-3/ffi-shutdown-in-place-poll

Conversation

@kixelated

@kixelated kixelated commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

test::shutdown_waits_for_an_in_place_poll in moq-ffi aborted under load. The held poll armed a 1ms host timer. Whenever the thread was descheduled for more than 1ms between arming that timer and polling it, the timer fired on its first poll, and KIO_DENY_HOST_TIMERS=1 aborted the process ("a 994µs timer fired on the host clock").

The test only needs the held poll to arm a timer on tokio's host driver after shutdown has begun, since a timer polled after its driver is gone panics. Whether that timer fires was never the point. The held poll now sleeps until the clock's last instant (kio::clock::now().saturating_add(Duration::MAX)), which no suspension reaches, and the future after it is unreachable!. That still polls, and so arms, tokio's host timer, unlike sleep(Duration::MAX), which parks unarmed. A manual clock (the quest's other option) would bypass tokio's timer driver, which is what the test guards, so it doesn't fit here.

Deletes quest/m0/test-flakes-3/ffi-shutdown-in-place-poll.md.

Validation

  • Reproduced before the fix: 600 child-process runs of the test at 32 in parallel, alongside one busy-loop hog per CPU (.tmp/repro.sh). 13 aborted (9 + 4 across two sweeps), every one in kio::clock::runtime::guard::fired from the held poll's first Sleep::poll, having waited 991 to 994µs.
  • After the fix: the same sweep passed 600 of 600 (with a 3600s sleep first, then again with the final end-of-time deadline).
  • Mutation check: I made Runtime::shutdown skip waiting for in-flight polls and delayed the test's release by 200ms. The fixed test then fails 3 of 3 runs with tokio's time/entry.rs:534 panic, so the end-of-time timer still registers on the driver and still catches the bug. Without the delay the mutant usually passes, because the release races the runtime teardown. That gap is older than this PR (the 1ms version has the same race) and this change leaves it unchanged.
  • just check passes.

Impact

Public API: none. Wire: none. Test-only change.

Caused by

  • moq-dev/moq#8081 - made a fired host timer abort the test under KIO_DENY_HOST_TIMERS=1, which this test's 1ms sleep (from moq-dev/moq#5140) trips whenever it is already due on its first poll (found by check)

Decisions

  • Took an end-of-time host timer over the quest's suggested mocked clock, so the test keeps exercising tokio's timer driver (reasons above). Round 1 of the local review flagged a 3600s sleep as a longer real wait (REVIEW.md, Tests), so the deadline became the clock's last instant. A later review can undo this.

Follow-ups

None.

Models

Size: S
Implementer: Claude Code, claude-opus-5-5, medium effort

🤖 Generated with Claude Code

(Written by Claude Opus 5.5 with medium)

Review

Rewritten by quest review after each round.

  • Head: 9c3fac4

  • Base: origin/main at 5f5daf2

  • Risk: medium, the base has no RISK.md to name what is always high or customer-facing, so every change is medium

  • Classified: medium by default

  • Reviewer: Codex, gpt-6.1-sol, medium effort

  • Session: 01a1f309-3639-7290-8c1a-378f9c0d3eaa, round 2

  • Score: 10/10, The revised deadline removes the suspension-triggered flake while preserving host timer driver access during shutdown, with no new issues found in the incremental change.

  • Rounds: 2; first round 1 (medium 1); findings 1 (medium 1): fixed 1, refuted 0, open 0

  • Findings: none

  • The last round's findings, at 8686725:

    • Fixed: medium, rs/moq-ffi/src/test.rs:8895: Replace the real-time deadline with deterministic synchronization; The head saturates the deadline to the clock's last representable instant, and clock/timer.rs plus clock/runtime.rs confirm that it still polls Tokio's host timer while remaining pending through any practical suspension.

kixelated and others added 2 commits November 19, 2026 00:43
The held poll armed a 1ms host timer. Descheduled for over 1ms between
arming and polling it, the timer fired on its first poll, which
KIO_DENY_HOST_TIMERS aborts. The test only needs the timer armed on
tokio's driver after shutdown begins, so it now never comes due.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title moq-ffi's in-place shutdown test holds under load fix(moq-ffi): the in-place shutdown test's timer is never due under load Nov 19, 2026
A longer real wait is still a real wait. The end of time is never
reached, yet still arms the timer on tokio's host driver.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review November 19, 2026 09:07
@kixelated

Copy link
Copy Markdown
Collaborator Author

Merge pick (maintainer away, granted the coordinator merges it confidently recommends): ✅ /quest-merge (test-only fix, standing pick), accepting the decision in the PR body to target the clock's last instant instead of a mocked clock.

(Written by Claude Opus 5.5 with high)

@kixelated
kixelated enabled auto-merge November 19, 2026 09:07

@pullfrog pullfrog Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No new issues found.

Approach: sound. The held poll still arms tokio's host timer, which is what the test guards. The deadline is now one that no descheduling can reach, so KIO_DENY_HOST_TIMERS=1 can no longer abort the child.

Reviewed changes

Reviewed the full diff: the test change in rs/moq-ffi/src/test.rs and the deleted quest file.

  • Held poll deadline: shutdown_waits_for_an_in_place_poll now sleeps until kio::clock::now().saturating_add(Duration::MAX) instead of 1ms. It ends in unreachable!, replacing the pending() tail. I confirmed sleep_until and Instant::saturating_add exist in rs/kio/src/clock.
  • Quest file removal: deletes the finished ffi-shutdown-in-place-poll.md quest.

Pullfrog  | View workflow run | Using claude-sonnet-5-5 | 𝕏

@kixelated
kixelated added this pull request to the merge queue Nov 19, 2026
@coderabbitai

coderabbitai Bot commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

The shutdown test now holds a Task::run poll open while shutdown starts. After the test releases the poll, it awaits a timer scheduled at now + Duration::MAX and expects the in-flight call to resolve as Cancelled. The related test-flake note was removed.

Merge Risk: Minimal · up to 9c3fa

This test-only change removes a load-dependent flake in the FFI shutdown test, and it has no public API or runtime impact. No merge-blocking risk remains.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the fix and its subject, moq-ffi's in-place shutdown test. It states the cause as a timer that is never due under load. The changeset replaces the 1ms sleep with an end-of-time deadlin…
Description check Passed The description states the failing test, the 1ms host timer cause, the replacement end-of-time deadline, validation runs, and the deleted quest file. These all relate directly to the changeset, which …
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Title check

Explanation

The title names the fix and its subject, moq-ffi's in-place shutdown test. It states the cause as a timer that is never due under load. The changeset replaces the 1ms sleep with an end-of-time deadline in that test, which matches the title.

Full details: Description check

Explanation

The description states the failing test, the 1ms host timer cause, the replacement end-of-time deadline, validation runs, and the deleted quest file. These all relate directly to the changeset, which deletes the quest markdown file and edits the test.

Finishing Touches

Simplify code

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Recent review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: d7e66a9a-9ce4-4bb2-b92e-49c3227fe42d
Commits

Reviewing files that changed from the base of the PR and between 61e4a38 and 9c3fac4.

Files selected for processing: 2

  • quest/m0/test-flakes-3/ffi-shutdown-in-place-poll.md
  • rs/moq-ffi/src/test.rs
Files with no reviewable changes: 1

  • quest/m0/test-flakes-3/ffi-shutdown-in-place-poll.md

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 1 remain after this review.

Merged via the queue into main with commit ecd3407 Nov 19, 2026
7 checks passed
@kixelated
kixelated deleted the quest/m0/test-flakes-3/ffi-shutdown-in-place-poll branch November 19, 2026 10:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant