Skip to content

fix: the stalled DNS connection test reads the close in real time - #738

Merged
tgockel merged 1 commit into
trunkfrom
678-dns-tcp-test-clock
Oct 27, 2026
Merged

tgockel merged 1 commit into
trunkfrom
678-dns-tcp-test-clock

Conversation

@tgockel

@tgockel tgockel commented Oct 27, 2026

Copy link
Copy Markdown
Owner

a_client_that_reads_no_answers_does_not_keep_its_connection failed in 15 of the 113 trunk-targeting CI runs since it landed, 11 of the 20 red runs since Oct 20, always as "the stalled connection was not closed" and on changes that did not touch the DNS listener (#678).

The test is start_paused over a real loopback socket. Paused, tokio polls I/O once without waiting and then advances the clock to the next timer, so a read that is waiting on the kernel -- here for the close the server sends once its write times out -- finds nothing, the one-second timeout fires at once, and the panic blames a listener that did close. Nothing in virtual time waits for the kernel, so any real delay in delivering the close reads as the close not coming.

The clock is resumed after the virtual idle sleep, so the final read loop's second is a real one. Everything that needed a paused clock is before that point: the fill loop's settle waits, and the server's ten second write timeout, which the sleep still advances past.

Reproduced here by delaying only the TCP FIN and RST segments on loopback in a private network namespace (tc netem behind a flower filter on the flags). With the close 3 ms late the test failed with the CI panic, after the whole scenario had run, and again at 50 ms; with this change it passes at 3, 50 and 500 ms. CPU contention did not reproduce it on this 48-core machine: 120 runs, 12 at a time pinned to four cores, passed both before and after the change. With the server's write timeout removed the test still fails with the same message, so it still checks that a stalled connection is closed.

the_fixed_listener_port_binds_again_after_a_server_first_close, the issue's lesser case, has not been seen in any CI log and is not touched.

Closes #678

Summary

What does this change, and why?

Checklist

  • cargo fmt --all -- --check passes
  • cargo clippy --workspace --all-targets -- -D warnings is clean
  • cargo test --workspace passes
  • Docs in doc/ updated if behavior changed (mdbook build still clean)
  • CHANGELOG updated for user-visible changes

Notes for reviewers

Anything worth calling out -- tricky bits, follow-ups, or intentionally out-of-scope items.

a_client_that_reads_no_answers_does_not_keep_its_connection failed in 15
of the 113 trunk-targeting CI runs since it landed, 11 of the 20 red
runs since Oct 20, always as "the stalled connection was not closed" and
on changes that did not touch the DNS listener (#678).

The test is start_paused over a real loopback socket. Paused, tokio
polls I/O once without waiting and then advances the clock to the next
timer, so a read that is waiting on the kernel -- here for the close the
server sends once its write times out -- finds nothing, the one-second
timeout fires at once, and the panic blames a listener that did close.
Nothing in virtual time waits for the kernel, so any real delay in
delivering the close reads as the close not coming.

The clock is resumed after the virtual idle sleep, so the final read
loop's second is a real one. Everything that needed a paused clock is
before that point: the fill loop's settle waits, and the server's ten
second write timeout, which the sleep still advances past.

Reproduced here by delaying only the TCP FIN and RST segments on
loopback in a private network namespace (tc netem behind a flower
filter on the flags). With the close 3 ms late the test failed with the
CI panic, after the whole scenario had run, and again at 50 ms; with
this change it passes at 3, 50 and 500 ms. CPU contention did not
reproduce it on this 48-core machine: 120 runs, 12 at a time pinned to
four cores, passed both before and after the change. With the server's
write timeout removed the test still fails with the same message, so it
still checks that a stalled connection is closed.

the_fixed_listener_port_binds_again_after_a_server_first_close, the
issue's lesser case, has not been seen in any CI log and is not touched.

Closes #678
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI fails intermittently on unrelated changes: a_client_that_reads_no_answers_does_not_keep_its_connection times a real socket against a paused clock

1 participant