Repository navigation
test: stop the relay-client tests redialling a port they freed - #36663
bolichen97 wants to merge 1 commit into
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS This is a test-only change aimed at a real cross-worker port race. Keeping each port held fixes the cause. Product code and assertions are unchanged. Notes
[DESIGN-REVIEWED] ffef948 Description read[DESCRIPTION-READ] 5cd16fee9f6215908f93b51151b910160088601425890087f5e3999ec57a0267 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Code change approver recommendation: ApproveWhy: the decider estimates a 5% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes. Test-only fix for a port-reuse race in relay-client tests. A bound-but-never-listening socket keeps each port held so every call to it is refused. The fake lease service gets a An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
|
Review passed on this head. Please rebase onto the latest main so CI can run fresh; the only red tests on this head come from main's shared conductor-parity failure, which main fixed after this run was created. The approval will follow once the rebased head is green. |
Several relay-client tests made a fake server or a probe unreachable by freeing its loopback port, then let the client call that port and expected the call to fail. Under xdist another listener on the host could take the freed port in between, so the call was answered (CI saw a 404) instead of failing. The fake relay and the fake lease service now drop every call in place while they keep their port, and the relay dial test and the lease tests use that. The unreachable-crew test keeps its probe socket bound, never listening, for the whole test: the connect still fails with an error whose text names the port, which is what the test checks the relay is never sent. A bound socket is not used for the dial and lease tests because macOS drops a connect to one until it times out instead of refusing it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Rebased: main moved in test/test_relay_client_lease.py since this head ran CI (c22557c -> bdc7e7d); CI re-running. The only reds on the previous head were Backend Tests shard 3 (Linux and Windows) on [operator: bolichen97] |
964e946 to
ffef948
Compare
Problem / Motivation
Goal: The relay-client tests that check what happens when the relay, the lease service or the Crew's web server cannot be reached stop going red when another test on the same machine takes the port they just let go of.
test_a_dial_that_keeps_failing_says_so_in_statusfailed once on a pull request that does not touch the relay client: the status readthe last relay connection ended: upgrade answered 404where the test expects it to start withthe last relay connection ended: dial failed:. The client reached something on the fake relay's old port and got a 404, where the test expects the dial itself to fail.Why it matters
A red that has nothing to do with a pull request costs its author a rerun and an investigation, and it teaches people to rerun reds without reading them. Four more relay-client tests carry the same race and simply have not lost it yet.
Not a goal
What changed (motivation → approach → change)
Each of these tests now keeps hold of the port it expects to fail, so nothing else on the machine can answer on it.
The tests made a server or a probe socket unreachable by closing it, which frees its loopback port, and then let the client call that port again. Under xdist another worker's test server can be handed the freed port in between, so the call that should fail is answered instead (CI saw a 404).
test_a_dial_that_keeps_failing_says_so_in_statusreachableflag is off, drops every dial with no answer throughdrop_unansweredintest/relay_client_helpers.py; the status still readsdial failed:test_a_service_it_cannot_reach_is_retried_later,test_a_spent_host_token_is_not_dialed_while_renewals_fail,test_a_move_with_no_usable_host_token_keeps_the_old_connectionreachableflag on the fake lease service; the source still reportscould not be reachedtest_an_unreachable_crew_sends_the_relay_no_port_or_os_error_textThe relay and lease tests drop calls instead of holding a bound socket because macOS does not refuse a connect to a socket that is bound but not listening: it drops it until the connect times out, which would take the dial test past its wait. Every assertion is unchanged.
Backwards compatibility
Compatible: a test-only change; the fake relay and the fake lease service only gain a
reachableflag that starts on.Removes nothing: every test keeps its assertions; only the way the unreachable port is produced changes.
Tests
Proof: not provable -- test-only change
The fix is proven with the forced condition from testing-conventions ("Proving a determinism fix"), run on Ubuntu 24.04 (Python 3.12.3, aiohttp 3.14.5): a throwaway pytest plugin, not committed, makes another listener take every port a closed aiohttp test server or a closed bound-but-never-listening socket frees, and answer on it.
bdc7e7da79, plugin on: all six node ids red, with the reported text:'the last relay connection ended: upgrade answered 404'for the dial test,'the lease service refused the renewal (HTTP 404)'for the four lease ones, and a200response where the unreachable-crew test expectscrew_unreachable.test/test_relay_client*.pyfile under the plugin at-n 4 --dist loadgroup: 615 passed on this head; on the parent the same sweep went red on exactly these six ids and nowhere else (6 failed, 609 passed).-n 4 --dist loadgroup(187 passed each); one shuffled order of the 187 ids, seed 14186, all passed.test/test_flake_pattern_ratchet.pyandtest/test_wait_discipline_ratchet.pypass; semgrep on the diff: 0 findings.Manual verification
N/A — test-only change; the forced-condition runs above are the reproduction and the check that it is gone.
Related Issues
Closes #35371
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)