Skip to content

test: stop the relay-client tests redialling a port they freed - #36663

Open
bolichen97 wants to merge 1 commit into
mainfrom
fix/35371-held-refusing-port
Open

bolichen97 wants to merge 1 commit into
mainfrom
fix/35371-held-refusing-port

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The relay-client tests that check what happens when the relay, the lease service or the Crew's web server cannot be reached stop going red when another test on the same machine takes the port they just let go of.

test_a_dial_that_keeps_failing_says_so_in_status failed once on a pull request that does not touch the relay client: the status read the last relay connection ended: upgrade answered 404 where the test expects it to start with the last relay connection ended: dial failed: . The client reached something on the fake relay's old port and got a 404, where the test expects the dial itself to fail.

Why it matters

A red that has nothing to do with a pull request costs its author a rerun and an investigation, and it teaches people to rerun reds without reading them. Four more relay-client tests carry the same race and simply have not lost it yet.

Not a goal

  • No change to the relay client or the lease token source: product code is untouched.
  • No change to how a failed dial or an unreachable lease service is reported.
  • No change to how tests outside the relay-client suite pick or release ports.

What changed (motivation → approach → change)

Each of these tests now keeps hold of the port it expects to fail, so nothing else on the machine can answer on it.

The tests made a server or a probe socket unreachable by closing it, which frees its loopback port, and then let the client call that port again. Under xdist another worker's test server can be handed the freed port in between, so the call that should fail is answered instead (CI saw a 404).

Test How the port stays held
test_a_dial_that_keeps_failing_says_so_in_status the fake relay keeps running and, once its new reachable flag is off, drops every dial with no answer through drop_unanswered in test/relay_client_helpers.py; the status still reads dial failed:
test_a_service_it_cannot_reach_is_retried_later, test_a_spent_host_token_is_not_dialed_while_renewals_fail, test_a_move_with_no_usable_host_token_keeps_the_old_connection the same reachable flag on the fake lease service; the source still reports could not be reached
test_an_unreachable_crew_sends_the_relay_no_port_or_os_error_text its probe socket stays bound, never listening, for the whole test instead of being closed before the client starts. The connect still fails with an error whose own text names the port, which is what the test checks the relay never receives

The relay and lease tests drop calls instead of holding a bound socket because macOS does not refuse a connect to a socket that is bound but not listening: it drops it until the connect times out, which would take the dial test past its wait. Every assertion is unchanged.

Backwards compatibility

Compatible: a test-only change; the fake relay and the fake lease service only gain a reachable flag that starts on.

Removes nothing: every test keeps its assertions; only the way the unreachable port is produced changes.

Tests

Proof: not provable -- test-only change

The fix is proven with the forced condition from testing-conventions ("Proving a determinism fix"), run on Ubuntu 24.04 (Python 3.12.3, aiohttp 3.14.5): a throwaway pytest plugin, not committed, makes another listener take every port a closed aiohttp test server or a closed bound-but-never-listening socket frees, and answer on it.

  • Parent bdc7e7da79, plugin on: all six node ids red, with the reported text: 'the last relay connection ended: upgrade answered 404' for the dial test, 'the lease service refused the renewal (HTTP 404)' for the four lease ones, and a 200 response where the unreachable-crew test expects crew_unreachable.
  • This head, same plugin: 6 passed. Every test/test_relay_client*.py file under the plugin at -n 4 --dist loadgroup: 615 passed on this head; on the parent the same sweep went red on exactly these six ids and nowhere else (6 failed, 609 passed).
  • Repeats, unforced: the three changed test files 3/3 green at -n 4 --dist loadgroup (187 passed each); one shuffled order of the 187 ids, seed 14186, all passed.
  • test/test_flake_pattern_ratchet.py and test/test_wait_discipline_ratchet.py pass; semgrep on the diff: 0 findings.

Manual verification

N/A — test-only change; the forced-condition runs above are the reproduction and the check that it is gone.

Related Issues

Closes #35371

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision ffef948bcc5cf96f972456b9e11540cf49ae104d touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision ffef948bcc5cf96f972456b9e11540cf49ae104d ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of ffef948bcc5cf96f972456b9e11540cf49ae104d — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

This is a test-only change aimed at a real cross-worker port race. Keeping each port held fixes the cause. Product code and assertions are unchanged.

Notes

  • On macOS, test_an_unreachable_crew_sends_the_relay_no_port_or_os_error_text may only fail after the 10 s sock_connect timeout in relay_client/ingress.py:409, because macOS drops a connect to a bound socket that is not listening. That is the reason the description gives for not using this approach in the relay and lease tests. Check that await_until's bound covers 10 s, or let the fake crew drop the request with drop_unanswered as the relay and lease fakes do.

[DESIGN-REVIEWED] ffef948

Description read

[DESCRIPTION-READ] 5cd16fee9f6215908f93b51151b910160088601425890087f5e3999ec57a0267

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed ffef948bcc5cf96f972456b9e11540cf49ae104d — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] ffef948

Verdict parsed from the review's SHA-scoped output markers for commit ffef948bcc5cf96f972456b9e11540cf49ae104d.

False positive or not applicable? A repository writer can comment:
/ai-review override fable ffef948bcc5cf96f972456b9e11540cf49ae104d false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of ffef948bcc5cf96f972456b9e11540cf49ae104d and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] ffef948

False positive or not applicable? A repository writer can comment:
/ai-review override gpt ffef948bcc5cf96f972456b9e11540cf49ae104d false-positive: <reason, 20 to 500 characters>

@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 5% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

Test-only fix for a port-reuse race in relay-client tests. A bound-but-never-listening socket keeps each port held so every call to it is refused. The fake lease service gets a reachable flag that drops calls while keeping its port. The dropped connection surfaces as an aiohttp ClientError, which lease.py:977 reports as "could not be reached", so every assertion still holds. make_client stays compatible with FakeRelay callers. No product code changes. Backend test shards were still pending at review time.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 964e9469eb · 115 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@bolichen97
bolichen97 enabled auto-merge (squash) November 13, 2026 14:51
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 13, 2026
@buluoray

Copy link
Copy Markdown
Contributor

Review passed on this head. Please rebase onto the latest main so CI can run fresh; the only red tests on this head come from main's shared conductor-parity failure, which main fixed after this run was created. The approval will follow once the rebased head is green.

Several relay-client tests made a fake server or a probe unreachable by
freeing its loopback port, then let the client call that port and
expected the call to fail. Under xdist another listener on the host
could take the freed port in between, so the call was answered (CI
saw a 404) instead of failing.

The fake relay and the fake lease service now drop every call in
place while they keep their port, and the relay dial test and the
lease tests use that. The unreachable-crew test keeps its probe
socket bound, never listening, for the whole test: the connect still
fails with an error whose text names the port, which is what the test
checks the relay is never sent. A bound socket is not used for the
dial and lease tests because macOS drops a connect to one until it
times out instead of refusing it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in test/test_relay_client_lease.py since this head ran CI (c22557c -> bdc7e7d); CI re-running.

The only reds on the previous head were Backend Tests shard 3 (Linux and Windows) on test/test_mate_conductor_parity.py::test_every_shared_rule_has_an_ultra_counterpart_or_a_named_gap, which was main's own red and is fixed on main by #36652; the new base carries that fix. The new commit also changes how two of the tests make their target unreachable: the dial test and the lease test that used a bound socket that never listens now keep the fake server up and have it drop every call, because macOS drops a connect to a bound, non-listening socket until it times out instead of refusing it. The forced-condition proof and the related tests were re-run on this exact commit.

[operator: bolichen97]

@bolichen97
bolichen97 force-pushed the fix/35371-held-refusing-port branch from 964e946 to ffef948 Compare November 17, 2026 23:05

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: action required A blocking check or review needs attention

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: test_a_dial_that_keeps_failing_says_so_in_status sees 'upgrade answered 404' where it expects 'dial failed'

3 participants