Skip to content

test(instances): make three hop-guard cases hermetic on macOS - #14651

Merged
bolichen97 merged 1 commit into
mainfrom
fix/macos-flaky-hop-guard-chaining
Sep 28, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/macos-flaky-hop-guard-chaining

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The macOS nightly passes the hop-guard tests and the hop ownership invariant test every time, without loosening what they check.

The 2026-09-28 Nightly Build (macOS shard 2) failed three tests that landed in #13936 and had never run on a macOS lane before:

Test Failure
test_a_stale_client_is_reset_at_once_and_its_bytes_are_never_read expected ECONNRESET, got EPIPE from sendall
test_binding_alone_would_not_have_held_it a second SO_REUSEADDR bind was REFUSED 48, not BOUND
test_it_holds_after_the_recovery_gives_up Timeout >180.0s inside asyncio.run's exit

Why it matters

A red nightly on a shard nobody's PR touched is noise that hides the next real red, and the first of these looked like the guard failing to reset a stale client, which is the one property that keeps a chained crew's bearer token off a stranger's port. Each of the three is a platform fact the test mistook for a property of our code.

Not a goal

  • No change to HopPortGuard or the tunnel manager's self-heal; the code under test is right on both platforms.
  • No skip of a whole test on macOS; every case still runs there.
  • No change to what the reset assertion refuses: ECONNREFUSED (a port never listened on) still fails it.

What changed (motivation → approach → change)

Test-only. Three platform facts, one fix each.

Case Linux macOS (BSD stack) Fix
write after the guard's RST ECONNRESET EPIPE accept either; both mean the bytes never left
second SO_REUSEADDR bind on a bound, unlistened port allowed refused (needs SO_REUSEPORT) pin each platform's own outcome (BOUND / REFUSED), then the listen half everywhere
give-up path schedules a diagnosis spawns ssh c-host same, and the exit cancel hung stub _schedule_diagnosis, assert it was asked for

test_hop_port_guard.py: BSD's sosend checks its can't-send-more flag before so_error, so a write that lands after the RST reports EPIPE where Linux hands it the pending ECONNRESET. Same fact, two spellings, and a client can read neither as a reply. The reset assertion takes both. The bind-alone case records a Linux reason for why the hold must listen; on macOS an exact duplicate bind is already refused before listen. The case pins each platform's own pre-listen outcome, BOUND on Linux and REFUSED elsewhere, so a kernel that started behaving like the other one fails here instead of passing on the listen half alone; then it checks that the listening shape refuses everywhere.

test_remote_crew_chaining.py: the give-up path calls _schedule_diagnosis, and the real diagnosis spawns ssh c-host on the runner. asyncio.run then has to cancel that task on the way out, and a cancel that lands while the subprocess transport is still being set up is a wait the runner cannot bound; on the macOS lane it ran past the 180s test timeout. The invariant under test says nothing about the diagnosis, so the case stubs _schedule_diagnosis the way test_recover_attempt_cap_then_diagnose in test_instances.py already does, and asserts the give-up path asked for one.

Backwards compatibility

Compatible: no production code changes; two test files only.

Tests

  • test_a_stale_client_is_reset_at_once_and_its_bytes_are_never_read still fails on a clean EOF, a hang past 10s, or ECONNREFUSED; it now also passes when the reset reaches the write as EPIPE.
  • test_binding_alone_would_not_have_held_it pins on Linux that bind alone does not hold, on other POSIX platforms that it already refuses, and everywhere that listening refuses.
  • test_it_holds_after_the_recovery_gives_up gains assert diagnoses == ["c"], so a give-up path that stopped scheduling a diagnosis would fail here instead of passing silently.

Run locally on macOS: test/test_hop_port_guard.py (13 passed) and test/test_remote_crew_chaining.py (163 passed), serial -n 0. Before the change, test_binding_alone_would_not_have_held_it failed on this Mac every run.

Manual verification

N/A — the failing runner is the macOS CI lane; this PR's own macOS shards are the verification.

Related Issues

no linked issue: the failures were found in the nightly run linked above and no issue was filed for them.

Pattern harvest

Rule candidate: review-prompt
Pattern: a socket-level test asserts one errno or one bind outcome that is Linux behaviour, not POSIX; on a BSD stack the write after RST is EPIPE and an exact duplicate SO_REUSEADDR bind is refused before listen.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

The nightly's first macOS run of test_hop_port_guard.py and the hop
ownership invariant cases (both new in #13936) failed on three platform
facts the tests took for Linux ones.

A stale client whose write lands after the guard's RST gets EPIPE on a
BSD stack (sosend checks SS_CANTSENDMORE before so_error), where Linux
hands that write the pending ECONNRESET. Both say the bytes never went
out, so the reset assertion accepts either and still refuses
ECONNREFUSED.

A bound-but-not-listening SO_REUSEADDR socket already refuses an exact
duplicate bind on macOS (that needs SO_REUSEPORT there), so the
"bind alone does not hold" premise is Linux's; it is asserted on Linux
only and the listen half runs everywhere.

The give-up recovery case let the real diagnosis schedule `ssh c-host`,
which asyncio.run then had to cancel on exit; on the macOS lane that
cancel outran the 180s test timeout. The case stubs _schedule_diagnosis
the way test_instances.py already does and asserts it was asked for.
@bolichen97
bolichen97 requested a review from a team as a code owner September 28, 2026 07:24
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 28, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 7abdd1bf7985315a8b7def5bdc51378713aa8195 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Confirm this PR's own macOS shards ran green at HEAD 7abdd1b — that run is the stated verification.

[DESIGN-REVIEWED] 7abdd1b

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 7abdd1bf7985315a8b7def5bdc51378713aa8195 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 7abdd1b

Verdict parsed from the review's SHA-scoped output markers for commit 7abdd1bf7985315a8b7def5bdc51378713aa8195.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 7abdd1bf7985315a8b7def5bdc51378713aa8195: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Sep 28, 2026
@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 7abdd1bf7985315a8b7def5bdc51378713aa8195 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 7abdd1b

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 7abdd1bf7985315a8b7def5bdc51378713aa8195: <one-sentence reason>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 28, 2026
@bolichen97
bolichen97 merged commit 35fac20 into main Sep 28, 2026
96 of 97 checks passed
@bolichen97
bolichen97 deleted the fix/macos-flaky-hop-guard-chaining branch September 28, 2026 08:06
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant