Repository navigation
test(instances): make three hop-guard cases hermetic on macOS - #14651
Conversation
The nightly's first macOS run of test_hop_port_guard.py and the hop ownership invariant cases (both new in #13936) failed on three platform facts the tests took for Linux ones. A stale client whose write lands after the guard's RST gets EPIPE on a BSD stack (sosend checks SS_CANTSENDMORE before so_error), where Linux hands that write the pending ECONNRESET. Both say the bytes never went out, so the reset assertion accepts either and still refuses ECONNREFUSED. A bound-but-not-listening SO_REUSEADDR socket already refuses an exact duplicate bind on macOS (that needs SO_REUSEPORT there), so the "bind alone does not hold" premise is Linux's; it is asserted on Linux only and the listen half runs everywhere. The give-up recovery case let the real diagnosis schedule `ssh c-host`, which asyncio.run then had to cancel on exit; on the macOS lane that cancel outran the 180s test timeout. The case stubs _schedule_diagnosis the way test_instances.py already does and asserts it was asked for.
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Confirm this PR's own macOS shards ran green at HEAD 7abdd1b — that run is the stated verification. [DESIGN-REVIEWED] 7abdd1b |
Opus 5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Problem / Motivation
Goal: The macOS nightly passes the hop-guard tests and the hop ownership invariant test every time, without loosening what they check.
The 2026-09-28 Nightly Build (macOS shard 2) failed three tests that landed in #13936 and had never run on a macOS lane before:
test_a_stale_client_is_reset_at_once_and_its_bytes_are_never_readECONNRESET, gotEPIPEfromsendalltest_binding_alone_would_not_have_held_itSO_REUSEADDRbind wasREFUSED 48, notBOUNDtest_it_holds_after_the_recovery_gives_upTimeout >180.0sinsideasyncio.run's exitWhy it matters
A red nightly on a shard nobody's PR touched is noise that hides the next real red, and the first of these looked like the guard failing to reset a stale client, which is the one property that keeps a chained crew's bearer token off a stranger's port. Each of the three is a platform fact the test mistook for a property of our code.
Not a goal
HopPortGuardor the tunnel manager's self-heal; the code under test is right on both platforms.ECONNREFUSED(a port never listened on) still fails it.What changed (motivation → approach → change)
Test-only. Three platform facts, one fix each.
ECONNRESETEPIPESO_REUSEADDRbind on a bound, unlistened portSO_REUSEPORT)BOUND/REFUSED), then the listen half everywheressh c-host_schedule_diagnosis, assert it was asked fortest_hop_port_guard.py: BSD'ssosendchecks its can't-send-more flag beforeso_error, so a write that lands after the RST reportsEPIPEwhere Linux hands it the pendingECONNRESET. Same fact, two spellings, and a client can read neither as a reply. The reset assertion takes both. The bind-alone case records a Linux reason for why the hold must listen; on macOS an exact duplicate bind is already refused beforelisten. The case pins each platform's own pre-listen outcome,BOUNDon Linux andREFUSEDelsewhere, so a kernel that started behaving like the other one fails here instead of passing on the listen half alone; then it checks that the listening shape refuses everywhere.test_remote_crew_chaining.py: the give-up path calls_schedule_diagnosis, and the real diagnosis spawnsssh c-hoston the runner.asyncio.runthen has to cancel that task on the way out, and a cancel that lands while the subprocess transport is still being set up is a wait the runner cannot bound; on the macOS lane it ran past the 180s test timeout. The invariant under test says nothing about the diagnosis, so the case stubs_schedule_diagnosisthe waytest_recover_attempt_cap_then_diagnoseintest_instances.pyalready does, and asserts the give-up path asked for one.Backwards compatibility
Compatible: no production code changes; two test files only.
Tests
test_a_stale_client_is_reset_at_once_and_its_bytes_are_never_readstill fails on a clean EOF, a hang past 10s, orECONNREFUSED; it now also passes when the reset reaches the write asEPIPE.test_binding_alone_would_not_have_held_itpins on Linux that bind alone does not hold, on other POSIX platforms that it already refuses, and everywhere that listening refuses.test_it_holds_after_the_recovery_gives_upgainsassert diagnoses == ["c"], so a give-up path that stopped scheduling a diagnosis would fail here instead of passing silently.Run locally on macOS:
test/test_hop_port_guard.py(13 passed) andtest/test_remote_crew_chaining.py(163 passed), serial-n 0. Before the change,test_binding_alone_would_not_have_held_itfailed on this Mac every run.Manual verification
N/A — the failing runner is the macOS CI lane; this PR's own macOS shards are the verification.
Related Issues
no linked issue: the failures were found in the nightly run linked above and no issue was filed for them.
Pattern harvest
Rule candidate: review-prompt
Pattern: a socket-level test asserts one errno or one bind outcome that is Linux behaviour, not POSIX; on a BSD stack the write after RST is
EPIPEand an exact duplicateSO_REUSEADDRbind is refused beforelisten.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)