Skip to content

test: pin the port probe in the remote-crew manager tests - #16199

Merged
bolichen97 merged 1 commit into
mainfrom
test/macos-flake-port-probe-spec
Oct 2, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
test/macos-flake-port-probe-spec

Conversation

@bolichen97

@bolichen97 bolichen97 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The remote-crew manager tests stop depending on the host's free ports, and the testing spec records today's three macOS flake shapes so new tests avoid them.

PR #16139's macOS run went red on test/test_instances.py::TestProxyRequest::test_remint_exchanges_the_new_link_at_once with assert False is True. The captured log says why: local port 53500 was taken while connecting. The test hands the manager a fixed base port and leaves its real loopback probe in place, so it passes only when nothing else on the runner grabs that port mid-connect.

Why it matters

The nightly's Platform Tests run this file on macOS. A red leg skips every publish job, so one flaky test here stops the nightly build from going out. The same shape fired twice more today (#16122, #16139), each found only after it reached someone else's PR.

Not a goal

What changed (motivation → approach → change)

Where What
test/test_instances.py TestProxyRequest and TestPeerRequestSharedDance pin the port probe like every other manager test class
test/test_instances.py new ratchet: a test class that connects a manager on the real probe fails, unless it is named with a reason
docs/system-specs/common/testing-conventions.md three rules: a fixed loopback port, http.server's name lookup, a process-wide verdict cache

The flake. connect() allocates a port, then probes it again for real before it starts the tunnel. Port 53500 sits in the ephemeral range, so any outgoing connect on a busy macOS runner can take it between the two probes. Then connect() returns an error status, and the remint test reads False. Nine other manager test classes already stub the probe with _patch_port_probe. These two were the only ones that did not.

Keeping it fixed. test_every_manager_connect_test_pins_the_port_probe reads every test file that builds a SshTunnelManager. It fails a class that connects a manager, or a module-level helper that builds one for its file to connect, unless the probe is pinned outside a single test_ method. A pin can sit in an autouse fixture, the manager helper, or a module helper such as test_remote_crew_chaining.py's _free_ports. The one class allowed to keep the real probe is TestOrphanForwarderReclaim, listed in _REAL_PORT_PROBE_CLASSES.

The spec. The testing conventions gain one rule per flake shape found on macOS today: pin a fixed real port or bind port 0 (this PR); skip getfqdn in a spawned http.server that a test waits on (#16122); reset a process-wide verdict cache a test patches behind (#16139).

Backwards compatibility

Compatible: test and doc change only; no production code is touched.

Tests

  • TestProxyRequest and TestPeerRequestSharedDance — unchanged assertions; the probe is pinned, so the result no longer depends on which ports the host holds.
  • test_every_manager_connect_test_pins_the_port_probe — new. It fails and names the node when either class loses its fixture, when test_remote_crew_chaining.py's _mgr stops calling _free_ports, or when TestOrphanForwarderReclaim leaves the allowlist.

Locally (macOS): test/test_instances.py and test/test_remote_crew_chaining.py (587 passed), plus isort, flake8, black, docs-lint and the comment-history, sync-io, brand and testpaths gates.

Manual verification

N/A — test-only change; the macOS shards on this head are the check. This PR carries ci:macos so they run.

Related Issues

no linked issue: found on PR #16139's hosted macOS run.

Pattern harvest

Rule candidate: lint
Pattern: a unit test drives code that probes a fixed real loopback port and leaves the probe real. Encoded here as the ratchet test for SshTunnelManager.connect.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested a review from a team as a code owner October 2, 2026 16:43
@bolichen97 bolichen97 added the ci:macos Run the on-demand macOS suite against this PR head label Oct 2, 2026
@bolichen97 bolichen97 added the ci:macos Run the on-demand macOS suite against this PR head label Oct 2, 2026
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 2, 2026
@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of a6d38462d5fa006343d77b0945b9cc311eb5257b — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Confirm the macOS shards on this head ran green — the ratchet's offender list is empty only if test_instances_fargate.py's pins sit outside test_ methods.

[DESIGN-REVIEWED] a6d3846

Description read

[DESCRIPTION-READ] 8600b0004838e35062f8b58e75082912f9750814bce7d2716c9d7e40e50bc4e8

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of a6d38462d5fa006343d77b0945b9cc311eb5257b and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] a6d3846

False positive or not applicable? A repository writer can comment:
/ai-review override gpt a6d38462d5fa006343d77b0945b9cc311eb5257b: <one-sentence reason>

@bolichen97

bolichen97 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=8496171c2206 Ratchet misses a fixture-built manager connected from another node

fixed in a6d3846: a module-level non-test helper that builds a SshTunnelManager in a file that connects one is now checked, and a pin reached through a module helper (test_remote_crew_chaining.py's _free_ports) counts.
Not widened to every .connect( in a file: that would match unrelated socket/aiohttp connects; the check keys on the manager's construction site instead.

@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed a6d38462d5fa006343d77b0945b9cc311eb5257b — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] a6d3846

Verdict parsed from the review's SHA-scoped output markers for commit a6d38462d5fa006343d77b0945b9cc311eb5257b.

False positive or not applicable? A repository writer can comment:
/ai-review override fable a6d38462d5fa006343d77b0945b9cc311eb5257b: <one-sentence reason>

TestProxyRequest and TestPeerRequestSharedDance hand SshTunnelManager a
fixed base port and leave connect()'s loopback probe real. On a busy
macOS runner that port (in the ephemeral range) is sometimes taken
between allocation and the re-probe, connect() returns an error status,
and the remint test reads False. Pin the probe like every other manager
test class, and add a ratchet that fails any class connecting a manager
on the real probe unless it is named with a reason.

Record the three macOS flake shapes fixed today in the testing spec:
a fixed real port, http.server's name lookup between bind and listen,
and a process-wide verdict cache that skips the code a test patched.
@bolichen97
bolichen97 force-pushed the test/macos-flake-port-probe-spec branch from aab0807 to a6d3846 Compare October 2, 2026 17:13
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=710ae5f8f56b Class-wide marker match let a test-local patch satisfy the ratchet

fixed in a6d3846: a class now counts as pinned only when a non-test_ method (autouse fixture or manager helper) pins the probe, so the _is_port_free patches inside TestOrphanForwarderReclaim's own test methods no longer exempt it.
Proven: deleting TestOrphanForwarderReclaim from _REAL_PORT_PROBE_CLASSES now fails the ratchet and names that class.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=7980bbae993b Ratchet exempted the allowlisted class by accident and never inspected module-level manager builders

fixed in a6d3846: pins count only outside test_ methods, so the allowlist entry is live; and a module-level non-test helper that builds a SshTunnelManager in a file that connects one must pin the probe itself or call a module helper that does (e.g. _free_ports).
Proven by mutation: dropping _free_ports(monkeypatch) from test_remote_crew_chaining.py::_mgr fails the ratchet naming _mgr; dropping the allowlist entry fails it naming TestOrphanForwarderReclaim.
The "either namespace" point is not a flake source: a class that stubs only the manager re-probe leaves the allocator real, which only walks past held ports; the re-probe is what turned a lost race into an error status.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 2, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) October 2, 2026 17:52

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary. TestProxyRequest and TestPeerRequestSharedDance now pin connect()'s loopback probe with _patch_port_probe. A source-scanning ratchet stops future manager-connecting classes and module helpers from relying on the host's real port probe. Test and spec only, no production change. The pin removes the flake's cause, and the pinned tests still go red when the code under test regresses.

What I verified (at a6d3846, worktree-loaded kiro_crew)

  • Root cause reproduced. SshTunnelManager.connect re-probes the allocated port (ssh_tunnel_manager.py:3524) and returns local port N was taken while connecting when it loses the race. I simulated that TOCTOU loss: a listener held 127.0.0.1:53500 and a plugin let the allocator see it free while the manager's re-probe stayed real. On origin/main's test file, 16 of 24 TestProxyRequest/TestPeerRequestSharedDance tests failed, test_remint_exchanges_the_new_link_at_once and test_401_gets_exactly_one_remint_retry among them. With this PR's file under the same conditions, 24/24 pass. Holding the port without the race (allocator real too) passes on both: the allocator just walks to 53501. That matches the PR's claim that the race between the two probes, not plain occupancy, is the trigger.
  • The pin does not hide regressions in the code under test. Disabling the single re-mint retry in proxy_request (ssh_tunnel_manager.py:4909) reddens TestProxyRequest::test_401_gets_exactly_one_remint_retry, test_failed_exchange_sends_no_link_and_remints and test_refused_exchange_still_gets_exactly_one_remint[401]. The conflict path the pin bypasses keeps its own coverage: removing the re-probe check at :3524 reddens TestPortMirror::test_port_conflict_hard_fails, which drives manager_free=False.
  • Ratchet mutations (each restored, git status --short clean at the end):
    • removing TestProxyRequest's pin → ratchet fails naming test_instances.py::TestProxyRequest
    • removing TestPeerRequestSharedDance's pin → fails naming test_instances.py::TestPeerRequestSharedDance
    • removing TestOrphanForwarderReclaim from _REAL_PORT_PROBE_CLASSES → fails naming it (so the allowlist entry is doing real work)
    • removing _free_ports(monkeypatch) from test_remote_crew_chaining.py::_mgr → fails naming test_remote_crew_chaining.py::_mgr
  • test/test_instances.py passes in full locally (426). Head CI: 86 success, 11 skipped, 0 failing.
  • Spec. The other two new testing-conventions entries cite symbols that exist on this head: cron_script._shell_is_posix_strict, _probe_blip in test_sandbox_interpreter_enoent_retry.py, the server_bind stand-ins in test_decisions_local_runtime_real_spawn.py and decisions/local_servers/plumb_cpu.py. The commit message and PR body are accurate.

Blocking: none.

Follow-ups (non-blocking)

  • The pinned tests still touch the real loopback port. Once the probe reports 53500 free, connect() proceeds to its connect-time link exchange against 127.0.0.1:53500 before the test patches _exchange_link. With a foreign listener that accepts and never answers, each test still passes but takes ~11s (3 tests at 11s, 57s for the two classes, vs about 4s otherwise). On a runner where 53500 is only an ephemeral client port the connect is refused and fast, so this is a slowness risk, not a correctness one. A later hardening could stub the connect-time exchange in _mgr, or bind port 0 for base_port.
  • The ratchet's pin detection is textual ("_patch_port_probe(" or '"_is_port_free", lambda' in a non-test_ method). A non-test helper that mentions the call but is never invoked would count as a pin. This is fine as a ratchet against the common mistake, but it is not proof.

Bot-concern dispositions

  • Design Review PASS, note "confirm macOS shards green; offender list empty only if test_instances_fargate.py's pins sit outside test_ methods": verified. The ratchet scans every test_*.py and passes on this head, and head CI has no failures.
  • GPT 5.6 and Opus 5: no findings on this head. The author's prior-round dispositions (span 710ae5f8f56b: class-wide marker match; span 7980bbae993b: allowlist accidentally exempted plus module builders unchecked; design span 8496171c2206: fixture-built manager missed) are verified fixed by the allowlist and _mgr mutations above. The Opus "either namespace" rebuttal holds: stubbing only the manager re-probe leaves the allocator walking past held ports, which never produces an error status.

@bolichen97
bolichen97 added this pull request to the merge queue Oct 2, 2026
Merged via the queue into main with commit a7249fb Oct 2, 2026
100 checks passed
@bolichen97
bolichen97 deleted the test/macos-flake-port-probe-spec branch October 2, 2026 18:49
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:macos Run the on-demand macOS suite against this PR head

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants