Repository navigation
test: pin the port probe in the remote-crew manager tests - #16199
Conversation
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Confirm the macOS shards on this head ran green — the ratchet's offender list is empty only if [DESIGN-REVIEWED] a6d3846 Description read[DESCRIPTION-READ] 8600b0004838e35062f8b58e75082912f9750814bce7d2716c9d7e40e50bc4e8 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
|
Opus 5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
TestProxyRequest and TestPeerRequestSharedDance hand SshTunnelManager a fixed base port and leave connect()'s loopback probe real. On a busy macOS runner that port (in the ephemeral range) is sometimes taken between allocation and the re-probe, connect() returns an error status, and the remint test reads False. Pin the probe like every other manager test class, and add a ratchet that fails any class connecting a manager on the real probe unless it is named with a reason. Record the three macOS flake shapes fixed today in the testing spec: a fixed real port, http.server's name lookup between bind and listen, and a process-wide verdict cache that skips the code a test patched.
aab0807 to
a6d3846
Compare
|
|
buluoray
left a comment
There was a problem hiding this comment.
Summary. TestProxyRequest and TestPeerRequestSharedDance now pin connect()'s loopback probe with _patch_port_probe. A source-scanning ratchet stops future manager-connecting classes and module helpers from relying on the host's real port probe. Test and spec only, no production change. The pin removes the flake's cause, and the pinned tests still go red when the code under test regresses.
What I verified (at a6d3846, worktree-loaded kiro_crew)
- Root cause reproduced.
SshTunnelManager.connectre-probes the allocated port (ssh_tunnel_manager.py:3524) and returnslocal port N was taken while connectingwhen it loses the race. I simulated that TOCTOU loss: a listener held 127.0.0.1:53500 and a plugin let the allocator see it free while the manager's re-probe stayed real. Onorigin/main's test file, 16 of 24TestProxyRequest/TestPeerRequestSharedDancetests failed,test_remint_exchanges_the_new_link_at_onceandtest_401_gets_exactly_one_remint_retryamong them. With this PR's file under the same conditions, 24/24 pass. Holding the port without the race (allocator real too) passes on both: the allocator just walks to 53501. That matches the PR's claim that the race between the two probes, not plain occupancy, is the trigger. - The pin does not hide regressions in the code under test. Disabling the single re-mint retry in
proxy_request(ssh_tunnel_manager.py:4909) reddensTestProxyRequest::test_401_gets_exactly_one_remint_retry,test_failed_exchange_sends_no_link_and_remintsandtest_refused_exchange_still_gets_exactly_one_remint[401]. The conflict path the pin bypasses keeps its own coverage: removing the re-probe check at:3524reddensTestPortMirror::test_port_conflict_hard_fails, which drivesmanager_free=False. - Ratchet mutations (each restored,
git status --shortclean at the end):- removing
TestProxyRequest's pin → ratchet fails namingtest_instances.py::TestProxyRequest - removing
TestPeerRequestSharedDance's pin → fails namingtest_instances.py::TestPeerRequestSharedDance - removing
TestOrphanForwarderReclaimfrom_REAL_PORT_PROBE_CLASSES→ fails naming it (so the allowlist entry is doing real work) - removing
_free_ports(monkeypatch)fromtest_remote_crew_chaining.py::_mgr→ fails namingtest_remote_crew_chaining.py::_mgr
- removing
test/test_instances.pypasses in full locally (426). Head CI: 86 success, 11 skipped, 0 failing.- Spec. The other two new testing-conventions entries cite symbols that exist on this head:
cron_script._shell_is_posix_strict,_probe_blipintest_sandbox_interpreter_enoent_retry.py, theserver_bindstand-ins intest_decisions_local_runtime_real_spawn.pyanddecisions/local_servers/plumb_cpu.py. The commit message and PR body are accurate.
Blocking: none.
Follow-ups (non-blocking)
- The pinned tests still touch the real loopback port. Once the probe reports 53500 free,
connect()proceeds to its connect-time link exchange against127.0.0.1:53500before the test patches_exchange_link. With a foreign listener that accepts and never answers, each test still passes but takes ~11s (3 tests at 11s, 57s for the two classes, vs about 4s otherwise). On a runner where 53500 is only an ephemeral client port the connect is refused and fast, so this is a slowness risk, not a correctness one. A later hardening could stub the connect-time exchange in_mgr, or bind port 0 forbase_port. - The ratchet's pin detection is textual (
"_patch_port_probe("or'"_is_port_free", lambda'in a non-test_method). A non-test helper that mentions the call but is never invoked would count as a pin. This is fine as a ratchet against the common mistake, but it is not proof.
Bot-concern dispositions
- Design Review PASS, note "confirm macOS shards green; offender list empty only if
test_instances_fargate.py's pins sit outsidetest_methods": verified. The ratchet scans everytest_*.pyand passes on this head, and head CI has no failures. - GPT 5.6 and Opus 5: no findings on this head. The author's prior-round dispositions (span 710ae5f8f56b: class-wide marker match; span 7980bbae993b: allowlist accidentally exempted plus module builders unchecked; design span 8496171c2206: fixture-built manager missed) are verified fixed by the allowlist and
_mgrmutations above. The Opus "either namespace" rebuttal holds: stubbing only the manager re-probe leaves the allocator walking past held ports, which never produces an error status.
Problem / Motivation
Goal: The remote-crew manager tests stop depending on the host's free ports, and the testing spec records today's three macOS flake shapes so new tests avoid them.
PR #16139's macOS run went red on
test/test_instances.py::TestProxyRequest::test_remint_exchanges_the_new_link_at_oncewithassert False is True. The captured log says why:local port 53500 was taken while connecting. The test hands the manager a fixed base port and leaves its real loopback probe in place, so it passes only when nothing else on the runner grabs that port mid-connect.Why it matters
The nightly's Platform Tests run this file on macOS. A red leg skips every publish job, so one flaky test here stops the nightly build from going out. The same shape fired twice more today (#16122, #16139), each found only after it reached someone else's PR.
Not a goal
SshTunnelManagerorPortAllocator. Production should keep probing real ports.TestOrphanForwarderReclaim, whose subject is a real port held by a real process.What changed (motivation → approach → change)
test/test_instances.pyTestProxyRequestandTestPeerRequestSharedDancepin the port probe like every other manager test classtest/test_instances.pydocs/system-specs/common/testing-conventions.mdhttp.server's name lookup, a process-wide verdict cacheThe flake.
connect()allocates a port, then probes it again for real before it starts the tunnel. Port 53500 sits in the ephemeral range, so any outgoing connect on a busy macOS runner can take it between the two probes. Thenconnect()returns an error status, and the remint test readsFalse. Nine other manager test classes already stub the probe with_patch_port_probe. These two were the only ones that did not.Keeping it fixed.
test_every_manager_connect_test_pins_the_port_probereads every test file that builds aSshTunnelManager. It fails a class that connects a manager, or a module-level helper that builds one for its file to connect, unless the probe is pinned outside a singletest_method. A pin can sit in an autouse fixture, the manager helper, or a module helper such astest_remote_crew_chaining.py's_free_ports. The one class allowed to keep the real probe isTestOrphanForwarderReclaim, listed in_REAL_PORT_PROBE_CLASSES.The spec. The testing conventions gain one rule per flake shape found on macOS today: pin a fixed real port or bind port 0 (this PR); skip
getfqdnin a spawnedhttp.serverthat a test waits on (#16122); reset a process-wide verdict cache a test patches behind (#16139).Backwards compatibility
Compatible: test and doc change only; no production code is touched.
Tests
TestProxyRequestandTestPeerRequestSharedDance— unchanged assertions; the probe is pinned, so the result no longer depends on which ports the host holds.test_every_manager_connect_test_pins_the_port_probe— new. It fails and names the node when either class loses its fixture, whentest_remote_crew_chaining.py's_mgrstops calling_free_ports, or whenTestOrphanForwarderReclaimleaves the allowlist.Locally (macOS):
test/test_instances.pyandtest/test_remote_crew_chaining.py(587 passed), plus isort, flake8, black, docs-lint and the comment-history, sync-io, brand and testpaths gates.Manual verification
N/A — test-only change; the macOS shards on this head are the check. This PR carries
ci:macosso they run.Related Issues
no linked issue: found on PR #16139's hosted macOS run.
Pattern harvest
Rule candidate: lint
Pattern: a unit test drives code that probes a fixed real loopback port and leaves the probe real. Encoded here as the ratchet test for
SshTunnelManager.connect.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)