Skip to content

test: refuse fake tunnels' peer and health connects at once - #39236

Merged
iamwhatever merged 2 commits into
mainfrom
crew/testoptimizer/windows-closed-port
Nov 19, 2026
Merged

iamwhatever merged 2 commits into
mainfrom
crew/testoptimizer/windows-closed-port

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The instance-tunnel and app-off tests stop waiting on Windows' closed-port SYN retry, with every check unchanged.

The tests' fake tunnels bind nothing. After each successful SshTunnelManager.connect(), the manager primes the peer session (_exchange_link, an aiohttp GET to 127.0.0.1:<local_port>), and that GET meets a closed port. Linux refuses it in under a millisecond. Windows retries the SYN and refuses after about 2 s, once per prime. On main that is 187 refused connects across 152 tests in 17 files. test_app_off_contract.py does the same through the app health probe: 16 tests dial port 9301, which nothing serves.

Why it matters

The Windows shards that carry these files are among the slowest, and every PR waits on them. The closed-port waits come to about 320 s of worker time for the peer prime and about 32 s for the health probe (closed-port scan of main run 44329986204).

Not a goal

  • No change under src/ or to test/conftest.py.
  • No change to what any test asserts or how many tests there are.
  • Tests whose subject is a refused connect keep the real one: test_instances_orphan_reclaim.py's _forwarder_listener_gone and test_remote_crew_chaining_credential_destination.py::test_an_unreachable_port_is_no_answer (which goes through ssh_tunnel_manager_transport, not the peer module).
  • The global aiohttp module is not touched: real aiohttp servers run in test_instances_proxy.py and test_remote_crew_chaining_credential_destination.py.

What changed (motivation → approach → change)

Closed port on Windows → about 2 s per connect → fail that connect at once, in the same way, so the product code takes the same path.

  • New test/closed_peer_helpers.py. refused_peer_connects replaces aiohttp for the ssh_tunnel_manager family only, through the facade, whose mirror rebinds it in every owner that holds it (_child, _peer, _transport); patching one owner directly is what test_facade_mirror_guards.py forbids, because the facade's undo would restore the other owners to that owner's value. The replacement is a view that resolves every name to the real module except ClientSession. While aiohttp.ClientSession is the real class, the view returns a session whose get / post / request raise aiohttp.ClientConnectorError built the way aiohttp's connector builds it: a real ConnectionKey for the URL's host and port, and ConnectionRefusedError(ECONNREFUSED, "Connect call failed (host, port)"). Its str() and repr() match a real refused connect byte for byte (checked against one locally), and the fixture asserts that str() renders before it patches anything. So _exchange_link's and token_validates' own except branches still run and log Peer link exchange failed (ClientConnectorError) as before. A test that already replaced aiohttp.ClientSession with its own fake keeps that fake, because the view passes any non-real class through.
  • The 17 peer files use it with pytestmark = pytest.mark.usefixtures("refused_peer_connects"). Four tests opt back out with @pytest.mark.usefixtures("real_peer_connects"): the two test_instances_proxy.py tests that call _exchange_link against a real aiohttp server, test_the_crews_id_comes_back_through_two_hops (a real server through _transport), and test_an_unreachable_port_is_no_answer, whose subject is a real refused connect.
  • test_app_off_contract.py: an autouse fixture (patching through _floor_monkeypatch, so a test's monkeypatch.undo() cannot lift it) wraps loopback_http.build_loopback_opener so the opener's open raises URLError(ConnectionRefusedError(ECONNREFUSED)) for port 9301 only. _health_probe and loopback_urlopen still run and classify the same error. Tests that patch _health_probe never reach it.
  • Separate commit: test_remote_crew_chaining_chained_refresh_and_repair.py assigned _FakeMintSession.reply, a class attribute, and left it at a token of "T". test_remote_crew_chaining.py::TestChainedToken::test_the_mint_asks_the_parent_by_the_parents_own_id_not_ours then failed in any process that ran it later. A serial run of the 18 files on main fails that test every time. The reply is now set through monkeypatch, which puts it back at teardown.

Rejected: patching the global aiohttp.ClientSession (it would break the real servers above), and having the fake tunnels bind a real listener (their ports are fixed and would collide across workers).

Backwards compatibility

Compatible: test files only.

Removes nothing: every test and assertion stays; only the fake tunnels' closed-port connects fail sooner.

Tests

All 18 files, Linux, same host, main 0d68cae9e6 against this branch:

main branch
tests, -n0 with coverage 1 failed, 643 passed 644 passed
refused loopback connects (probe plugin) 187 in 152 tests 5 in 4 tests (the deliberate ones above)
src/kiro_crew lines / branches covered, -n0 --cov-branch 126,327 / 4,023 126,327 / 4,023

Every src file has identical executed lines and branches. One line moves between runs on main itself, and it is not caused by this change: subprocess_pool/executor.py:372, the return False when the pool's background reaper thread finds a slot's lock already held. Whether that happens depends on thread timing. Main hit it in one of its three runs (one -n 4, two -n0) and missed it in the other two; the table shows main's second -n0 run.

Plants (each reverted afterwards), all 18 files at -n 4:

  • _exchange_link returning a value instead of raising _PeerUnavailable from its except: the same 5 test_instances_proxy.py tests fail on main and on this branch.
  • token_validates returning True from its except: test_instances.py::TestSshTunnelManager::test_token_validates_denies_on_error fails on both.

Main also fails the leaked-reply test above in both plant runs. That test passes on the branch.

Three runs at -n 4 --dist loadgroup and one in reversed order at -n0: 644 passed each time. test_flake_pattern_ratchet.py, comment-history, brand, subprocess-encoding, black, flake8, isort and file-line-cap gates pass.

Proof: not provable -- test-only change

Manual verification

N/A — unit coverage sufficient: test-only change; the counts, coverage and plants above were run locally.

Real-machine verification

N/A -- test-only change; Windows timing is measured by CI's Windows shards.

Related Issues

Part of #30257

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners November 18, 2026 20:08
@iamwhatever
iamwhatever enabled auto-merge (squash) November 18, 2026 20:09
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision aac861ca42ade4deb1d1bd2293a750b3af72c642 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision aac861ca42ade4deb1d1bd2293a750b3af72c642 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of aac861ca42ade4deb1d1bd2293a750b3af72c642 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

This is a test-only change: it fails the fake tunnels' connects at once without touching the product code or any assertion, and it takes nothing away.

Notes

  • The opt-in is per file, so a new test file that uses fake tunnels has to remember to add pytestmark = pytest.mark.usefixtures("refused_peer_connects"). If it forgets, it slows down again on Windows without failing. A fixture scoped to the tunnel helper (instances_helpers and remote_crew_chaining_helpers) would apply it everywhere they are used, so nobody has to remember.

[DESIGN-REVIEWED] aac861c

Description read

[DESCRIPTION-READ] 41e94fbfb69c6bcaad231ac854955eee31173a06cf5f341917438eb55a354e70

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed aac861ca42ade4deb1d1bd2293a750b3af72c642 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] aac861c

Verdict parsed from the review's SHA-scoped output markers for commit aac861ca42ade4deb1d1bd2293a750b3af72c642.

False positive or not applicable? A repository writer can comment:
/ai-review override fable aac861ca42ade4deb1d1bd2293a750b3af72c642 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of aac861ca42ade4deb1d1bd2293a750b3af72c642 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] aac861c

False positive or not applicable? A repository writer can comment:
/ai-review override gpt aac861ca42ade4deb1d1bd2293a750b3af72c642 false-positive: <reason, 20 to 500 characters>

@iamwhatever
iamwhatever disabled auto-merge November 18, 2026 20:52
@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Nov 18, 2026
@iamwhatever
iamwhatever force-pushed the crew/testoptimizer/windows-closed-port branch from 65aa36f to 86c58b4 Compare November 18, 2026 21:09
@iamwhatever
iamwhatever enabled auto-merge (squash) November 18, 2026 21:09
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Backend Tests (3.12, 8) red on 86c58b4 is main's, not this diff: test/test_bare_session_handle_guard.py::test_no_test_builds_a_bare_session_handle_directly flags test_kas_token_refresh_recovery.py:245 (#38978 and #38996 crossed on main). Tracked in #39261; fix PRs #39262 / #39274. This PR will rebase onto main once that fix lands. The other three reds (Analyze (actions), Dependency License Gate, Publish readiness signal) are GitHub API rate limits.

@iamwhatever
iamwhatever disabled auto-merge November 18, 2026 21:52
zejiangg added 2 commits November 18, 2026 21:53
test_remote_crew_chaining_chained_refresh_and_repair.py assigned
_FakeMintSession.reply, a class attribute, and left it at a token of
"T". A later test in the same process that reads the default reply,
TestChainedToken::test_the_mint_asks_the_parent_by_the_parents_own_id_not_ours,
then failed. Serial runs of both files on main fail that way every time.
Setting the reply through monkeypatch restores it at teardown.
Fake tunnels bind nothing, so the peer-session prime after each
connect (_exchange_link) dials a closed loopback port. Linux refuses
that in under a millisecond; Windows retries the SYN and refuses after
about 2 s, once per prime, across 152 tests in 17 files.

closed_peer_helpers.refused_peer_connects replaces aiohttp only inside
ssh_tunnel_manager_peer with a view whose ClientSession raises the same
ClientConnectorError a refused connect raises, so the product's except
branches still run. A test that swapped ClientSession itself keeps its
stand-in, and the two tests that dial a real aiohttp server opt back
out with real_peer_connects.

test_app_off_contract.py refuses the health probe's connect to its
unserved port 9301 at the opener, so probe and loopback_urlopen still
run.
@iamwhatever
iamwhatever force-pushed the crew/testoptimizer/windows-closed-port branch from 86c58b4 to aac861c Compare November 18, 2026 21:55
@iamwhatever
iamwhatever enabled auto-merge (squash) November 18, 2026 21:55
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 18, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. This review checked the following on aac861c:

  • The aiohttp patch goes through the ssh_tunnel_manager facade mirror. One write rebinds the child, peer and transport owners, and teardown restores all three.
  • The refusing session covers every ClientSession call in that family (get, post, awaited request, close, async with). It raises a real ClientConnectorError with ECONNREFUSED, so the product's except branches and the redial check run as before.
  • The four tests that need a real server or a real closed port opt out with real_peer_connects.
  • The build_loopback_opener patch in test_app_off_contract takes effect at call time.
  • The _FakeMintSession.reply fix restores the class default at teardown, and no other test leaks that attribute.
  • No assertion is loosened, and the PR adds no sleep, clock, port literal or environment write.

Done well: the PR measures its claims. It shows the same src line and branch coverage on both sides, a count of refused connects, and planted-bug runs that fail the same tests on main and on the branch.

Optional:

  • Opt-in is per file, so a new test file that uses _FakeTunnel without the pytestmark line goes back to the slow connect without failing. A reply to the Design note could explain why a fixture in a helper module would still need an import in each file.

@iamwhatever
iamwhatever merged commit 9ac9383 into main Nov 19, 2026
101 of 102 checks passed
@iamwhatever
iamwhatever deleted the crew/testoptimizer/windows-closed-port branch November 19, 2026 01:35
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants