Skip to content

fix(instances): connect remote crews when an ssh_config forward is taken - #35407

Merged
bolichen97 merged 1 commit into
mainfrom
fix/34734-ssh-config-forwards
Nov 12, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/34734-ssh-config-forwards

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A Remote Crew SSH tunnel connects even when the host's ~/.ssh/config block adds LocalForward ports that another ssh session already holds.

Connecting a Remote Crew to a host whose ssh_config block has LocalForward lines fails while a terminal ssh to the same host is open. The dashboard shows ssh forward bind failed (local port already in use): bind [127.0.0.1]:<port>: Address already in use, for a port Kiro Crew never asked for, and ExitOnForwardFailure no in the user's config does not help.

Why it matters

A terminal session open to a box next to its Remote Crew connection is the normal workflow, and LocalForward lines in a host block are common. Every connect to that host fails until the user closes their own session or edits their ssh_config.

Not a goal

  • Dropping the host block's inherited forwards: ClearAllForwardings=yes also clears the tunnel's own -L (checked with ssh -G on OpenSSH 9.9).
  • Changing how the tunnel's own port is chosen, held or recovered.
  • The SSM transport, and the user-run launchd tunnel in the remote-and-mobile guide.

What changed (motivation → approach → change)

A forward that cannot bind Before After
one the host's ssh_config block adds on another port ssh exits, the connect fails a warning on stderr, the tunnel connects
the tunnel's own -L ssh exits; the connect fails, or a tunnel already up goes to self-heal, with ssh forward bind failed the gateway ends ssh; the same outcome and error

The tunnel is ssh -N -L ... and pinned ExitOnForwardFailure=yes. ssh opens every LocalForward the host's ssh_config block names besides it, and a command-line pin overrides the user's own setting, so one of those ports already taken made ssh exit.

The tunnel runs with ExitOnForwardFailure=no, which treats such a forward the way the user's own sessions do. Its own forward failing still ends the child: ssh prints cannot listen to port: <n> for each forward it could not bind, and the stderr drain reads every complete line as it arrives and stops ssh when <n> is the tunnel's own port, at connect or any time later, so the readiness wait or the exit monitor reports it exactly as ssh's own exit was. LogLevel=INFO is pinned so a user's LogLevel QUIET cannot hide that line. The drain remembers that it saw the tunnel's own line, and the exit classifier labels the exit from that memory before it reads any other phrase in the tail, such as an auth word in a server banner. A burst of other forwards' lines that pushes it out of the kept 2,000-character stderr tail cannot hide it. A line cut mid-number at a read boundary is not read as the tunnel's port, and a line about another port never labels a later exit as a bind failure. The instances spec (§7, §9) and the comments that named the old flag in instances/ say so.

Backwards compatibility

Compatible: the tunnel's own bind failure still fails the connect with the same ssh forward bind failed (local port already in use) text; only a failure on a forward the user's ssh_config adds on another port stops ending the tunnel; one on the tunnel's own port number still reads as the tunnel's own, as every taken forward did before.
Removes nothing: no option, key or message is taken away. A forwarder left running by an older gateway whose record has no argv fingerprint no longer matches the rebuilt argv, so the orphan reclaim leaves it and its port alone, as after any earlier argv change.

Tests

  • test/test_instances.py::TestSshTunnelMultiplexing::test_per_host_ssh_config_is_still_inherited: a host block with LocalForward 8888 localhost:8888, ExitOnForwardFailure yes and LogLevel QUIET; the real ssh -G resolves the tunnel's argv to exitonforwardfailure no and loglevel INFO with both forwards present.
  • test/test_ssh_tunnel_manager_cov80.py::TestStderrIsDrainedWhileRunning::test_a_failed_start_still_reports_its_stderr: the spawned stand-in does what ssh does under no when the tunnel's own port is taken (prints the two bind lines naming that port and keeps running); start() still fails with ssh forward bind failed.
  • test/test_ssh_tunnel_manager_cov80.py::TestStderrIsDrainedWhileRunning::test_a_bind_lost_after_the_start_ends_the_child: another process already listens on the port, so the tunnel comes up first; the stand-in then reports its own port unbindable and the tunnel goes to ERROR with ssh forward bind failed and fires on_exit.
  • test/test_ssh_tunnel_manager_cov80.py::TestAForwardTheSshConfigAdds::test_its_bind_failure_is_not_the_tunnels: a bind failure on port 8888 does not stop the child and leaves a later exit unlabelled; the same line for the tunnel's own port, read by the drain, labels it.
  • test/test_ssh_tunnel_manager_cov80.py::TestAForwardTheSshConfigAdds::test_the_tunnels_failure_counts_after_the_tail_drops_it: one read carries the tunnel's own line and then 20 failures for port 8888, enough to push it out of the stderr tail; the child is still stopped and the exit reads ssh forward bind failed.
  • test/test_ssh_tunnel_manager_cov80.py::TestAForwardTheSshConfigAdds::test_a_line_cut_at_the_tunnels_port_is_not_the_tunnels: a tunnel on port 5400 gets cannot listen to port: 5400 in one read and 7 in the next; the child keeps running and the exit is not a bind failure.
  • test/test_ssh_tunnel_manager_cov80.py::TestAForwardTheSshConfigAdds::test_the_tunnels_failure_outranks_a_banner_phrase: a banner line with Permission denied comes before the tunnel's own bind line; the exit still reads ssh forward bind failed, not an auth failure.

Proof: PROVEN test/test_instances.py::TestSshTunnelMultiplexing::test_per_host_ssh_config_is_still_inherited, test/test_ssh_tunnel_manager_cov80.py::TestStderrIsDrainedWhileRunning::test_a_failed_start_still_reports_its_stderr, test/test_ssh_tunnel_manager_cov80.py::TestStderrIsDrainedWhileRunning::test_a_bind_lost_after_the_start_ends_the_child, test/test_ssh_tunnel_manager_cov80.py::TestAForwardTheSshConfigAdds::test_its_bind_failure_is_not_the_tunnels, test/test_ssh_tunnel_manager_cov80.py::TestAForwardTheSshConfigAdds::test_the_tunnels_failure_counts_after_the_tail_drops_it (head a277b65)

Determinism: the four TestAForwardTheSshConfigAdds tests passed 20 of 20 runs at -n0 and 10 of 10 at -n 4 --dist loadgroup --max-worker-restart=2 beside test/test_ssh_tunnel_manager_cov80.py and test/test_ssh_tunnel_manager_bindings.py (229 passed each run); the 104 tests of test/test_instances.py and test/test_ssh_tunnel_manager_cov80.py passed in a shuffled order (seed 3473416). prove.py prints at most five names; the two it leaves out are red on the parent too: it labels the cut line a bind failure and the banner's Permission denied an auth failure.

Manual verification

N/A — unit coverage sufficient: the argv is checked against the real ssh -G, and the stderr drain that reads the line is driven by a spawned child writing the exact text OpenSSH prints; reproducing the user's setup needs a second, already-bound ssh session to a live host.

Related Issues

Closes #34734

Sibling sweep

Swept: git grep -n "ExitOnForwardFailure"
Sibling: docs/guides/assets/com.kirocrew.tunnel.plist -- not affected -- a tunnel the user runs from their own launchd job with their own config, not one Kiro Crew spawns
Sibling: src/kiro_crew/instances/token_mint.py -- not affected -- the mint, restart and diagnostics ssh calls pin no ExitOnForwardFailure, so the user's own setting already applies to them

Pattern harvest

Not generalizable: a single ssh option pinned on one argv; the reason it is now no is recorded beside it and in the instances spec §9.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 11, 2026 16:42
@bolichen97
bolichen97 requested a review from CrysisDeu November 11, 2026 16:42
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 11, 2026
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision a277b6572ee1102d9284a9901b97548833ac65f1 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of a277b6572ee1102d9284a9901b97548833ac65f1 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. Changing the ExitOnForwardFailure pin fixes the cause reported in #34734, and the stderr latch is the smallest way to still catch the tunnel's own bind failure.

Not checked: whether Windows OpenSSH and older OpenSSH print cannot listen to port: <n> and keep running under no.

What this change ships

Inventory (5 items) — 5 justified

Intent: let a Remote Crew connect while the user's own ssh session holds a LocalForward port from the same host block. This is a FIX.

  1. A connect no longer fails when a LocalForward from the host block, on another port, is already taken. It prints a warning and the tunnel comes up. — justified
  2. When the tunnel's own forward cannot bind, ssh is still stopped, at connect or later, with the same ssh forward bind failed error. — justified
  3. The tunnel's ssh child now runs at LogLevel=INFO, overriding the user's QUIET or FATAL for that child only. — justified
  4. A bind failure seen on the tunnel's own port now outranks an auth phrase in a server banner. A bind line about another port no longer labels the exit. — justified
  5. The instances spec (§7, §9) and the module comments now describe ExitOnForwardFailure=no instead of yes. — justified

[FIRST-PRINCIPLES-REVIEWED] a277b65

Description read

[DESCRIPTION-READ] ba8594b4eeca096451b092af578c657b596ef287fa0d5ff566003eeecc4ae781

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of a277b6572ee1102d9284a9901b97548833ac65f1 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the fix targets the real cause (the pinned ExitOnForwardFailure=yes that every inherited LocalForward was held to), and the tunnel's own bind failure is still detected.

Notes

  • The tunnel now finds out its own bind failed only by matching OpenSSH's cannot listen to port: <n> text. If a future ssh build words that line differently, the failure would no longer end the child: the connect waits out the readiness timeout, or reports success if another process already holds the port. A test that pins the wording against the ssh binary on the host would turn that drift into a red test.

[DESIGN-REVIEWED] a277b65

Description read

[DESCRIPTION-READ] 02145377998071482c70672600863f2a89f8743f9fdf9300b5d2a17bf9c7fb72

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed a277b6572ee1102d9284a9901b97548833ac65f1 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] a277b65

Verdict parsed from the review's SHA-scoped output markers for commit a277b6572ee1102d9284a9901b97548833ac65f1.

False positive or not applicable? A repository writer can comment:
/ai-review override fable a277b6572ee1102d9284a9901b97548833ac65f1 false-positive: <reason, 20 to 500 characters>

@bolichen97
bolichen97 enabled auto-merge (squash) November 11, 2026 16:48
@github-actions

github-actions Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of a277b6572ee1102d9284a9901b97548833ac65f1 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] a277b65

False positive or not applicable? A repository writer can comment:
/ai-review override gpt a277b6572ee1102d9284a9901b97548833ac65f1 false-positive: <reason, 20 to 500 characters>

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes on 6f2b2a6. Moving the tunnel to ExitOnForwardFailure=no is the right fix for #34734, and there is no shell interpolation and no new non-loopback listener. But the guard that replaces the exit (stderr scan for the tunnel's own port) misses the tunnel's own bind failure when about 14 or more inherited forwards also fail in one read, and it can match a partial line, so a competing local listener can pass readiness. Decisive lines: src/kiro_crew/instances/ssh_tunnel_manager_child.py:944-949, src/kiro_crew/instances/ssh_tunnel_manager_child_text.py:192-204

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 11, 2026
@bolichen97
bolichen97 force-pushed the fix/34734-ssh-config-forwards branch from 6f2b2a6 to 56770dc Compare November 12, 2026 01:39
@bolichen97

bolichen97 commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: the tunnel's own bind failure is read off ssh's stderr instead of ExitOnForwardFailure

  • Parse complete bind diagnostics before truncating stderr span=f741840cf03c

fixed in 56770dc: the stderr drain checks each complete line as it arrives, before the 2,000-character tail can drop it, and carries an unterminated line into the next read.
A seen own-port line is latched on the tunnel; the exit classifier reads that latch first, never the truncated tail.
Pinned by test_the_tunnels_failure_counts_after_the_tail_drops_it (own line plus 20 inherited-forward failures in one read) and test_a_line_cut_at_the_tunnels_port_is_not_the_tunnels (port: 5400 | 7).

[operator: bolichen97]

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] @buluoray thanks, both halves were real; fixed in 56770dc.

  • The stderr drain checks each complete line as it arrives, before the 2,000-character tail can drop it, and carries an unfinished line into the next read. A burst of inherited-forward failures can no longer hide the tunnel's own cannot listen to port: <n> line, and ...port: 5400 followed by 7 in the next read is not taken for port 5400.
  • The own-port observation is latched on the tunnel, and the exit classifier reads the latch first, so a dropped line or an auth word in a server banner does not change the label.
  • Pinned by test_the_tunnels_failure_counts_after_the_tail_drops_it, test_a_line_cut_at_the_tunnels_port_is_not_the_tunnels and test_the_tunnels_failure_outranks_a_banner_phrase.

Could you take another look when you have a moment?

@KiroCrewCaptain

KiroCrewCaptain commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

The PR switches the ssh tunnel to ExitOnForwardFailure=no and pins LogLevel=INFO. The stderr reader now matches cannot listen to port: <n> against the tunnel's own port, remembers it in a flag and stops the child, so the tunnel's own bind failure still fails the connect or triggers self-heal with the same error text. A LocalForward from the user's ssh_config on another port no longer kills the tunnel. I checked who sets and reads the flag (a fresh tunnel object is built on every connect, self-heal rebuild and tunnel-keeper retry, so it can't go stale), that stderr is fully read before the error is built on every exit path, how lines split across reads are handled, the old tail-based bind label that was removed (existing tests still pass through the general message) and the SSM path (untouched). The tests cover the new behaviour well. Nothing blocks.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit a277b6572e · 274 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

buluoray
buluoray previously approved these changes Nov 12, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve on 56770dc. Both points from our earlier review are fixed. The stderr reader now checks each complete line for the tunnel's own "cannot listen to port" line before the bounded tail can drop it, and it keeps an unfinished line for the next read, so a burst of failures from the user's own ssh_config forwards can no longer hide the tunnel's bind failure, and a line cut in the middle of a port number is not mistaken for the tunnel's port. The finding is kept in a flag that the exit message reads first. New tests cover both cases and turn red when either fix is removed. A user whose ssh_config adds forwards that are already taken can now connect a remote crew, and the tunnel's own forward still listens on loopback only. The four red backend shards fail only on two tests that were failing on main before the main fix, and the Coverage Gate failed only because those shards failed. The PR needs a rebase onto main to clear them. Decisive lines: src/kiro_crew/instances/ssh_tunnel_manager_child.py:948-957, src/kiro_crew/instances/ssh_tunnel_manager_child.py:703-705, src/kiro_crew/instances/ssh_tunnel_manager_child_text.py:192-204

@bolichen97
bolichen97 force-pushed the fix/34734-ssh-config-forwards branch from 56770dc to 85a60e6 Compare November 12, 2026 07:15
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in src/kiro_crew/instances/ssh_tunnel_manager_child.py and docs/system-specs/modules/instances.md since this head's CI ran (ee7a026 -> 80cf3df); CI re-running.

The four Backend Tests reds on 56770dc were main's, not this PR's: test/test_wait_discipline_ratchet.py::test_the_scan_matches_known_sites and test/test_updates_handlers_facade.py::test_every_name_served_from_outside_the_family_is_visible_to_type_checkers came in with #35088 and were fixed by #35625, which merged after that run started. Both pass on 85a60e6. The patch itself is unchanged (git range-diff reports it identical).
[operator: bolichen97]

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 12, 2026
@bolichen97

bolichen97 commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: the tunnel's own bind failure is read off ssh's stderr instead of ExitOnForwardFailure

  • Port-only matching terminates a healthy forward on another address span=4864b2d6e67c

Rebutted: disproportionate, and not a regression. ssh's cannot listen to port: <n> line names only the port. Checking the bind [addr]:port line before it could skip the other-address case, but no stderr-only check covers the class: an inherited LocalForward <n> on the tunnel's own address prints exactly the tunnel's own failure text.
The case needs an inherited forward on exactly the port the allocator picks, on a loopback address the allocator does not probe (it probes 127.0.0.1 and ::1), already held by another process.
When it happens the connect fails loudly with "ssh forward bind failed (local port already in use)". Before this PR, ExitOnForwardFailure=yes gave that same failure for every taken inherited forward on any port; this PR only shrinks that set.
The comments, spec and body that said an inherited forward never ends the tunnel now name this same-port exception (a277b65); the code path is unchanged. The GPT lane's own adjudication on this head downgraded the finding for the same reason.

[operator: bolichen97]

The Remote Crew tunnel pinned ExitOnForwardFailure=yes on its ssh
command line. ssh also sets up every LocalForward the host's ssh_config
block names, and the pin overrides the user's own setting, so one of
those ports already held by their terminal session made ssh exit and
every connect fail with "ssh forward bind failed".

The tunnel now runs with ExitOnForwardFailure=no. Its own forward
failing still ends the child: ssh names the port of each forward it
could not bind on stderr, and the stderr drain ends the child when that
port is the tunnel's own, at connect or later, so the readiness wait or
the exit monitor reports it as before. LogLevel=INFO is pinned so a
user's QUIET or FATAL cannot hide that line. The exit classifier reads
the port the same way, so a line about another port never labels a
later exit as a bind failure.

Closes #34734
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/system-specs/modules/ (the directory of this PR's instances.md; no file this PR touches) since this head's CI ran (80cf3df -> f9fb5aa); CI re-running.

The one Backend Tests red on 85a60e6 was main's, not this PR's: test/test_remove_slot_for_history_key_crew_log.py::TestSessionLedgerOnPermanentDelete::test_a_ledger_the_emitter_itself_opened_is_still_removed is the known flake #34376, fixed by #35705, which merged after that run; Coverage Gate failed closed on it. It passes 5 of 5 on a277b65. No executable line changed: a277b65 only qualifies comments, the spec and the body with the same-port exception named in the GPT disposition above.
[operator: bolichen97]

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Nov 12, 2026
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 12, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] @buluoray the rebase onto main for the #35705 flake fix dismissed your approval. No executable line changed since you approved: a277b65 only adds the "on another port" qualifier to comments, test docstrings, the instances spec and the body (the same-port exception in the GPT disposition above). Could you take another look when you have a moment?

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve on a277b65. Since the head we approved, the author rebased onto main and changed only comments, test docstrings and the instances spec. They now say that a forward from the user's ssh_config on another port cannot end the tunnel, and that one on the tunnel's own port number counts as the tunnel's own. No executable line changed. The two points from our earlier change request are still fixed, and tests turn red when either fix is removed. The tunnel's own forward still listens on loopback and still forwards to loopback on the host the user named; a user's ssh_config cannot widen it. A crew without such a forward gets the same connect as before, apart from two pinned ssh options. Every finished check on this head is green. Decisive lines: src/kiro_crew/instances/ssh_tunnel_manager_child.py:955-963, src/kiro_crew/instances/ssh_tunnel_manager_child.py:708-710, src/kiro_crew/instances/ssh_tunnel_manager_child_text.py:192-205, src/kiro_crew/instances/ssh_tunnel_manager_child_text.py:287-291

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 12, 2026
@bolichen97
bolichen97 merged commit 8d04c52 into main Nov 12, 2026
101 checks passed
@bolichen97
bolichen97 deleted the fix/34734-ssh-config-forwards branch November 12, 2026 16:41
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Remote Crew SSH tunnel fails when LocalForward ports from ssh_config are already bound

3 participants