Repository navigation
fix(instances): connect remote crews when an ssh_config forward is taken - #35407
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. Changing the Not checked: whether Windows OpenSSH and older OpenSSH print What this change shipsInventory (5 items) — 5 justifiedIntent: let a Remote Crew connect while the user's own ssh session holds a
[FIRST-PRINCIPLES-REVIEWED] a277b65 Description read[DESCRIPTION-READ] ba8594b4eeca096451b092af578c657b596ef287fa0d5ff566003eeecc4ae781 That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the fix targets the real cause (the pinned Notes
[DESIGN-REVIEWED] a277b65 Description read[DESCRIPTION-READ] 02145377998071482c70672600863f2a89f8743f9fdf9300b5d2a17bf9c7fb72 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
buluoray
left a comment
There was a problem hiding this comment.
Request changes on 6f2b2a6. Moving the tunnel to ExitOnForwardFailure=no is the right fix for #34734, and there is no shell interpolation and no new non-loopback listener. But the guard that replaces the exit (stderr scan for the tunnel's own port) misses the tunnel's own bind failure when about 14 or more inherited forwards also fail in one read, and it can match a partial line, so a competing local listener can pass readiness. Decisive lines: src/kiro_crew/instances/ssh_tunnel_manager_child.py:944-949, src/kiro_crew/instances/ssh_tunnel_manager_child_text.py:192-204
6f2b2a6 to
56770dc
Compare
|
self-added: yes
[operator: bolichen97] |
|
[operator: bolichen97] @buluoray thanks, both halves were real; fixed in 56770dc.
Could you take another look when you have a moment? |
Code change approver recommendation: ApproveWhy: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes. The PR switches the ssh tunnel to ExitOnForwardFailure=no and pins LogLevel=INFO. The stderr reader now matches An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
buluoray
left a comment
There was a problem hiding this comment.
Approve on 56770dc. Both points from our earlier review are fixed. The stderr reader now checks each complete line for the tunnel's own "cannot listen to port" line before the bounded tail can drop it, and it keeps an unfinished line for the next read, so a burst of failures from the user's own ssh_config forwards can no longer hide the tunnel's bind failure, and a line cut in the middle of a port number is not mistaken for the tunnel's port. The finding is kept in a flag that the exit message reads first. New tests cover both cases and turn red when either fix is removed. A user whose ssh_config adds forwards that are already taken can now connect a remote crew, and the tunnel's own forward still listens on loopback only. The four red backend shards fail only on two tests that were failing on main before the main fix, and the Coverage Gate failed only because those shards failed. The PR needs a rebase onto main to clear them. Decisive lines: src/kiro_crew/instances/ssh_tunnel_manager_child.py:948-957, src/kiro_crew/instances/ssh_tunnel_manager_child.py:703-705, src/kiro_crew/instances/ssh_tunnel_manager_child_text.py:192-204
56770dc to
85a60e6
Compare
|
Rebased: main moved in src/kiro_crew/instances/ssh_tunnel_manager_child.py and docs/system-specs/modules/instances.md since this head's CI ran (ee7a026 -> 80cf3df); CI re-running. The four Backend Tests reds on 56770dc were main's, not this PR's: |
|
self-added: yes
[operator: bolichen97] |
The Remote Crew tunnel pinned ExitOnForwardFailure=yes on its ssh command line. ssh also sets up every LocalForward the host's ssh_config block names, and the pin overrides the user's own setting, so one of those ports already held by their terminal session made ssh exit and every connect fail with "ssh forward bind failed". The tunnel now runs with ExitOnForwardFailure=no. Its own forward failing still ends the child: ssh names the port of each forward it could not bind on stderr, and the stderr drain ends the child when that port is the tunnel's own, at connect or later, so the readiness wait or the exit monitor reports it as before. LogLevel=INFO is pinned so a user's QUIET or FATAL cannot hide that line. The exit classifier reads the port the same way, so a line about another port never labels a later exit as a bind failure. Closes #34734
85a60e6 to
a277b65
Compare
|
Rebased: main moved in docs/system-specs/modules/ (the directory of this PR's instances.md; no file this PR touches) since this head's CI ran (80cf3df -> f9fb5aa); CI re-running. The one Backend Tests red on 85a60e6 was main's, not this PR's: |
|
[operator: bolichen97] @buluoray the rebase onto main for the #35705 flake fix dismissed your approval. No executable line changed since you approved: a277b65 only adds the "on another port" qualifier to comments, test docstrings, the instances spec and the body (the same-port exception in the GPT disposition above). Could you take another look when you have a moment? |
buluoray
left a comment
There was a problem hiding this comment.
Approve on a277b65. Since the head we approved, the author rebased onto main and changed only comments, test docstrings and the instances spec. They now say that a forward from the user's ssh_config on another port cannot end the tunnel, and that one on the tunnel's own port number counts as the tunnel's own. No executable line changed. The two points from our earlier change request are still fixed, and tests turn red when either fix is removed. The tunnel's own forward still listens on loopback and still forwards to loopback on the host the user named; a user's ssh_config cannot widen it. A crew without such a forward gets the same connect as before, apart from two pinned ssh options. Every finished check on this head is green. Decisive lines: src/kiro_crew/instances/ssh_tunnel_manager_child.py:955-963, src/kiro_crew/instances/ssh_tunnel_manager_child.py:708-710, src/kiro_crew/instances/ssh_tunnel_manager_child_text.py:192-205, src/kiro_crew/instances/ssh_tunnel_manager_child_text.py:287-291
Problem / Motivation
Goal: A Remote Crew SSH tunnel connects even when the host's
~/.ssh/configblock addsLocalForwardports that another ssh session already holds.Connecting a Remote Crew to a host whose ssh_config block has
LocalForwardlines fails while a terminalsshto the same host is open. The dashboard showsssh forward bind failed (local port already in use): bind [127.0.0.1]:<port>: Address already in use, for a port Kiro Crew never asked for, andExitOnForwardFailure noin the user's config does not help.Why it matters
A terminal session open to a box next to its Remote Crew connection is the normal workflow, and
LocalForwardlines in a host block are common. Every connect to that host fails until the user closes their own session or edits their ssh_config.Not a goal
ClearAllForwardings=yesalso clears the tunnel's own-L(checked withssh -Gon OpenSSH 9.9).What changed (motivation → approach → change)
-Lssh forward bind failedThe tunnel is
ssh -N -L ...and pinnedExitOnForwardFailure=yes. ssh opens everyLocalForwardthe host's ssh_config block names besides it, and a command-line pin overrides the user's own setting, so one of those ports already taken made ssh exit.The tunnel runs with
ExitOnForwardFailure=no, which treats such a forward the way the user's own sessions do. Its own forward failing still ends the child: ssh printscannot listen to port: <n>for each forward it could not bind, and the stderr drain reads every complete line as it arrives and stops ssh when<n>is the tunnel's own port, at connect or any time later, so the readiness wait or the exit monitor reports it exactly as ssh's own exit was.LogLevel=INFOis pinned so a user'sLogLevel QUIETcannot hide that line. The drain remembers that it saw the tunnel's own line, and the exit classifier labels the exit from that memory before it reads any other phrase in the tail, such as an auth word in a server banner. A burst of other forwards' lines that pushes it out of the kept 2,000-character stderr tail cannot hide it. A line cut mid-number at a read boundary is not read as the tunnel's port, and a line about another port never labels a later exit as a bind failure. The instances spec (§7, §9) and the comments that named the old flag ininstances/say so.Backwards compatibility
Compatible: the tunnel's own bind failure still fails the connect with the same
ssh forward bind failed (local port already in use)text; only a failure on a forward the user's ssh_config adds on another port stops ending the tunnel; one on the tunnel's own port number still reads as the tunnel's own, as every taken forward did before.Removes nothing: no option, key or message is taken away. A forwarder left running by an older gateway whose record has no argv fingerprint no longer matches the rebuilt argv, so the orphan reclaim leaves it and its port alone, as after any earlier argv change.
Tests
test/test_instances.py::TestSshTunnelMultiplexing::test_per_host_ssh_config_is_still_inherited: a host block withLocalForward 8888 localhost:8888,ExitOnForwardFailure yesandLogLevel QUIET; the realssh -Gresolves the tunnel's argv toexitonforwardfailure noandloglevel INFOwith both forwards present.test/test_ssh_tunnel_manager_cov80.py::TestStderrIsDrainedWhileRunning::test_a_failed_start_still_reports_its_stderr: the spawned stand-in does what ssh does undernowhen the tunnel's own port is taken (prints the two bind lines naming that port and keeps running);start()still fails withssh forward bind failed.test/test_ssh_tunnel_manager_cov80.py::TestStderrIsDrainedWhileRunning::test_a_bind_lost_after_the_start_ends_the_child: another process already listens on the port, so the tunnel comes up first; the stand-in then reports its own port unbindable and the tunnel goes to ERROR withssh forward bind failedand fireson_exit.test/test_ssh_tunnel_manager_cov80.py::TestAForwardTheSshConfigAdds::test_its_bind_failure_is_not_the_tunnels: a bind failure on port 8888 does not stop the child and leaves a later exit unlabelled; the same line for the tunnel's own port, read by the drain, labels it.test/test_ssh_tunnel_manager_cov80.py::TestAForwardTheSshConfigAdds::test_the_tunnels_failure_counts_after_the_tail_drops_it: one read carries the tunnel's own line and then 20 failures for port 8888, enough to push it out of the stderr tail; the child is still stopped and the exit readsssh forward bind failed.test/test_ssh_tunnel_manager_cov80.py::TestAForwardTheSshConfigAdds::test_a_line_cut_at_the_tunnels_port_is_not_the_tunnels: a tunnel on port 5400 getscannot listen to port: 5400in one read and7in the next; the child keeps running and the exit is not a bind failure.test/test_ssh_tunnel_manager_cov80.py::TestAForwardTheSshConfigAdds::test_the_tunnels_failure_outranks_a_banner_phrase: a banner line withPermission deniedcomes before the tunnel's own bind line; the exit still readsssh forward bind failed, not an auth failure.Proof: PROVEN test/test_instances.py::TestSshTunnelMultiplexing::test_per_host_ssh_config_is_still_inherited, test/test_ssh_tunnel_manager_cov80.py::TestStderrIsDrainedWhileRunning::test_a_failed_start_still_reports_its_stderr, test/test_ssh_tunnel_manager_cov80.py::TestStderrIsDrainedWhileRunning::test_a_bind_lost_after_the_start_ends_the_child, test/test_ssh_tunnel_manager_cov80.py::TestAForwardTheSshConfigAdds::test_its_bind_failure_is_not_the_tunnels, test/test_ssh_tunnel_manager_cov80.py::TestAForwardTheSshConfigAdds::test_the_tunnels_failure_counts_after_the_tail_drops_it (head a277b65)
Determinism: the four
TestAForwardTheSshConfigAddstests passed 20 of 20 runs at-n0and 10 of 10 at-n 4 --dist loadgroup --max-worker-restart=2besidetest/test_ssh_tunnel_manager_cov80.pyandtest/test_ssh_tunnel_manager_bindings.py(229 passed each run); the 104 tests oftest/test_instances.pyandtest/test_ssh_tunnel_manager_cov80.pypassed in a shuffled order (seed 3473416).prove.pyprints at most five names; the two it leaves out are red on the parent too: it labels the cut line a bind failure and the banner'sPermission deniedan auth failure.Manual verification
N/A — unit coverage sufficient: the argv is checked against the real
ssh -G, and the stderr drain that reads the line is driven by a spawned child writing the exact text OpenSSH prints; reproducing the user's setup needs a second, already-bound ssh session to a live host.Related Issues
Closes #34734
Sibling sweep
Swept: git grep -n "ExitOnForwardFailure"
Sibling: docs/guides/assets/com.kirocrew.tunnel.plist -- not affected -- a tunnel the user runs from their own launchd job with their own config, not one Kiro Crew spawns
Sibling: src/kiro_crew/instances/token_mint.py -- not affected -- the mint, restart and diagnostics ssh calls pin no ExitOnForwardFailure, so the user's own setting already applies to them
Pattern harvest
Not generalizable: a single ssh option pinned on one argv; the reason it is now
nois recorded beside it and in the instances spec §9.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)