Repository navigation
fix(instances): close the ssh probe's stdin so a proxied host isn't false-failed - #13158
Conversation
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify once on a real ProxyCommand host with no console stdin that the probe now reports reachable — the tests pin argv/kwargs, not the hang itself. [DESIGN-REVIEWED] 819b2f2 |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Verify the rider's premise: What this change shipsInventory (3 items) — 2 justifiedIntent: stop a healthy proxied SSH host being reported unreachable because the probe's ssh child inherits a console-less gateway stdin — a FIX (issue #9360, plus two tests red on base).
Counts run: [FIRST-PRINCIPLES-REVIEWED] 819b2f2 |
Opus 5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
…alse-failed The diagnostics ssh probe builds its argv without `-n` and spawns without setting `stdin`, so the ssh child inherits the gateway's stdin. Through a `ProxyCommand` ssh keeps that channel open after the remote command exits; on a gateway with no usable console the EOF never arrives, ssh waits, the probe's wall-clock cap kills it, and the ladder renders that as `SSH_UNREACHABLE` on a host that connects fine everywhere else. `ConnectTimeout` cannot bound it: connect, KEX, auth and the remote `true` all succeed, so the hang is strictly after the command returns. Fix both halves of the cause: `-n` in the shared `_build_ssh_argv` (covers both probes plus `mint_remote_token` and `run_remote_kirocrew`), and `stdin=asyncio.subprocess.DEVNULL` in `_run_ok` and `_run_stdout` so no probe child inherits the gateway's stdin regardless of argv. No caller on these paths writes the child's stdin. Tests pin both: `-n` present and before the host in the argv, and `create_subprocess_exec` called with `stdin=DEVNULL` on the `_run_ok` and `_run_stdout` probe paths. Both red before this change. Also prunes one graduated entry from the black baseline (`test/test_knowledge.py`, formatted clean by #13140 without a prune), which the shrink-only black gate fails on for every PR until someone removes it. Deletion only; no file is reformatted.
a098d16 to
819b2f2
Compare
…alse-failed (kirodotdev#13158) The diagnostics ssh probe builds its argv without `-n` and spawns without setting `stdin`, so the ssh child inherits the gateway's stdin. Through a `ProxyCommand` ssh keeps that channel open after the remote command exits; on a gateway with no usable console the EOF never arrives, ssh waits, the probe's wall-clock cap kills it, and the ladder renders that as `SSH_UNREACHABLE` on a host that connects fine everywhere else. `ConnectTimeout` cannot bound it: connect, KEX, auth and the remote `true` all succeed, so the hang is strictly after the command returns. Fix both halves of the cause: `-n` in the shared `_build_ssh_argv` (covers both probes plus `mint_remote_token` and `run_remote_kirocrew`), and `stdin=asyncio.subprocess.DEVNULL` in `_run_ok` and `_run_stdout` so no probe child inherits the gateway's stdin regardless of argv. No caller on these paths writes the child's stdin. Tests pin both: `-n` present and before the host in the argv, and `create_subprocess_exec` called with `stdin=DEVNULL` on the `_run_ok` and `_run_stdout` probe paths. Both red before this change. Also prunes one graduated entry from the black baseline (`test/test_knowledge.py`, formatted clean by kirodotdev#13140 without a prune), which the shrink-only black gate fails on for every PR until someone removes it. Deletion only; no file is reformatted. Co-authored-by: Joe Guo <zejiangg@amazon.com>
Closes #9360
Problem
A reachable, authenticating instance is reported
ssh_unreachable/probes: ssh=FAILEDwhen the SSH host routes through a
ProxyCommandand the gateway has no usable consolestdin (a service, a detached process, a Windows foreground gateway).
The probe argv built by
token_mint._build_ssh_argv()carries no-n, anddiagnostics._run_ok()/_run_stdout()spawn withstdinunset, so thesshchildinherits the gateway's stdin. Through a
ProxyCommandssh keeps that channel open afterthe remote command exits; with no console the EOF never arrives and ssh waits.
ConnectTimeoutdoes not bound it: connect, KEX, auth and the remotetrueallsucceed, so the hang is strictly after the command returns. The probe's wall-clock cap
(
connect_timeout_secs + 2,diagnostics.py:163) is the only bound left, and itsexpiry is mapped to
SSH_UNREACHABLE— a healthy host reported unreachable.Fix
Three product lines, both halves of the root cause, either of which is sufficient:
token_mint._build_ssh_argv():-nimmediately afterssh, so ssh itself redirectsits stdin from the null device. This covers all four call sites that share the builder
(both probes,
mint_remote_token,run_remote_kirocrew).diagnostics._run_ok()and_run_stdout():stdin=asyncio.subprocess.DEVNULL, so noprobe child can inherit the gateway's stdin regardless of the argv.
No caller feeds the child's stdin:
git grep stdinovertoken_mint.pyreturns nothing,and every
communicate()call on these paths is passed no input, so-ncannot break astdin-feeding path.
Deliberately not done: suggestion (3) from the issue (stop
DEVNULL-ing probe stderr andcarry a redacted tail into
DiagnosisResult.reason). That needs a reason field plumbedthrough the ladder plus a redaction policy for proxy-controlled stderr — a separate,
larger change, and not needed to fix the false positive. The tunnel argv
(
ssh_tunnel_manager._build_ssh_tunnel_argv,-N, no remote command) is a separate pathand is left alone.
Tests
Both extended in
test/test_instances.py, both red on main (5c23469f7e), green here:TestTokenMint::test_ssh_argv_shape—-nis present and precedes the host.Red on main:
AssertionError: assert '-n' in ['ssh', '-o', 'BatchMode=yes', ...].TestDiagnostics::test_probes_honor_connect_timeout_secs— the patchedcreate_subprocess_execis called withstdin=asyncio.subprocess.DEVNULLon both the_run_okpath (_probe_ssh) and the_run_stdoutpath (_probe_remote_dashboard).Red on main:
AssertionError: assert None is -3.Gates run locally on the changed files:
isort,flake8,black --checkclean;mypy src/kiro_crew/clean (1706 files);pytest test/test_instances.py -k 'ssh_argv_shape or probe'11 passed;scripts/check_black_formatting.pypasses.Unrelated rider: one black-baseline prune
.github/black-baseline.txtloses one line,test/test_knowledge.py. That file wasformatted clean by #13140 without a prune, and the black gate is shrink-only, so it fails
every PR on this base with "1 baselined file(s) are now black-clean". Verified
main-inherited: the file is byte-identical to
origin/mainhere andblack --checkonit is clean at that ref. Deletion only — no file is reformatted, and the baseline still
lists 971 entries. Carried here because the gate is a required lane; the same rider
pattern appears on #12467, #12303, #10904 and #9748.
Pattern harvest
Rule candidate: a subprocess spawned by a long-lived daemon must set
stdinexplicitly(
DEVNULLunless it is written), and ansshargv that runs a remote command must carry-n. An inherited stdin is invisible in a terminal and only hangs where there is noconsole, so it survives review and manual testing and surfaces as a wrong diagnosis
rather than as a hang. Greppable as
create_subprocess_execcalls withstdout=orstderr=but nostdin=, and as"ssh"argv literals without-n.