Skip to content

fix(instances): close the ssh probe's stdin so a proxied host isn't false-failed - #13158

Merged
chenmingwei23 merged 1 commit into
mainfrom
fix/9360-ssh-probe-stdin
Sep 24, 2026
Merged

chenmingwei23 merged 1 commit into
mainfrom
fix/9360-ssh-probe-stdin

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Closes #9360

Problem

A reachable, authenticating instance is reported ssh_unreachable / probes: ssh=FAILED
when the SSH host routes through a ProxyCommand and the gateway has no usable console
stdin (a service, a detached process, a Windows foreground gateway).

The probe argv built by token_mint._build_ssh_argv() carries no -n, and
diagnostics._run_ok() / _run_stdout() spawn with stdin unset, so the ssh child
inherits the gateway's stdin. Through a ProxyCommand ssh keeps that channel open after
the remote command exits; with no console the EOF never arrives and ssh waits.

ConnectTimeout does not bound it: connect, KEX, auth and the remote true all
succeed, so the hang is strictly after the command returns. The probe's wall-clock cap
(connect_timeout_secs + 2, diagnostics.py:163) is the only bound left, and its
expiry is mapped to SSH_UNREACHABLE — a healthy host reported unreachable.

Fix

Three product lines, both halves of the root cause, either of which is sufficient:

  • token_mint._build_ssh_argv(): -n immediately after ssh, so ssh itself redirects
    its stdin from the null device. This covers all four call sites that share the builder
    (both probes, mint_remote_token, run_remote_kirocrew).
  • diagnostics._run_ok() and _run_stdout(): stdin=asyncio.subprocess.DEVNULL, so no
    probe child can inherit the gateway's stdin regardless of the argv.

No caller feeds the child's stdin: git grep stdin over token_mint.py returns nothing,
and every communicate() call on these paths is passed no input, so -n cannot break a
stdin-feeding path.

Deliberately not done: suggestion (3) from the issue (stop DEVNULL-ing probe stderr and
carry a redacted tail into DiagnosisResult.reason). That needs a reason field plumbed
through the ladder plus a redaction policy for proxy-controlled stderr — a separate,
larger change, and not needed to fix the false positive. The tunnel argv
(ssh_tunnel_manager._build_ssh_tunnel_argv, -N, no remote command) is a separate path
and is left alone.

Tests

Both extended in test/test_instances.py, both red on main (5c23469f7e), green here:

  • TestTokenMint::test_ssh_argv_shape — -n is present and precedes the host.
    Red on main: AssertionError: assert '-n' in ['ssh', '-o', 'BatchMode=yes', ...].
  • TestDiagnostics::test_probes_honor_connect_timeout_secs — the patched
    create_subprocess_exec is called with stdin=asyncio.subprocess.DEVNULL on both the
    _run_ok path (_probe_ssh) and the _run_stdout path (_probe_remote_dashboard).
    Red on main: AssertionError: assert None is -3.

Gates run locally on the changed files: isort, flake8, black --check clean;
mypy src/kiro_crew/ clean (1706 files); pytest test/test_instances.py -k 'ssh_argv_shape or probe' 11 passed; scripts/check_black_formatting.py passes.

Unrelated rider: one black-baseline prune

.github/black-baseline.txt loses one line, test/test_knowledge.py. That file was
formatted clean by #13140 without a prune, and the black gate is shrink-only, so it fails
every PR on this base with "1 baselined file(s) are now black-clean". Verified
main-inherited: the file is byte-identical to origin/main here and black --check on
it is clean at that ref. Deletion only — no file is reformatted, and the baseline still
lists 971 entries. Carried here because the gate is a required lane; the same rider
pattern appears on #12467, #12303, #10904 and #9748.

Pattern harvest

Rule candidate: a subprocess spawned by a long-lived daemon must set stdin explicitly
(DEVNULL unless it is written), and an ssh argv that runs a remote command must carry
-n. An inherited stdin is invisible in a terminal and only hangs where there is no
console, so it survives review and manual testing and surfaces as a wrong diagnosis
rather than as a hang. Greppable as create_subprocess_exec calls with stdout= or
stderr= but no stdin=, and as "ssh" argv literals without -n.

@iamwhatever
iamwhatever requested review from a team and chenmingwei23 September 23, 2026 19:41
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 23, 2026
@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 819b2f27841bf7477904df6f3db44cd97f1edd85 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify once on a real ProxyCommand host with no console stdin that the probe now reports reachable — the tests pin argv/kwargs, not the hang itself.

[DESIGN-REVIEWED] 819b2f2

@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 819b2f27841bf7477904df6f3db44cd97f1edd85 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 819b2f2

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 819b2f27841bf7477904df6f3db44cd97f1edd85: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 819b2f27841bf7477904df6f3db44cd97f1edd85 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify the rider's premise: black --check test/test_knowledge.py is clean on this base — the review environment cannot run black to confirm it.

What this change ships

Inventory (3 items) — 2 justified

Intent: stop a healthy proxied SSH host being reported unreachable because the probe's ssh child inherits a console-less gateway stdin — a FIX (issue #9360, plus two tests red on base).

  1. Every ssh child built by _build_ssh_argv (both probes, token mint, remote launcher) now reads stdin from the null device (-n) — justified
  2. The diagnostics probe spawners (_run_ok, _run_stdout) no longer let any child inherit the gateway's stdin — justified
  3. One line pruned from .github/black-baseline.txt (test/test_knowledge.py) — rides along: declared, unwedges the shrink-only black gate that fails every PR on this base

Counts run: _build_ssh_argv has 4 real consumers (token_mint.py:455,538; diagnostics.py:163,182), matching the description; communicate\(|\.stdin over src/kiro_crew/instances/ shows no caller feeding stdin, so -n breaks nothing; the only other ssh spawn (ssh_tunnel_manager.py:614) runs no remote command, so zero unfixed siblings of this cause.

[FIRST-PRINCIPLES-REVIEWED] 819b2f2

@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 819b2f27841bf7477904df6f3db44cd97f1edd85 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 819b2f2

Verdict parsed from the review's SHA-scoped output markers for commit 819b2f27841bf7477904df6f3db44cd97f1edd85.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 819b2f27841bf7477904df6f3db44cd97f1edd85: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 23, 2026
…alse-failed

The diagnostics ssh probe builds its argv without `-n` and spawns without
setting `stdin`, so the ssh child inherits the gateway's stdin. Through a
`ProxyCommand` ssh keeps that channel open after the remote command exits;
on a gateway with no usable console the EOF never arrives, ssh waits, the
probe's wall-clock cap kills it, and the ladder renders that as
`SSH_UNREACHABLE` on a host that connects fine everywhere else.

`ConnectTimeout` cannot bound it: connect, KEX, auth and the remote `true`
all succeed, so the hang is strictly after the command returns.

Fix both halves of the cause: `-n` in the shared `_build_ssh_argv` (covers
both probes plus `mint_remote_token` and `run_remote_kirocrew`), and
`stdin=asyncio.subprocess.DEVNULL` in `_run_ok` and `_run_stdout` so no
probe child inherits the gateway's stdin regardless of argv. No caller on
these paths writes the child's stdin.

Tests pin both: `-n` present and before the host in the argv, and
`create_subprocess_exec` called with `stdin=DEVNULL` on the `_run_ok` and
`_run_stdout` probe paths. Both red before this change.

Also prunes one graduated entry from the black baseline
(`test/test_knowledge.py`, formatted clean by #13140 without a prune), which
the shrink-only black gate fails on for every PR until someone removes it.
Deletion only; no file is reformatted.
@iamwhatever
iamwhatever force-pushed the fix/9360-ssh-probe-stdin branch from a098d16 to 819b2f2 Compare September 23, 2026 20:20
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 23, 2026
@chenmingwei23
chenmingwei23 merged commit 06835f9 into main Sep 24, 2026
99 of 128 checks passed
@chenmingwei23
chenmingwei23 deleted the fix/9360-ssh-probe-stdin branch September 24, 2026 01:17
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 24, 2026
wknight1 pushed a commit to wknight1/KiroCrew that referenced this pull request Oct 5, 2026
…alse-failed (kirodotdev#13158)

The diagnostics ssh probe builds its argv without `-n` and spawns without
setting `stdin`, so the ssh child inherits the gateway's stdin. Through a
`ProxyCommand` ssh keeps that channel open after the remote command exits;
on a gateway with no usable console the EOF never arrives, ssh waits, the
probe's wall-clock cap kills it, and the ladder renders that as
`SSH_UNREACHABLE` on a host that connects fine everywhere else.

`ConnectTimeout` cannot bound it: connect, KEX, auth and the remote `true`
all succeed, so the hang is strictly after the command returns.

Fix both halves of the cause: `-n` in the shared `_build_ssh_argv` (covers
both probes plus `mint_remote_token` and `run_remote_kirocrew`), and
`stdin=asyncio.subprocess.DEVNULL` in `_run_ok` and `_run_stdout` so no
probe child inherits the gateway's stdin regardless of argv. No caller on
these paths writes the child's stdin.

Tests pin both: `-n` present and before the host in the argv, and
`create_subprocess_exec` called with `stdin=DEVNULL` on the `_run_ok` and
`_run_stdout` probe paths. Both red before this change.

Also prunes one graduated entry from the black baseline
(`test/test_knowledge.py`, formatted clean by kirodotdev#13140 without a prune), which
the shrink-only black gate fails on for every PR until someone removes it.
Deletion only; no file is reformatted.

Co-authored-by: Joe Guo <zejiangg@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants