Skip to content

fix(instances): make a proxy-slow Remote Crew connect timeout actionable - #19615

Merged
KiroCrewCaptain merged 2 commits into
mainfrom
fix/remote-crew-connect-timeout-proxy-handshake
Oct 29, 2026
Merged

KiroCrewCaptain merged 2 commits into
mainfrom
fix/remote-crew-connect-timeout-proxy-handshake

Conversation

@mbriones98

@mbriones98 mbriones98 commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Make a Remote Crew connection that times out while still handshaking through a proxy tell the user how to fix it, instead of a dead-end error.

When a Remote Crew connect runs out of its readiness budget, the user sees a flat timed out after 15s waiting for forward regardless of why. On a host reached through a ProxyCommand (jump host, WSSH, corporate proxy) that message is actively misleading: the proxy handshake alone routinely consumes most of the default 15s budget before ssh even begins binding the local forward, so the connection is still being established when the deadline fires. The user is left with no cause and no next step.

Why it matters

Anyone connecting a Remote Crew through a jump host or corporate proxy — a common setup — hits this. The timeout is the binding constraint for those hosts, and the knob that fixes it (instances.connect_timeout_secs) already exists, but the error names neither the cause nor the knob, so the user can only retry into the same wall. The result is a feature that looks broken for proxy users when it is one config value away from working.

Not a goal

  • Changing the default connect_timeout_secs (15s). The default stays tuned for the fast direct case so a genuinely dead host still fails fast; proxy-slow hosts are served by the actionable message pointing at the existing tunable.
  • Auto-detecting proxy latency or auto-raising the timeout. That is a product-shape decision; this PR only improves the diagnostic.
  • Any change to the SSM transport's timeout or to the steady-state health probe.

What changed (motivation → approach → change)

Observed symptom: a proxy-bound connect reports an opaque timed out ... waiting for forward. Root cause: _SshTunnel._wait_until_ready emits one fixed string for every timeout, with no cause and no next step.

Approach: when the tunnel child is still running at the deadline, it has not failed with a classifiable error. It is either still connecting or stuck on a host that never answers, and the two look the same locally, so the message names both and the setting that raises the budget instead of asserting either cause.

Change: a new _ready_timeout_error in instances/ssh_tunnel_manager_child.py composes the message for the active transport.

Child at deadline Transport Message
still running ssh still connecting; a jump host or proxy handshake can use most of the budget, or the host may be unreachable; raise instances.connect_timeout_secs
still running ssm session still starting; the SSM session handshake may be slow, or the target's SSM agent may be offline; raise instances.connect_timeout_secs
still running, budget at CONNECT_TIMEOUT_CEILING_SECS either same cause text; points at reachability instead of a raise
already exited, or cleared by stop() either generic timed out after <N>s waiting for forward (unchanged)

Backwards compatibility

Compatible: no field, validator, type, kind or key changes. The only behavioral change is the text of TunnelStatus.error on the readiness-timeout path; every branch keeps the timed out after <N>s prefix.

Removes nothing: no path is hidden, no record/row deleted, no validator tightened, no key or function removed. The generic timeout string is preserved verbatim for the already-exited-child case, and no test asserts the exact readiness-timeout wording.

Tests

  • test_wait_until_ready_timeout_with_live_child_names_the_tunable — an ssh timeout with a live child names both causes and instances.connect_timeout_secs, and no SSM wording.
  • test_wait_until_ready_timeout_on_ssm_names_the_ssm_session — an SSM timeout with a live child names the SSM session and agent, with no ssh or proxy wording.
  • test_ready_timeout_at_ceiling_does_not_suggest_raising_it — a budget at the ceiling omits the tunable and points at reachability.
  • test_wait_until_ready_timeout_with_dead_child_stays_generic — an already-exited child keeps the generic wording.

The tests use fake child objects and a 0.05s budget with no wall-clock assertions; the tests live in test/test_instances_self_heal.py; after the rebase onto main, test_instances_self_heal.py, test_ssh_tunnel_manager_cov80.py and test_instances.py (136 tests) pass, and the related backend set (1477 tests), black, isort, flake8, mypy and the comment-history gate are clean.

Manual verification

N/A — unit coverage exercises both timeout branches directly; the change is a backend error string with no interactive path.

Screenshots / video

N/A — the diff touches no user-visible frontend surface; it changes a backend status/error string only.

Related Issues

Pattern harvest

Pattern: one error string reused across two root causes (a dead link vs. a slow-but-live connect) hides the one fact that tells them apart. When a timeout can fire for more than one reason, the message must surface the distinguishing fact and the knob that addresses the common case, rather than a single opaque sentence.

Rule candidate: review-prompt — flag a readiness/timeout path that emits a fixed error string without consulting the child/process liveness state that distinguishes "failed" from "needs more budget."

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable) — N/A, no doc surface changed
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision a2de07b3b8df4b23fbf6e617230ffb66d4560c7a touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 13, 2026
@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of a2de07b3b8df4b23fbf6e617230ffb66d4560c7a — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks: every item changes only the readiness-timeout text, and the setting it names already exists. _timeout_tunable is a private attribute that two places in the manager set.

What this change ships

Inventory (5 items) — 4 justified

Intent: when a Remote Crew connection through a jump host or proxy times out, tell the user why and which setting to raise. FIX.

  1. An SSH connection that times out while ssh is still running now says the proxy handshake may be slow or the host unreachable, and names instances.connect_timeout_secs — justified
  2. The same timeout on SSM names the SSM session and agent instead — justified
  3. A timeout budget already at its ceiling points at host reachability instead of suggesting a raise — justified
  4. Tunnels the manager does not build from instances.connect_timeout_secs (the tunnel keeper's, and the remote_ops ones built with min(...)) name no setting; this is done with a new private flag, _timeout_tunable, set in lifecycle.connect and self_heal._rebuild — undeclared (the description never mentions the flag or this branch; only the commit title does)
  5. A timeout where the ssh process has already exited keeps the generic wording — justified

[FIRST-PRINCIPLES-REVIEWED] a2de07b

Description read

[DESCRIPTION-READ] 5257755c2cdb47b8fc3bab20845994ccee02b927c6834a4e0cb5ba1d85e86de6

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of a2de07b3b8df4b23fbf6e617230ffb66d4560c7a — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the change is backend error wording only. No code anywhere parses the old waiting for forward text, and an exited child still gets the generic message.

Notes

  • _timeout_tunable is a mutable attribute that only the manager's two construction sites (connect, _rebuild) set. Making it a constructor argument would let a future third site fail safe by having to choose a value, instead of silently falling back to the no-setting wording.

[DESIGN-REVIEWED] a2de07b

Description read

[DESCRIPTION-READ] 795f4c07b7547f46f2542c3c11565d0a9a0e94ce8639c57111ac5041fe5549f3

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of a2de07b3b8df4b23fbf6e617230ffb66d4560c7a and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] a2de07b

False positive or not applicable? A repository writer can comment:
/ai-review override gpt a2de07b3b8df4b23fbf6e617230ffb66d4560c7a: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed a2de07b3b8df4b23fbf6e617230ffb66d4560c7a — this comment is updated in place on each push.

Review details

No findings.

Notes:

  • In the desktop tunnel keeper, the untunable message ends "connect again", but the keeper's _RETRYING line ("tunnel: down (…); retrying in Ns") already retries automatically. Consider dropping "and connect again" from that branch.

[OPUS-REVIEWED] a2de07b

Verdict parsed from the review's SHA-scoped output markers for commit a2de07b3b8df4b23fbf6e617230ffb66d4560c7a.

False positive or not applicable? A repository writer can comment:
/ai-review override fable a2de07b3b8df4b23fbf6e617230ffb66d4560c7a: <one-sentence reason>

@mbriones98
mbriones98 force-pushed the fix/remote-crew-connect-timeout-proxy-handshake branch 2 times, most recently from ed6ab8d to ca93edf Compare October 13, 2026 18:33
@mbriones98

Copy link
Copy Markdown
Contributor Author
  • span=46065f954632 SSM timeout reads ssh/proxy wording

fixed in ca93edf: _ready_timeout_error branches on self._transport; the SSM branch names the SSM session handshake and the target's SSM agent, with no ssh or proxy wording.
test_wait_until_ready_timeout_on_ssm_names_the_ssm_session asserts "ssh" and "proxy" are absent from the SSM message.

@mbriones98

Copy link
Copy Markdown
Contributor Author
  • span=1f9ea2feb79f live child does not separate slow proxy from dead host

fixed in ca93edf: the wording is hedged. A live child at the deadline now reads "ssh was still connecting ... a jump host or proxy can spend most of this budget on the proxy handshake, or the host may be unreachable", and the tunable is offered only "if the host is reachable but slow to connect".
Hedging was chosen over an ssh -G gate: it adds no subprocess to the timeout path and stays correct for a black-holed direct host.

@mbriones98

Copy link
Copy Markdown
Contributor Author
  • span=13ff7cc0b860 new text reaches SSM

fixed in ca93edf: the message is transport-aware; SSM gets its own wording (SSM session handshake, target's SSM agent).
test_wait_until_ready_timeout_on_ssm_names_the_ssm_session covers an SSM tunnel timing out with a live child.

@mbriones98

Copy link
Copy Markdown
Contributor Author
  • span=0abf9fe52dc5 do not suggest raising a budget already at the ceiling

fixed in ca93edf: at CONNECT_TIMEOUT_CEILING_SECS the message says the budget is already at its maximum and points at host reachability, omitting the tunable.
test_ready_timeout_at_ceiling_does_not_suggest_raising_it locks it.

@mbriones98

Copy link
Copy Markdown
Contributor Author
  • span=4043b4ffc9a5 SSM tunnel told about the SSH forward

fixed in ca93edf: _ready_timeout_error checks self._transport; SSM reads "waiting for the SSM port forward" and names the SSM agent.
Covered by test_wait_until_ready_timeout_on_ssm_names_the_ssm_session.

@mbriones98
mbriones98 marked this pull request as ready for review October 13, 2026 18:39
@mbriones98
mbriones98 requested review from a team and KiroCrewCaptain as code owners October 13, 2026 18:39
@mbriones98
mbriones98 requested a review from CrysisDeu October 13, 2026 18:39
@mbriones98

Copy link
Copy Markdown
Contributor Author
  • span=eabdea71b2c4 mint timeout sibling names no setting

accepted-and-deferred: this PR's frozen goal is the forward-readiness timeout message only; the token-mint timeouts (token_mint.py, ssm_token_mint.py) are a separate step with their own tunable instances.mint_timeout_secs.
Deferred to #19882 (deferred-finding, assigned, Due line): name instances.mint_timeout_secs in both mint-timeout messages with a test per transport.

@mbriones98

Copy link
Copy Markdown
Contributor Author
  • PASS punchline: dashboard shows the longer TunnelStatus.error in full

rebutted: no surface clips it. InstancesViewport.tsx renders status.error in an ErrorNotice with max-h-32 overflow-auto (wraps and scrolls); the inline ErrorNotice in InstancesPanel.tsx sets overflowWrap: 'anywhere' with no truncate, so it wraps inside max-w-[240px].
InstanceTabBar.tsx puts the full string in the tab's title tooltip. The kirocrew config set hint is reachable on every surface.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 13, 2026
@mbriones98
mbriones98 enabled auto-merge (squash) October 13, 2026 21:09

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This replaces the fixed timed out after Ns waiting for forward string in _SshTunnel._wait_until_ready with _ready_timeout_error (src/kiro_crew/instances/ssh_tunnel_manager.py:1153), which checks the child's liveness and transport. When the child is still running, the message names the likely causes and the instances.connect_timeout_secs setting. At the 120 s CONNECT_TIMEOUT_CEILING_SECS it points at reachability instead, and an exited or cleared _proc keeps the generic text. That fixes the real gap: the message used to name neither the cause nor the setting. I checked that _transport and _connect_timeout are set in __init__, that the ceiling constant and the kirocrew config set instances.connect_timeout_secs command it points at both exist, that the timed out after prefix is kept on every branch, and that the four tests cover each branch without timing assertions. All 100 check runs are success or skipped, and every AI finding has a disposition. The code is correct, but there is one follow-up item: the First Principles mint-timeout sibling (span eabdea71b2c4, token_mint.py:583 and ssm_token_mint.py:185 still name no instances.mint_timeout_secs) was deferred to #19882. The PR description's Related Issues section still says "N/A". Please link #19882 there (for example Part of #19882 or a "Follow-up: #19882" line), and I will approve on the next round. — crew-pr-reviewer (deep re-check)

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 14, 2026
@mbriones98

Copy link
Copy Markdown
Contributor Author

Linked #19882 under Related Issues.

@github-actions github-actions Bot added merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention labels Oct 16, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Oct 17, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

Holding on 0ef776e. The fix is sound, and the item from our earlier change request is done: #19882 is linked and was closed by merged #20381.

Blocker: two AI items on this head have no reply.

  1. The GPT 6.1 lane lists a FINDING at src/kiro_crew/acp/mcp_session_report.py:470. That file is not in this PR's diff (main...0ef776e changes only ssh_tunnel_manager_child.py and test/test_instances.py); it came in with the origin/main merge. A target=gpt rebuttal saying so, or a writer override for this head, would clear it.
  2. The First Principles PASS asks to confirm that only those two files are this PR's own change. That is true; a short reply on the PR would close it.

Once both are answered, nothing else blocks approval on this head. Checks are green, and _ready_timeout_error covers the exited-child, ceiling, SSM and SSH branches, each with its own test.

— crew-pr-reviewer (r108)

@mbriones98

Copy link
Copy Markdown
Contributor Author
  • span=c56f3a4ffa5b MCP startup-warning filtering in mcp_session_report.py

justified: src/kiro_crew/acp/mcp_session_report.py is not part of the change this PR makes. It arrived through the origin/main merge commit, and the flagged if deferring and not _reports_zero_tools(server) line is already on main unchanged.
git diff --name-only $(git merge-base origin/main 0ef776ec03)...0ef776ec03 lists only src/kiro_crew/instances/ssh_tunnel_manager_child.py and test/test_instances.py, so this PR has nothing to move to a separate PR.

@mbriones98

Copy link
Copy Markdown
Contributor Author
  • PASS punchline: confirm the change this PR makes is only two files

confirmed: against the merge base with main, the change touches only src/kiro_crew/instances/ssh_tunnel_manager_child.py and test/test_instances.py (2 files, +138/-1).
The other files in the patch view come from the origin/main merge commit and match main.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 17, 2026
A Remote Crew connect that ran out of its readiness budget reported a
bare "timed out after 15s waiting for forward" regardless of why. On a
host reached through a jump host or proxy the handshake can consume most
of the default 15s budget, yet the message named neither a likely cause
nor the setting that raises the budget.

When the tunnel child is still running at the deadline, the message now
says the connection was still being established, names both plausible
causes (a slow handshake or an unreachable host) and the
instances.connect_timeout_secs tunable. The wording follows the
transport: SSH mentions the jump host or proxy handshake, SSM mentions
the session handshake and the target's SSM agent. A budget already at
the ceiling points at reachability instead of suggesting a raise. A
child that already exited keeps the generic wording. The 15s default is
unchanged.
@iamwhatever
iamwhatever force-pushed the fix/remote-crew-connect-timeout-proxy-handshake branch from 0ef776e to 1ad6f22 Compare October 23, 2026 01:24
@iamwhatever

Copy link
Copy Markdown
Contributor

Rebased onto latest main as 1ad6f22 (one commit, author kept).

Reply to the CHANGES_REQUESTED review (ca93edf), point by point:

  1. Link fix(instances): name instances.mint_timeout_secs when a Remote Crew token mint times out #19882 under Related Issues: done. The description has "Follow-up: fix(instances): name instances.mint_timeout_secs when a Remote Crew token mint times out #19882". fix(instances): name instances.mint_timeout_secs when a Remote Crew token mint times out #19882 was closed by merged fix(instances): name instances.mint_timeout_secs on a mint timeout #20381.

Rebase notes:

  • main split ssh_tunnel_manager.py and test/test_instances.py. The change now sits in src/kiro_crew/instances/ssh_tunnel_manager_child.py (same _ready_timeout_error, unchanged text), and the four tests moved to test/test_instances_self_heal.py in TestSelfHealRefreshRestart.
  • The diff against main is still only these two files (+138/-1).
  • Local: test_instances_self_heal.py, test_ssh_tunnel_manager_cov80.py, test_instances.py: 136 passed. black clean.

The old review is on a previous head; a maintainer would need to dismiss it or re-review.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Oct 23, 2026
The desktop tunnel keeper builds _SshTunnel without the setting, so its readiness timeout must not tell the user to raise it. The manager marks the tunnels it builds from instances.connect_timeout_secs; any other tunnel points at host reachability instead.
@iamwhatever

Copy link
Copy Markdown
Contributor
  • BLOCKING ssh_tunnel_manager_child.py:503 — keeper is told to raise a setting it never reads

fixed in a2de07b: _SshTunnel gets a _timeout_tunable seam, False by default, assigned after construction like _on_healthy (so tunnel doubles need no new kwarg). SshTunnelManager sets it True in connect() (ssh_tunnel_manager_lifecycle.py) and in the self-heal rebuild (ssh_tunnel_manager_self_heal.py), the two places whose budget comes from instances.connect_timeout_secs. tunnel_keeper.keep_tunnel builds a bare tunnel, so its timeout keeps the cause text and says "Check that the host is reachable and connect again." with no setting named.
Tests: test_ready_timeout_on_an_untunable_tunnel_names_no_setting (ssh and ssm) and test_a_bare_tunnel_is_untunable_and_the_manager_marks_its_own (bare tunnel False; after connect() and after _recover() True).

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 23, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD at a2de07b. The fix is sound; two AI items on this head need a reply.

The new _timeout_tunable seam is correct: only the two manager-built tunnels, whose budget comes from instances.connect_timeout_secs, are told to raise it (lifecycle.py:471, self_heal.py:256), and every branch of _ready_timeout_error has a test. Our earlier change request is done: #19882 is linked and closed.

What holds it:

  1. First Principles item 4: the description never mentions _timeout_tunable or the branch that names no setting. Please add a row to the table in What changed for a tunnel not built from instances.connect_timeout_secs (the desktop tunnel keeper).
  2. Opus note: that branch says "connect again" (L507-L508), but the tunnel keeper already retries by itself. Drop the phrase, or reply on the PR explaining why it stays.

Once both are answered on this head, nothing else blocks approval. CI is green.

— crew-pr-reviewer (r125)

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving a2de07b.

_ready_timeout_error gives a live-child timeout a cause and a next step, keeps the generic text for an exited or cleared child, and names instances.connect_timeout_secs only for the two manager-built tunnels whose budget comes from that setting (lifecycle.py#L471, self_heal.py#L256). _timeout_tunable defaults to False, so any other tunnel names no setting. Each branch has its own test. Our earlier change request is done: #19882 is linked and closed. CI is green on this head (90 success, 16 skipped).

AI findings judged non-blocking

  • First Principles, item 4: the description does not mention _timeout_tunable or the no-setting branch. This is description completeness only; the code is correct and tested (child.py#L507-L508).
  • Opus 5.5 note: the keeper branch says "connect again" although the keeper retries. This is a wording nit. The text only appears inside the keeper's own tunnel: down (...); retrying in Ns line, which states the retry (tunnel_keeper.py#L110-L113).
  • Design note: _timeout_tunable could be a constructor argument. This is a style suggestion; the False default already fails safe (child.py#L213).

— crew-pr-reviewer (r132)

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved on behalf of the watch holder (bolichen). Design and First Principles pass on this head (or skipped / author override), UX skipped or pass, no conflict, no open change request.

@KiroCrewCaptain
KiroCrewCaptain merged commit 5deb30a into main Oct 29, 2026
156 of 157 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/remote-crew-connect-timeout-proxy-handshake branch October 29, 2026 20:01
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants