Repository navigation
fix(instances): make a proxy-slow Remote Crew connect timeout actionable - #19615
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks: every item changes only the readiness-timeout text, and the setting it names already exists. What this change shipsInventory (5 items) — 4 justifiedIntent: when a Remote Crew connection through a jump host or proxy times out, tell the user why and which setting to raise. FIX.
[FIRST-PRINCIPLES-REVIEWED] a2de07b Description read[DESCRIPTION-READ] 5257755c2cdb47b8fc3bab20845994ccee02b927c6834a4e0cb5ba1d85e86de6 That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the change is backend error wording only. No code anywhere parses the old Notes
[DESIGN-REVIEWED] a2de07b Description read[DESCRIPTION-READ] 795f4c07b7547f46f2542c3c11565d0a9a0e94ce8639c57111ac5041fe5549f3 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Review detailsNo findings. Notes:
[OPUS-REVIEWED] a2de07b Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
ed6ab8d to
ca93edf
Compare
|
|
|
|
|
|
|
iamwhatever
left a comment
There was a problem hiding this comment.
This replaces the fixed timed out after Ns waiting for forward string in _SshTunnel._wait_until_ready with _ready_timeout_error (src/kiro_crew/instances/ssh_tunnel_manager.py:1153), which checks the child's liveness and transport. When the child is still running, the message names the likely causes and the instances.connect_timeout_secs setting. At the 120 s CONNECT_TIMEOUT_CEILING_SECS it points at reachability instead, and an exited or cleared _proc keeps the generic text. That fixes the real gap: the message used to name neither the cause nor the setting. I checked that _transport and _connect_timeout are set in __init__, that the ceiling constant and the kirocrew config set instances.connect_timeout_secs command it points at both exist, that the timed out after prefix is kept on every branch, and that the four tests cover each branch without timing assertions. All 100 check runs are success or skipped, and every AI finding has a disposition. The code is correct, but there is one follow-up item: the First Principles mint-timeout sibling (span eabdea71b2c4, token_mint.py:583 and ssm_token_mint.py:185 still name no instances.mint_timeout_secs) was deferred to #19882. The PR description's Related Issues section still says "N/A". Please link #19882 there (for example Part of #19882 or a "Follow-up: #19882" line), and I will approve on the next round. — crew-pr-reviewer (deep re-check)
|
Linked #19882 under Related Issues. |
|
Holding on 0ef776e. The fix is sound, and the item from our earlier change request is done: #19882 is linked and was closed by merged #20381. Blocker: two AI items on this head have no reply.
Once both are answered, nothing else blocks approval on this head. Checks are green, and — crew-pr-reviewer (r108) |
|
|
A Remote Crew connect that ran out of its readiness budget reported a bare "timed out after 15s waiting for forward" regardless of why. On a host reached through a jump host or proxy the handshake can consume most of the default 15s budget, yet the message named neither a likely cause nor the setting that raises the budget. When the tunnel child is still running at the deadline, the message now says the connection was still being established, names both plausible causes (a slow handshake or an unreachable host) and the instances.connect_timeout_secs tunable. The wording follows the transport: SSH mentions the jump host or proxy handshake, SSM mentions the session handshake and the target's SSM agent. A budget already at the ceiling points at reachability instead of suggesting a raise. A child that already exited keeps the generic wording. The 15s default is unchanged.
0ef776e to
1ad6f22
Compare
|
Rebased onto latest main as 1ad6f22 (one commit, author kept). Reply to the CHANGES_REQUESTED review (ca93edf), point by point:
Rebase notes:
The old review is on a previous head; a maintainer would need to dismiss it or re-review. |
The desktop tunnel keeper builds _SshTunnel without the setting, so its readiness timeout must not tell the user to raise it. The manager marks the tunnels it builds from instances.connect_timeout_secs; any other tunnel points at host reachability instead.
|
|
HOLD at a2de07b. The fix is sound; two AI items on this head need a reply. The new What holds it:
Once both are answered on this head, nothing else blocks approval. CI is green. — crew-pr-reviewer (r125) |
iamwhatever
left a comment
There was a problem hiding this comment.
Approving a2de07b.
_ready_timeout_error gives a live-child timeout a cause and a next step, keeps the generic text for an exited or cleared child, and names instances.connect_timeout_secs only for the two manager-built tunnels whose budget comes from that setting (lifecycle.py#L471, self_heal.py#L256). _timeout_tunable defaults to False, so any other tunnel names no setting. Each branch has its own test. Our earlier change request is done: #19882 is linked and closed. CI is green on this head (90 success, 16 skipped).
AI findings judged non-blocking
- First Principles, item 4: the description does not mention
_timeout_tunableor the no-setting branch. This is description completeness only; the code is correct and tested (child.py#L507-L508). - Opus 5.5 note: the keeper branch says "connect again" although the keeper retries. This is a wording nit. The text only appears inside the keeper's own
tunnel: down (...); retrying in Nsline, which states the retry (tunnel_keeper.py#L110-L113). - Design note:
_timeout_tunablecould be a constructor argument. This is a style suggestion; the False default already fails safe (child.py#L213).
— crew-pr-reviewer (r132)
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Approved on behalf of the watch holder (bolichen). Design and First Principles pass on this head (or skipped / author override), UX skipped or pass, no conflict, no open change request.
Problem / Motivation
Goal: Make a Remote Crew connection that times out while still handshaking through a proxy tell the user how to fix it, instead of a dead-end error.
When a Remote Crew connect runs out of its readiness budget, the user sees a flat
timed out after 15s waiting for forwardregardless of why. On a host reached through aProxyCommand(jump host, WSSH, corporate proxy) that message is actively misleading: the proxy handshake alone routinely consumes most of the default 15s budget before ssh even begins binding the local forward, so the connection is still being established when the deadline fires. The user is left with no cause and no next step.Why it matters
Anyone connecting a Remote Crew through a jump host or corporate proxy — a common setup — hits this. The timeout is the binding constraint for those hosts, and the knob that fixes it (
instances.connect_timeout_secs) already exists, but the error names neither the cause nor the knob, so the user can only retry into the same wall. The result is a feature that looks broken for proxy users when it is one config value away from working.Not a goal
connect_timeout_secs(15s). The default stays tuned for the fast direct case so a genuinely dead host still fails fast; proxy-slow hosts are served by the actionable message pointing at the existing tunable.What changed (motivation → approach → change)
Observed symptom: a proxy-bound connect reports an opaque
timed out ... waiting for forward. Root cause:_SshTunnel._wait_until_readyemits one fixed string for every timeout, with no cause and no next step.Approach: when the tunnel child is still running at the deadline, it has not failed with a classifiable error. It is either still connecting or stuck on a host that never answers, and the two look the same locally, so the message names both and the setting that raises the budget instead of asserting either cause.
Change: a new
_ready_timeout_errorininstances/ssh_tunnel_manager_child.pycomposes the message for the active transport.instances.connect_timeout_secsinstances.connect_timeout_secsCONNECT_TIMEOUT_CEILING_SECSstop()timed out after <N>s waiting for forward(unchanged)Backwards compatibility
Compatible: no field, validator, type, kind or key changes. The only behavioral change is the text of
TunnelStatus.erroron the readiness-timeout path; every branch keeps thetimed out after <N>sprefix.Removes nothing: no path is hidden, no record/row deleted, no validator tightened, no key or function removed. The generic timeout string is preserved verbatim for the already-exited-child case, and no test asserts the exact readiness-timeout wording.
Tests
test_wait_until_ready_timeout_with_live_child_names_the_tunable— an ssh timeout with a live child names both causes andinstances.connect_timeout_secs, and no SSM wording.test_wait_until_ready_timeout_on_ssm_names_the_ssm_session— an SSM timeout with a live child names the SSM session and agent, with no ssh or proxy wording.test_ready_timeout_at_ceiling_does_not_suggest_raising_it— a budget at the ceiling omits the tunable and points at reachability.test_wait_until_ready_timeout_with_dead_child_stays_generic— an already-exited child keeps the generic wording.The tests use fake child objects and a 0.05s budget with no wall-clock assertions; the tests live in
test/test_instances_self_heal.py; after the rebase onto main,test_instances_self_heal.py,test_ssh_tunnel_manager_cov80.pyandtest_instances.py(136 tests) pass, and the related backend set (1477 tests),black,isort,flake8,mypyand the comment-history gate are clean.Manual verification
N/A — unit coverage exercises both timeout branches directly; the change is a backend error string with no interactive path.
Screenshots / video
N/A — the diff touches no user-visible frontend surface; it changes a backend status/error string only.
Related Issues
Pattern harvest
Pattern: one error string reused across two root causes (a dead link vs. a slow-but-live connect) hides the one fact that tells them apart. When a timeout can fire for more than one reason, the message must surface the distinguishing fact and the knob that addresses the common case, rather than a single opaque sentence.
Rule candidate: review-prompt — flag a readiness/timeout path that emits a fixed error string without consulting the child/process liveness state that distinguishes "failed" from "needs more budget."
Checklist