Skip to content

MOD-19933: Retry transient SSH connection failures in connect_remote_ssh - #570

Merged
fcostaoliveira merged 4 commits into
redis-performance:masterfrom
gabsow:fix/mod-19933-setup-ssh-retry
Oct 20, 2026
Merged

fcostaoliveira merged 4 commits into
redis-performance:masterfrom
gabsow:fix/mod-19933-setup-ssh-retry

Conversation

@gabsow

@gabsow gabsow commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

A brief SSH outage on an otherwise healthy host (NoValidConnectionsError: Unable to connect to port 22) failed a test outright in the RedisJSON nightly, even though the previous and next benchmarks on the same host ran fine (MOD-19933).

This retries transient failures when opening an SSH connection in connect_remote_ssh. No remote command has run at that point, so a retry can never replay a setup step or a benchmark measurement, and every SSH caller (server setup, tool setup, benchmark, post-processing) is covered, not just tool setup.

  • Retries TRANSIENT_SSH_ERRORS (refused/timed-out connections, SSH banner errors), reusing the set and the 2**attempt backoff from the result-fetch retry (Retry transient SSH/SFTP errors when fetching remote benchmark files #565). AuthenticationException is re-raised immediately.
  • SSH_CONNECT_MAX_RETRIES (default 5): up to 6 attempts, waiting 1, 2, 4, 8, 16 seconds.
  • SSH_CONNECT_TIMEOUT (default 30s) bounds each attempt (TCP connect and SSH banner). Refused connections fail immediately and only pay the 31s backoff; a truly unreachable host fails after about 3.5 minutes instead of waiting for the OS TCP connect timeout on every attempt.
  • Errors after the connection is open (e.g. during the benchmark) propagate as before.
  • remote_client.py is unchanged, so this doesn't conflict with the setup_remote_benchmark_tool refactor from MOD-19727 Add concurrent background benchmark client #568.
  • README documents both settings alongside SSH_FETCH_MAX_RETRIES.

The RedisJSON-side change suggested in the ticket (retrying errored runs in compare_local.py's record_attempt) is separate and not part of this PR.

Validation

  • tests/test_remote_connect_retry.py: first-try success, per-attempt timeout passed to connect(), retry after refused / timed-out / banner errors, giving up after the budget (every client closed, 1/2/4s backoff), no retry on authentication failure, and a failing remote command is not re-run.
  • Existing tests/test_remote_fetch_retry.py and tests/test_run_ssh.py pass; black and flake8 clean.

Jira: MOD-19933

🤖 Generated with Claude Code

- Bound setup-only NoValidConnectionsError retries to three with backoff so a brief SSH outage does not discard a benchmark before measurement.

- Keep authentication, command and measurement failures fatal; cover retry exhaustion and phase boundaries in tests.
@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

🤖 Automated first-pass review — a human maintainer's review is still required before merge.

Thanks for this, and for opening with a regression that fails on master. Moving the retry from remote_tool_pre_bench_step down into connect_remote_ssh is the better shape. Nothing has run on the remote host yet when the connection is retried, so a retry can't replay a setup step or a measurement. Every SSH caller gets the fix, not just tool setup. Reusing TRANSIENT_SSH_ERRORS and the 2**attempt backoff from the fetch retry keeps the two paths consistent.

What I checked: except paramiko.AuthenticationException comes before except TRANSIENT_SSH_ERRORS. That order matters because AuthenticationException is a subclass of SSHException, and the tests cover it. A fresh SSHClient is created and closed on every attempt, so a half-open client never gets reused. The key file loads outside the loop, so a missing key isn't retried as an OSError. test_remote_command_is_not_replayed_when_it_fails pins down the claim that only connection setup is retried.

A few things, most important first:

  1. The PR title and description describe the first commit, not the code that will merge. They say "retry only NoValidConnectionsError around [remote_tool_pre_bench_step]", "three retries with 1/2/4-second backoff", and list tests/test_remote_setup_retry.py / tests/test_remote_client.py as validation. The final diff does something different:

    • it retries in connect_remote_ssh
    • it retries all of TRANSIENT_SSH_ERRORS (including bare OSError and SSHException)
    • it defaults to 5 retries plus a 30s per-attempt timeout
    • the only test file is tests/test_remote_connect_retry.py

    The later commit messages explain the change well. Please update the title and body so the squash-merge message and release notes say what actually shipped. I'd fix this before merge.

  2. Worst-case wall clock (I checked the README's numbers). Against a host that drops packets: 6 attempts × 30s + (1+2+4+8+16)s of backoff = 211s, about 3.5 minutes, as documented. Adding the timeout was the right call: without it, each attempt waits for the OS TCP connect timeout (about 127s on Linux), so 6 attempts would take about 13 minutes. Two things to keep in mind:

    • That budget is paid per call, and there are ~46 execute_remote_commands call sites. Some cleanup-on-error paths call SSH again after a failure. For example, the except Exception cleanup at run_remote/standalone.py:923-936 reconnects to the same host. So a host that is really dead now takes about 7 minutes to fail there instead of failing once. Not a blocker, but the nightly CI will feel it if a host is truly gone.
    • Is a default of 5 retries right for every caller, given that MOD-19933 was a brief outage? The setting can be changed with SSH_CONNECT_MAX_RETRIES, so this is about the default, not missing configurability.
  3. Retrying the whole TRANSIENT_SSH_ERRORS tuple is broader than "connection refused". It also catches socket.gaierror (DNS failure, an OSError subclass) and any non-auth SSHException raised during the handshake. Both will now take the full backoff before failing. That's probably fine, since this repo connects by IP. A short note in the docstring would help whoever reads this next.

Small thing: the README paragraph sits between the supported-tools list and ## Installation. It might fit better under a remote/SSH section, if there's a natural place for it.

Cheers — this fixes a real nightly flake. Once the description matches the code, I'd be happy to see it land.

(This would be a COMMENTED review, not a request for changes.)

@codecov

codecov Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 50.10%. Comparing base (5745e56) to head (03f4901).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master     #570      +/-   ##
==========================================
+ Coverage   49.02%   50.10%   +1.08%     
==========================================
  Files          74       74              
  Lines        9065     9170     +105     
==========================================
+ Hits         4444     4595     +151     
+ Misses       4621     4575      -46     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

fcostaoliveira and others added 3 commits October 20, 2026 12:55
- a brief SSH outage ("NoValidConnectionsError: Unable to connect to port
  22") failed a test even though the previous and next benchmarks on the
  same host ran fine (MOD-19933)
- retry at connection establishment instead of around
  remote_tool_pre_bench_step: no remote command has run yet, so a retry
  can never replay setup steps or a measurement, and every SSH caller
  (server setup, benchmark, post-processing) is covered, not just setup
- reuse TRANSIENT_SSH_ERRORS and the 2**attempt backoff from the fetch
  retry (redis-performance#565); budget is env-configurable via SSH_CONNECT_MAX_RETRIES
  (default 5, up to 31s of backoff since the outage length is unknown and
  refused connections fail fast); authentication failures are not retried
- revert the remote_client.py change, which also avoids a conflict with
  the setup_remote_benchmark_tool refactor in redis-performance#568

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…commit

- the previous commit only carried the remote_client.py revert and the
  removal of the setup-level retry test; this adds the actual retry in
  connect_remote_ssh, its tests and the README update it describes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- connect() had no timeout, so against an unreachable host (dropped SYNs)
  every attempt waited for the OS TCP connect timeout (~127s on Linux):
  with 5 retries a dead host took ~13-14 min to fail instead of ~2 min,
  on every SSH caller including teardown
- pass timeout and banner_timeout (SSH_CONNECT_TIMEOUT, default 30s), so
  the worst case is ~3.5 min; refused connections still fail immediately
- document the real worst case instead of backoff-only "31 seconds"

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
attempt = 0
while True:
c = paramiko.SSHClient()
c.set_missing_host_key_policy(paramiko.AutoAddPolicy())
@fcostaoliveira fcostaoliveira changed the title MOD-19933: Retry SSH connection failures during benchmark tool setup MOD-19933: Retry transient SSH connection failures in connect_remote_ssh Oct 20, 2026
@fcostaoliveira
fcostaoliveira merged commit 2154998 into redis-performance:master Oct 20, 2026
8 of 9 checks passed
fcostaoliveira added a commit that referenced this pull request Oct 20, 2026
- release the concurrent background client (#568), the SSH connection
  retry for MOD-19933 (#570) and the CPU consumption report fields (#567)
  to PyPI so downstream CI (e.g. the RedisJSON nightly) can pick them up

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants