Repository navigation
MOD-19933: Retry transient SSH connection failures in connect_remote_ssh - #570
fcostaoliveira merged 4 commits into
Conversation
- Bound setup-only NoValidConnectionsError retries to three with backoff so a brief SSH outage does not discard a benchmark before measurement. - Keep authentication, command and measurement failures fatal; cover retry exhaustion and phase boundaries in tests.
|
🤖 Automated first-pass review — a human maintainer's review is still required before merge. Thanks for this, and for opening with a regression that fails on master. Moving the retry from What I checked: A few things, most important first:
Small thing: the README paragraph sits between the supported-tools list and Cheers — this fixes a real nightly flake. Once the description matches the code, I'd be happy to see it land. (This would be a COMMENTED review, not a request for changes.) |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #570 +/- ##
==========================================
+ Coverage 49.02% 50.10% +1.08%
==========================================
Files 74 74
Lines 9065 9170 +105
==========================================
+ Hits 4444 4595 +151
+ Misses 4621 4575 -46 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
- a brief SSH outage ("NoValidConnectionsError: Unable to connect to port
22") failed a test even though the previous and next benchmarks on the
same host ran fine (MOD-19933)
- retry at connection establishment instead of around
remote_tool_pre_bench_step: no remote command has run yet, so a retry
can never replay setup steps or a measurement, and every SSH caller
(server setup, benchmark, post-processing) is covered, not just setup
- reuse TRANSIENT_SSH_ERRORS and the 2**attempt backoff from the fetch
retry (redis-performance#565); budget is env-configurable via SSH_CONNECT_MAX_RETRIES
(default 5, up to 31s of backoff since the outage length is unknown and
refused connections fail fast); authentication failures are not retried
- revert the remote_client.py change, which also avoids a conflict with
the setup_remote_benchmark_tool refactor in redis-performance#568
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…commit - the previous commit only carried the remote_client.py revert and the removal of the setup-level retry test; this adds the actual retry in connect_remote_ssh, its tests and the README update it describes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- connect() had no timeout, so against an unreachable host (dropped SYNs) every attempt waited for the OS TCP connect timeout (~127s on Linux): with 5 retries a dead host took ~13-14 min to fail instead of ~2 min, on every SSH caller including teardown - pass timeout and banner_timeout (SSH_CONNECT_TIMEOUT, default 30s), so the worst case is ~3.5 min; refused connections still fail immediately - document the real worst case instead of backoff-only "31 seconds" Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| attempt = 0 | ||
| while True: | ||
| c = paramiko.SSHClient() | ||
| c.set_missing_host_key_policy(paramiko.AutoAddPolicy()) |
2154998
into
redis-performance:master
A brief SSH outage on an otherwise healthy host (
NoValidConnectionsError: Unable to connect to port 22) failed a test outright in the RedisJSON nightly, even though the previous and next benchmarks on the same host ran fine (MOD-19933).This retries transient failures when opening an SSH connection in
connect_remote_ssh. No remote command has run at that point, so a retry can never replay a setup step or a benchmark measurement, and every SSH caller (server setup, tool setup, benchmark, post-processing) is covered, not just tool setup.TRANSIENT_SSH_ERRORS(refused/timed-out connections, SSH banner errors), reusing the set and the2**attemptbackoff from the result-fetch retry (Retry transient SSH/SFTP errors when fetching remote benchmark files #565).AuthenticationExceptionis re-raised immediately.SSH_CONNECT_MAX_RETRIES(default 5): up to 6 attempts, waiting 1, 2, 4, 8, 16 seconds.SSH_CONNECT_TIMEOUT(default 30s) bounds each attempt (TCP connect and SSH banner). Refused connections fail immediately and only pay the 31s backoff; a truly unreachable host fails after about 3.5 minutes instead of waiting for the OS TCP connect timeout on every attempt.remote_client.pyis unchanged, so this doesn't conflict with thesetup_remote_benchmark_toolrefactor from MOD-19727 Add concurrent background benchmark client #568.SSH_FETCH_MAX_RETRIES.The RedisJSON-side change suggested in the ticket (retrying errored runs in
compare_local.py'srecord_attempt) is separate and not part of this PR.Validation
tests/test_remote_connect_retry.py: first-try success, per-attempt timeout passed toconnect(), retry after refused / timed-out / banner errors, giving up after the budget (every client closed, 1/2/4s backoff), no retry on authentication failure, and a failing remote command is not re-run.tests/test_remote_fetch_retry.pyandtests/test_run_ssh.pypass; black and flake8 clean.Jira: MOD-19933
🤖 Generated with Claude Code