Skip to content

Retry AVF runner connects refused by a full backlog - #9

Merged
einarfd merged 1 commit into
mainfrom
avf-socket-connect-retry
Oct 14, 2026
Merged

einarfd merged 1 commit into
mainfrom
avf-socket-connect-retry

Conversation

@einar-kindly

Copy link
Copy Markdown
Collaborator

Problem

After upgrading to macOS 27, agv start on an AVF VM with 7 [[forwards]] logged:

WARN agv::vm: could not resolve SSH endpoint for managed SSH config vm=kindly
error=failed to connect to AVF runner socket .../avf-control.sock: Connection refused (os error 61)

The VM itself was fine and agv ssh worked, but the managed ssh_config was left empty, so plain ssh <vm> (and IDE remote access) didn't.

Cause

At the end of agv start, agv spawns one forward supervisor per forward and then calls update_ssh_config. All of them call the runner's status op at roughly the same moment: 8 connects. The runner listened with listen(fd, 4), and macOS answers a full unix-socket backlog with an immediate ECONNREFUSED rather than queueing the connect (measured on macOS 27: a backlog of N accepts exactly N pending connects). The supervisors retry on their own, so forwards came up, but update_ssh_config is one-shot and lost the race.

Why it surfaced now isn't proven: likely a timing shift or a change in how the kernel counts the backlog on macOS 27. The race was latent either way.

Fix

  • src/vm/backend.rs: avf_rpc connects through a new connect_avf_socket, which retries on ECONNREFUSED with a short backoff (50 → 800 ms, ~1.5 s total), so a genuinely dead runner still fails promptly.
  • swift/avf-runner/.../main.swift: the control socket listens with SOMAXCONN instead of 4.

RUNNER_PROTOCOL_VERSION is deliberately not bumped: the client-side retry fixes this against any runner, old or new, so agv doesn't depend on the larger backlog.

Testing

  • cargo test passes; cargo clippy is clean; the Swift runner builds.
  • Running on the affected VM (macOS 27, AVF, 7 forwards) for several days: no warning on start, and plain ssh <vm> works.
  • Slow boot suite not run.
  • Note: cargo clippy --all-targets -- -D warnings fails locally on three AVF test files (unfulfilled_lint_expectations for map_unwrap_or), identically on main without this change. It looks like a local clippy 1.94 vs CI 1.99 difference, so CI is the authority there.

🤖 Generated with Claude Code

agv start spawns one forward supervisor per [[forwards]] entry, and
each calls the runner's status op at once, right before
update_ssh_config does the same. The runner listened with a backlog of
4, and macOS answers a full unix-socket backlog with an immediate
ECONNREFUSED instead of queueing. With 7 forwards the one-shot SSH
config update lost the race, leaving the managed ssh_config empty so
only `agv ssh` could reach the VM.

avf_rpc now retries a refused connect with a short backoff (~1.5s in
total, so a dead runner still fails promptly), and the runner listens
with SOMAXCONN.

RUNNER_PROTOCOL_VERSION is deliberately not bumped: the client-side
retry fixes this against any runner, so agv does not depend on the
larger backlog.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@einarfd
einarfd merged commit 2338a79 into main Oct 14, 2026
4 checks passed
@einarfd
einarfd deleted the avf-socket-connect-retry branch October 14, 2026 20:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants