Repository navigation
Add opt-in per-server dial backoff - #8
vikram-t-verkada wants to merge 1 commit into
Conversation
When FailureThreshold > 0, a server that fails that many dials in a row is backed off: new dials to it return ErrConnectionBroken immediately until the backoff (BaseBackoff, doubling up to MaxBackoff, with +/-50% jitter) elapses, then a single dial is let through to test it. Pooled connections are unaffected. Disabled by default. Prevents redial storms where every request to an unresponsive node opens a new (TLS) connection with no backoff, which kept memcached nodes pinned at 100% CPU in vaccel's prod1 incidents. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| n.mu.Lock() | ||
| defer n.mu.Unlock() | ||
|
|
||
| n.retryInFlight = false |
There was a problem hiding this comment.
record clears retryInFlight for every dial, but only the test dial should. With DialTimeout (3s in vaccel prod) > BaseBackoff, dials started before the block finish after it expires, release the test-dial slot so more test dials get through, and each of those failures re-extends and doubles the block (line 60). Suggest allow() return whether the caller is the test dial, and record(probe, failed) only clear retryInFlight / extend the block when probe is true. Worth a test for exactly this late-finishing-dial case.
| DefaultBaseBackoff = 1 * time.Second | ||
|
|
||
| // DefaultMaxBackoff is the default cap on a failing server's backoff. | ||
| DefaultMaxBackoff = 60 * time.Second |
There was a problem hiding this comment.
60s max (up to 90s with jitter) is long for a cache: every key on a recovered node misses until the next test dial. Fine as a library default since it's configurable; vaccel should probably set something like 10–15s.
Why
vaccel (vauth's caching proxy) hit repeated prod1 incidents on 2026-10-19/20 where a single memcached node became unresponsive and every vaccel pod redialed it on every request: no backoff, a fresh TLS handshake per attempt, ~1,600 dials/s against one node. The storm kept the node pinned at 100% CPU so it could never recover, and the stuck dials starved the rest of the cluster.
What
Opt-in per-server backoff around
dial()ingetConn:Client.FailureThreshold(0 = disabled, the default) — after this many consecutive failed dials to a server, new dials to it returnErrConnectionBrokenimmediately instead of dialing.BaseBackoff(default 1s) and doubles on each failed retry up toMaxBackoff(default 60s), with ±50% jitter so clients that backed off together don't retry together.With the defaults, a dead server sees roughly one dial per client per minute instead of one per request.
New code lives in
memcache/backoff.go;memcache.goonly gains the error, defaults,Clientfields, and the gate ingetConn.Status
Draft — looking for a direction check before adding tests. Planned tests:
ErrConnectionBrokenafter N refused dialsConsumers: vaccel will opt in via a new option once this merges;
engprod/memcachedis unaffected unless it setsFailureThreshold.🤖 Generated with Claude Code