Take a restarting Kibana replica out of the proxy rotation

[sscarduzio]

What this fixes

The 9.4.6, docker leg has been failing on nearly every pull request, on different specs each time, and passing on a re-run. I treated it as a flaky spec. It is not — it is the proxy in front of Kibana.

The mechanism

environments/elk-ror/base.docker-compose.yml runs two Kibana replicas, and kbn-proxy round-robins across them. Kibana-config.cy.ts changes ROR settings, which restarts Kibana — restart: always on kbn-ror carries a comment saying that is exactly why it is there.

While a replica restarts, half of every request goes to a container that is not listening. A stopped container drops the packets rather than refusing them, so those requests hang instead of failing. The old config set no proxy_connect_timeout, so nginx waited its default 60 seconds — longer than every Cypress timeout (defaultCommandTimeout 20s, requestTimeout 10s). The browser gave up first, the failure never reached nginx to be counted against the peer, and the peer stayed in the rotation.

Cypress's retries: 2 did not help. Three attempts, same alternation.

That accounts for every symptom:

Observed	Cause
Different specs fail each run	Depends which request lands on the dead peer
Fails all 3 Cypress retries	The peer is never ejected
Passes on a full job re-run	Fresh stack, both replicas alive
docker legs only, never eck	eck runs one Kibana (kind-cluster/ror/base/kbn.yml, count: 1) with no proxy

Sample failures, all timeouts on unrelated things:

AssertionError: Timed out retrying after 20000ms: Expected to find element:
  `[data-test-subj="lnsVisualizationContainer"]`, but never found it.
CypressError: Timed out retrying after 10000ms: cy.wait() timed out waiting 10000ms
  for the 1st request to the route: `spacesPlugin`. No request ever occurred.
CypressError: `cy.task('httpCall')` timed out after waiting `20000ms`.

The change

Three directives on the location, and the default max_fails/fail_timeout written out on the peer so the intent is on the page:

• proxy_connect_timeout 2s — the important one. It turns a hung connect into a countable failure long before the client gives up. Two seconds is far above a healthy connect on the container network and far below the 10s Cypress spends in cy.wait().
• proxy_next_upstream error timeout http_502 http_503 http_504 — send the request to the other replica. http_503 is in the list because a Kibana that is listening but still starting answers 503, which nginx does not treat as an error by default.
• proxy_next_upstream_tries 2 — there are two replicas.

Measured

Two-replica reproduction, one replica stopped, same config shape:

before: 000 200 000 200 000 200 000 200 000 200     50% lost, each hanging until the client's limit
after:  200/2.002s 200/0.0007s 200/0.0005s ...      one failover, then the peer is out of rotation

Sustained for 30s with the peer down: 110 requests, 0 failures, 3 over 0.5s (the 10s re-probe), worst case 2.0s — inside every Cypress timeout. When the replica answers again nginx takes it back on its own, verified.

nginx -t passes on the real file.

What I did not test

The http_503 path is reasoned, not measured: I reproduced the dropped-packet case, which is what a restarting Kibana does, and not the "listening but answering 503" case. It is a standard directive and costs nothing if that state never occurs.

I also left proxy_read_timeout and proxy_send_timeout at their 60s defaults. Lowering them would catch a replica that hangs after accepting the request, but Reporting.cy.ts can legitimately wait a long time on a response, and I would rather not trade this flake for a new one.

Base branch

master, per docs/dev/branching.md case 3 — a fix for a test that fails against the released plugins, and a flaky test counts.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Bug Fixes
  • Improved proxy resilience when connecting to Kibana replicas.
  • Requests retry across replicas after connection errors, timeouts, or temporary server errors.
  • Replicas recovering from brief restarts remain available for requests instead of being excluded temporarily, reducing avoidable 502 responses.

[coutoPL]

LGTM

Please take a look at proxy example in this repo. Maybe we should port the changes to it?