[Serve] Refresh active_versions when update_version reuses replicas

[Hoipang]

Problem

A count-only sky serve update (only the service: section changes, e.g. replicas: 1 → 2) can take every ready replica out of the load balancer for up to ~20 s. During that window the load balancer answers 503 No ready replicas even though the replica is READY and serving, and some in-flight requests hang until an upstream proxy times them out.

This contradicts the rolling-update guarantee (old replicas keep serving until new ones are ready). That guarantee holds for spec changes; it breaks only in the replica-reuse shortcut for count-only changes.

TokenVisor hits this on every SkyPilot replica-count change: manual edits, and every autoscaler scale event (TokenVisor PR EmbeddedLLM/TokenVisor#807 calls sky.serve.update(task, name, UpdateMode.ROLLING) with only the replica count changed).

Evidence (GPU cluster, 2026-10-06, image ghcr.io/embeddedllm/skypilot:v0.12.0, commit f579438 = ancestor of ellm-0.12.0)

One SkyPilot service, Qwen3-0.6B on an RTX A4000, open-loop load at 7 requests/s, autoscaled 1 → 2 at 14:39:34 UTC:

Time (UTC)	What happened
14:39:34	sky serve update (count 1 → 2). Replica 1 READY and serving.
14:39:50–14:40:10	Load balancer returned 503 No ready replicas 78 times (41 + 37 per 10 s). Replica 1 stayed READY throughout.
14:39:53–14:40:05	104 requests whose attempt started in this window got no response at all; the gateway in front cut them at its 300 s stream-idle timeout.
14:40:10 onwards	Traffic normal. Second replica READY at 14:53 (slow start, 2-CPU pod).
15:14:34	Scale 2 → 1: no errors.

The 503 window is exactly one load-balancer sync interval (20 s).

Root cause (proven from source; identical on skypilot-org/skypilot master)

1. The controller's /controller/update_service handler calls update_version synchronously (sky/serve/controller.py:184).
2. For a count-only change, SkyPilotReplicaManager.update_version (sky/serve/replica_managers.py:1474) reuses each running replica whose config matches, by rewriting its stored version: info.version = version + serve_state.add_or_update_replica(...).
3. The load balancer's ready list comes from get_active_replica_urls (replica_managers.py:1456), which returns READY replicas whose version is in the service record's active_versions.
4. active_versions is only recomputed by set_service_status_and_active_versions_from_replica (sky/serve/serve_utils.py:492), called from the replica prober loop (replica_managers.py:1443) every ENDPOINT_PROBE_INTERVAL_SECONDS = 10 s, after probing every replica with up to DEFAULT_READINESS_PROBE_TIMEOUT_SECONDS = 15 s each.
5. So between step 2 and the next prober pass, the reused replica is labelled v2 while active_versions is still [1] → it is filtered out → the ready list is empty. On a busy replica (slow readiness probe) that gap is ~10–25 s.
6. The load balancer pulls the list every LB_CONTROLLER_SYNC_INTERVAL_SECONDS = 20 s (sky/serve/load_balancer.py:88). If a pull lands in the gap, it keeps the empty list for a full 20 s. If no pull lands in it, nothing happens, which is why this is intermittent and why it shows up mostly under load.

Spec changes are not affected: replicas are not reused, they keep their old version, and active_versions (rolling = "versions with a ready replica") keeps them in rotation.

This PR

Right after the reuse loop in SkyPilotReplicaManager.update_version, recompute active_versions with the same function the prober uses, so the relabel and the active set change together:

serve_utils.set_service_status_and_active_versions_from_replica(
    self._service_name,
    serve_state.get_replica_infos(self._service_name), update_mode)

• Rolling: active = versions of ready replicas → includes the reused v2 replica immediately.
• Blue-green: get_latest_version_with_min_replicas picks the latest version meeting min_replicas, else the oldest version with a ready replica. After a reuse, v2 is the only version with ready replicas, so it is chosen.
• Idempotent with the prober: same function, same inputs; it also respects the SHUTTING_DOWN guard inside it.
• The early return for non-empty file_mounts is untouched (no reuse happens on that path, so no gap).

Open work for whoever picks this up

• Unit test: with a mocked serve_state, a READY v1 replica, call update_version(2, …) for a count-only change and assert get_active_replica_urls() still returns that replica, for both UpdateMode.ROLLING and UpdateMode.BLUE_GREEN. Also assert a spec change (no reuse) keeps v1 active.
• Residual window: two separate DB commits remain (relabel, then active set). A load-balancer pull between them still sees an empty list, now for milliseconds instead of ~20 s. Decide whether to close it fully, e.g. widen active_versions to include the new version before relabelling, then recompute after.
• The 104 hung requests (not proven): likely load-balancer side. When the ready list empties, _sync_with_controller_once pops and closes that replica's httpx.AsyncClient (load_balancer.py:128–136); when it returns, a new client is created (:126) and every queued/retried request hits it at once. Default httpx pool is 100 connections, LB_STREAM_TIMEOUT = 120 s applies to the pool wait too, and LB_MAX_RETRY = 3, so the tail can wait > 300 s. The LB logs were gone before we looked. Reproduce once with sky serve logs <service> --load-balancer captured to confirm. With this PR the client should no longer be dropped on a count change, so the hang may disappear with it; consider separately bounding total LB retry time and not closing a client whose URL disappears for one sync.
• Reproduce before/after: any count-only sky serve update on a service under steady traffic. It does not need GPUs: a CPU service (e.g. a tiny HTTP server with a readiness path) on Kubernetes, ~5–10 requests/s against the LB endpoint, then sky serve update with only replicas changed, repeated a few times (the bug needs an LB sync to land in the gap). Count 503 No ready replicas and requests > 60 s. Expect some before, none after.
• Ship to TokenVisor: the serve controller and load balancer run on the serve-controller pod, which TokenVisor launches through the API server image ghcr.io/embeddedllm/skypilot:v0.12.0 (k8s/manifest/skypilot-api-server-deploy.yaml, skypilot-helm-values.yaml). emu also pins a client wheel (tokenvisor-16042fbf release in services/api/pyproject.toml). Confirm which artifact the controller pod actually runs, rebuild it, and re-run the TokenVisor SkyPilot autoscaling e2e (suite D in the GPU trial report) to verify zero failures during 1 → 2.
• Upstream: the same code is on skypilot-org/skypilot master; open the same fix there once tested.

[Hoipang]

A/B test on the GPU cluster (2026-10-06): fix confirmed

Two runs, identical except for the API server image:

• old: ghcr.io/embeddedllm/skypilot:v0.12.0
• fix: ghcr.io/embeddedllm/skypilot:v0.12.0-pr8-7f89638, which is v0.12.0 plus only this PR's change to sky/serve/replica_managers.py. The serve controller picked up the patched wheel; grep in its site-packages showed the fix line, 0 before and 1 after.

Setup (no GPUs needed):

• A CPU SkyServe service on Kubernetes: a stdlib HTTP server that returns its hostname. /health sleeps 8 s to mimic a busy replica's slow readiness probe.
• Started at 2 replicas, with 20 requests/s against the load balancer.
• 10 count-only sky serve update --mode rolling calls alternating replicas: 3 and replicas: 2, 2 min apart.
• A poller inside the controller pod read serve_state every 0.5 s and recorded active_versions, the READY replicas with their versions, and how many of them get_active_replica_urls would return.

	old	fix
Stale window in the controller (READY replicas, 0 routable)	every update, 4.0–7.5 s each, 60.7 s total	0 s on all 10
Load balancer synced inside the window	3 of 10 updates	–
Client outage when it did	20.1–20.2 s with no successful reply, each time	none
503 No ready replicas	1,045	0
Requests that hung to the 60 s client timeout	160 (49–58 per outage)	0
Successful requests	25,495 / 26,700	26,700 / 26,700
Replicas launched over the 10 updates	27: every replica replaced on every update, ~2.4 min lifetime	7: replicas 1 and 2 served all 22 min; only the 3rd replica came and went with the count

Sample stale state on the old image: {"active": [1], "ready": [[1, 2], [2, 2]], "routable": 0}. Both replicas were READY and relabelled to v2, but active_versions still said [1].

Findings beyond the original report:

• The hang is reproduced. Requests sent during the empty window hung until the client timeout, with no 503 returned. This matches the 104 hung TokenVisor requests, and it disappears with the fix. The load-balancer-side mechanism is still unconfirmed.
• The old image churns replicas. On every count-only change it replaced all running replicas instead of reusing them. With the fix, the existing replicas keep serving across count changes as intended. The likely cause is the controller treating the briefly non-active v2 replicas as outdated, but this is observed, not traced.
• One run caveat. The first old run (no poller) showed one 20.2 s outage in 10 updates, with 351 × 503 and 51 hung requests, consistent with the polled run.

Raw data on amd-1: ~/hp/skyab/ (old2.*, new.*; .csv per request, .poll controller samples, .events timeline). Scripts: run_part3.sh, poller.py, analyze_poll.py, svc.yaml, client.py.

Still open from the checklist: unit tests, the residual millisecond window between the two DB writes, upstream PR, and shipping the image to TokenVisor clusters. The cluster was restored to v0.12.0 after the test.