Repository navigation
Drain Server pods cleanly during Kubernetes rolling updates #608
Description
Problem
Rolling updates of Server API pods on Kubernetes can cut in-flight requests and held long polls.
- The Helm chart has a readiness probe on
/api/ready,maxUnavailable: 0and a disruption budget. Neither the chart nor the raw manifests set apreStopdelay orterminationGracePeriodSeconds. - A terminating pod can keep receiving traffic until endpoints update. It gets Kubernetes' default 30-second grace, which equals the default worker long-poll timeout, so held polls and requests can be cut.
- The rolling upgrade guide gives generic load-balancer steps but nothing for draining pods.
Work isn't lost, because leases keep it durable, but clients see connection errors during routine rollouts.
Source-verified, not reproduced.
Acceptance criteria
- The chart sets a configurable
preStopdelay that covers endpoint propagation, and a termination grace period longer than the long-poll timeout plus that delay. Both are exposed in values, with defaults documented. - A rollout test shows no failed client or worker requests during a rolling update of the API pods.
- The deployment guide explains pod draining on Kubernetes.
Activity
rmcdaniel commentedon Oct 21, 2026
Accepted as a Kubernetes shutdown behavior defect with documentation follow-through. Qualify endpoint removal, application shutdown and outstanding polls together in a real rolling update; a manifest-only delay is not sufficient evidence. Make drain delay and grace configurable, validate that grace covers the configured poll duration plus drain time, and keep Helm and raw manifests consistent. Preserve bounded client retry/lease behavior and document the actual defaults in every supported locale.
This follows the active critical PHP image publication (#591). No rollout test, customer impact or source correction is claimed by the source finding alone.
The actual published Server 2.10.0 image metadata advertises
StopSignal=SIGWINCH. Include the API, queue-worker and scheduler roles' real signal/exit behavior in the reproducer, alongside endpoint propagation and held requests. A longer grace period alone does not establish role-correct shutdown. The patched image and Kubernetes runtime behavior still need their own actual qualification; no historical failure is attributed to this metadata observation.rmcdaniel commentedon Oct 27, 2026
Actual published Server 2.11.5 shutdown observations now supplement the earlier image-metadata finding. The immutable image is
ghcr.io/durable-workflow/server@sha256:f4f2ec64a6522ee18a31542a30554873dbaa2f5d3af3ccb52deb914591699c20; API and queue containers both inherit StopSignal=SIGWINCH.During #706 installed acceptance, the API stopped with exit 0 under that default. The idle database queue worker, with zero pending jobs, exceeded an explicit 15-second Docker stop grace and was killed with exit 137/no OOM.
A separate published #923 fixture supplies the role control: explicit SIGTERM stopped both the API and the database queue worker with exit 0/no OOM after both real workflow/activity cancellation cases completed and the queue was empty. This establishes the default queue-role signal mismatch; it does not qualify API draining under held requests, scheduler shutdown, endpoint propagation or a Kubernetes rolling update.
All exact test-labelled containers, networks and database volumes are removed. Keep #608 open. Next: use role-appropriate termination signals in the supported deployments, qualify scheduler behavior and actual API requests/held polls through endpoint removal and rolling termination, then document the measured drain/grace defaults. A blanket image StopSignal change is not justified by the idle API control.
rmcdaniel commentedon Nov 4, 2026
Server open-item pass, 2026-11-04.
Owner: Server for native process/chart drain semantics; General retains legacy PHP and later deployment qualification. The existing SIGTERM queue-role control and idle API control do not prove held-poll draining or endpoint propagation. A blanket StopSignal change is still unjustified.
Next source plan binds endpoint withdrawal, readiness/drain state, outstanding requests and the original lease/poll deadlines to an explicit grace budget. Native must stop a complete Runtime truthfully rather than claim independent PHP role shutdown. Keep actual scheduler continuity/recovery in #325 and the Kubernetes mixed-SDK customer path in #715. General: select the supported topology and future exact-image rollout plan; source render/signal controls can then be prepared without changing production.
rmcdaniel commentedon Nov 11, 2026
Triage 2026-11-11: applies to the native chart: graceful shutdown exists but there is no preStop, so the listener closes before Kubernetes stops routing. Next: preStop sleep of a few seconds plus a render test. Small. Owner: server.
rmcdaniel commentedon Nov 11, 2026
Delivered in Helm chart 1.2.0 (published by helm-chart-release on merge of #1127: lint, kind install and OCI publish/verify all passed). Independently pulled
oci://ghcr.io/durable-workflow/charts/durable-workflow1.2.0: appVersion 3.4.0, and a native render containspreStop: ["/bin/sleep", "5"]withterminationGracePeriodSeconds: 70and imagedocker.io/durableworkflow/server:3.4.0.Sleep before SIGTERM so native rolling updates drop no requests
Merge pull request durable-workflow#1127 from durable-workflow/fix/se…