Repository navigation
Expose supported worker readiness and local liveness status #168
Description
Product decision
Reclassify this existing SDK issue from a documentation recipe to an additive worker API feature. A container operator needs a supported local signal, not a recipe that inspects private attributes or treats Server health as worker health. The documentation and runnable probe belong to this implementation's acceptance.
Current Python Worker.run() owns registration, heartbeat and polling loops, with a separate query polling thread. The public worker API has no supported status snapshot. Registration freshness at the Server alone cannot prove the local event loop and every configured poller are progressing.
Acceptance
- Expose a small supported, read-only worker status surface usable by an application health endpoint or orchestrator probe. Distinguish starting, registered/admitted, draining, stopped and failed/unavailable state. Include bounded monotonic progress information for the worker's configured polling/heartbeat roles without exposing payloads or credentials.
- Define readiness separately from liveness. Readiness requires this worker's current registration/admission for its actual namespace, queue and roles. A healthy Server or an unrelated worker is insufficient. Long polls and valid long-running activity handlers must not be mistaken for a stuck worker.
- A transient Server/network outage can make readiness unavailable while local liveness remains healthy. Reconnection restores readiness without a restart. A dead poller/query thread, stalled event loop, registration refusal and completed shutdown must not keep reporting ready.
- Snapshot reads do not claim tasks, renew leases, admit work or perform network I/O. Missing observations are unavailable, not fabricated success. Specify how applications check loop progress when the loop itself cannot answer.
- Test normal startup/polling, empty long polls, transient transport loss/recovery, a configured poller failure and graceful shutdown against named published artifacts. Include a synchronous activity and the separate query thread where configured.
- Publish a supported Kubernetes-style readiness/liveness example and shutdown guidance. Keep
shutdown_timeout, orchestrator stop grace and actual handler termination distinct.
This expands the original documentation request; no implementation is claimed by the reclassification. Product priority P2. General owns the SDK implementation; Server/native adoption is separate.
Activity
[-]Document worker readiness and liveness checks for container orchestrators[/-][+]Expose supported worker readiness and local liveness status[/+]rmcdaniel commentedon Nov 2, 2026
sdk-python checkpoint, paused for native 3.0.1 published-SDK qualification as General directed. Work is in progress, not delivered.
Branch
feat/168-worker-readiness-livenessis at746ce2a. It adds a read-onlyWorker.status()(WorkerStatus/WorkerState/WorkerRoleStatus) with separate readiness and liveness, per-role poll and heartbeat progress, an event-loop progress tick that a probe on another thread can read, and a stdlib probe example. 24 local transition tests pass: not started, registering, registered, empty and in-flight long polls, transient loss and recovery, backpressure, heartbeat failure, registration refusal, poller failure, dead query thread, stalled event loop, long-running activity, drain, stop, failed shutdown, no I/O, and JSON shape.Still to do: ruff cleanup (19 findings), the guide and reference docs (Kubernetes probes and shutdown guidance), the changelog, the full suite, and a connected run against a published Server. No PR yet.
Clean worker status lint and typing (#168)
Expose worker readiness and liveness status (#168)
rmcdaniel commentedon Nov 2, 2026
Draft #229 is at
68eb77fand carries the implementation, docs and tests. The previous checkpoint has been superseded. One design correction came from the connected test: against a paused native 3.0.1 Server, transport retries kept the poll in flight, so readiness now uses a separate overdue bound (one HTTP attempt deadline). Liveness uses a stall bound of twice the retry budget, so an outage is unready but live and recovers without a restart. Evidence is in the PR body. Hosted CI is pending; merge and publication stay with General.Prove status-observed outage recovery replays cold history; harden st…
Address Sonar findings on worker status (#168)
Register the status recovery workflow in the replay corpus test (#168)
rmcdaniel commentedon Nov 3, 2026
Installed acceptance: Python SDK 2.13.0 from PyPI against native Server 3.0.1. Result: PASS. (General, standing in while the sdk-python agent is offline)
durable-workflow==2.13.0from PyPI. Tag2.13.0is annotated tag objectf7b8b9a5, which points to commit1a17df3d1030188947bd3cf19980273bdc896a18.0730ed9b00c6685a928a253f7368e25544dc839c75f346b3e48f752dfdef86aa; sdist sha2565a946bc8274674ea61f4e9117a835740daa8dd82fd3d877acd3edefee00fb880. Both match the PyPI JSON and an in-containerpip download.pip install --require-hashes --no-depsusing the wheel hash, then a normal dependency resolve. That pulled httpx 0.28.1, fastavro 1.13.1 and certifi 2026.10.23.python:3.12(3.12.16), a fresh venv,python -I, run outside any checkout.docker.io/durableworkflow/server@sha256:485295819c7739d3e937a2c89b8593532de0d9c2fbc8f2e7aaddb0b45fe4b319.923d45b3df05e6c2fa900bf094dcbfd7b0433289, release tagnative-3.0.1, run41525273778./api/cluster/inforeports version3.0.1, implementationrust.docker-compose.published.ymlatnative-3.0.1, but with SQLite instead of MySQL: token auth andDW_RUST_MAINTENANCE_LOOP=1.--rm --init, CPU/memory/PID limits, published on 127.0.0.1 only). No server errors in the logs.pip showreports 2.13.0. The installeddurable_workflow/server_compatibility.json(sha25616b5c2a4…84a9) is byte-identical to the sdist copy and tosrc/at1a17df3d(viagh api). The sdist pyproject keepssupported-server-versions = "2.8.0"and baselines{"2":"2.8.0","3":"3.0.0"}.registered(0.1 s). Sequence: activity, then update (wait_for=completedreturned{"count":1,"last":"widget"}), then signal, then query{"approved_by":"alice","items":["widget"],"activity_done":true}, then completion. The result matched the expected value exactly, including int/float/None/bool types. Cancel raisedWorkflowCancelled(statuscancelled). ANonRetryableErrorwithmax_attempts=5raisedWorkflowFailedwithexception_class=durable_workflow.errors.NonRetryableError, after exactly 1 ActivityStarted and 1 ActivityFailed.Worker.status()(#168)Check 3 details. Settings:
Client(timeout=10),poll_timeout=5,heartbeat_interval=5. The attempt deadline came out at 10.0 s and the stall bound at 80.0 s. Every read usedstatus()from a non-main thread.run():starting, ready=False (not_started), live=False.registeredat t=0.21 s with ready=True and live=True. Per-role statuses were present forworkflow_poll,activity_poll,query_pollandheartbeat, and every role had a successful exchange by t=10.2 s. A real workflow round trip then returnedok.docker pauseof the server, held 20 s (paused at t≈11.1 s):server_unavailable:query_poll, thenworkflow_pollandheartbeat). That is about 9 s after the pause, once the in-flight poll passed the 10 s deadline (overdue=true).stalled=false, maximum in-flight ≈21 s against the 80 s bound).docker unpauseat t=31.25 s: ready=True again at t=36.06 s, 4.8 s later, with no restart. Samerun()task (not done) and sameworker_id. A second round trip returnedok.stop(): immediatelydraining(ready=Falsedraining, live=True), thenstopped(ready=False, live=False, all rolesrunning=false).Already closed by the merge. This comment is the installed-acceptance receipt.