Expose worker readiness and liveness status

CHANGELOG.md

@@ -6,6 +6,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- `Worker.status()` returns a read-only `WorkerStatus` snapshot for health
+  endpoints and orchestrator probes. Readiness requires this worker's own
+  acknowledged registration, running pollers and a successful latest Server
+  exchange per role; liveness reports event-loop progress, exited pollers and
+  stalled requests. Snapshots perform no network I/O and can be read from
+  another thread. Adds `WorkerState` and `WorkerRoleStatus`, a runnable probe
+  example and a Kubernetes guide. Existing worker behavior is unchanged.
+
 ## [2.12.4] - 2026-11-02
 
 ### Added

docs/guides/worker-health.md

@@ -0,0 +1,128 @@
+# Worker readiness and liveness
+
+`Worker.status()` returns a read-only `WorkerStatus` snapshot for an
+application health endpoint or an orchestrator probe. It reports this worker's
+own registration and local progress. A healthy Server, or another worker on the
+same queue, does not make this worker ready.
+
+This API is in unreleased SDK source. Published packages need the release
+that includes it.
+
+## Ready versus live
+
+| | Ready | Live |
+|---|---|---|
+| Question | Should this worker count as serving its queue? | Is this worker's local machinery progressing? |
+| True when | Registration is acknowledged for its namespace and queue, every configured poller is running, the latest Server exchange of every role succeeded, and no request is overdue | The event loop advanced recently, no configured poller has exited, and no Server request has outlived its stall bound |
+| Transient Server or network outage | `False` (`server_unavailable:<role>`) | `True` |
+| Use it for | Readiness probes, load-balancer membership, rollout gates | Liveness probes that restart a stuck process |
+
+The roles are `heartbeat`, plus `workflow_poll`, `activity_poll` and
+`query_poll` when the worker registers workflows, activities or Server query
+tasks. Readiness returns without a restart on the next successful exchange of
+each failed role. A failed heartbeat stays unready until a later heartbeat
+succeeds, so recovery can take up to one heartbeat interval.
+
+An idle long poll is healthy. Two bounds apply to an outstanding poll or
+heartbeat:
+
+- `request_overdue_seconds` (readiness) defaults to one HTTP attempt deadline,
+  the larger of the client request timeout and the poll's absolute HTTP
+  deadline. A request unanswered for longer reports `server_unavailable` even
+  while the client is still retrying it, because the Server did not answer in
+  time.
+- `request_stall_seconds` (liveness) defaults to twice the client's complete
+  transport retry budget for one request. Only a request stuck beyond any
+  legitimate retry window marks the worker not live, so a Server outage alone
+  does not trigger a liveness restart.
+
+Long-running activity handlers never affect either signal.
+Poll-capacity backpressure is a Server answer for an admitted worker and keeps
+the worker ready.
+
+## States
+
+| `state` | `ready` | `live` | Meaning |
+|---|---|---|---|
+| `starting` | `False` (`not_started` or `registering`) | `False` before `run()`, then `True` | Constructed, or waiting for the registration acknowledgement |
+| `registered` | per the rules above | per the rules above | Polling and heartbeating |
+| `draining` | `False` (`draining`) | `True` while the loop progresses | `stop()` was requested; in-flight work is draining |
+| `stopped` | `False` | `False` | Shutdown finished and the registration was removed |
+| `failed` | `False` | `False` (`failed:registration`, `failed:runtime` or `failed:shutdown`) | Registration was refused, a poller failed, or shutdown could not confirm cleanup |
+
+A dead query-poller thread, a stalled event loop, a registration refusal and a
+completed shutdown never report ready.
+
+## Checking a stalled event loop
+
+`status()` performs no network I/O, claims no tasks, renews no leases and does
+not wait on the event loop. While the worker runs, a small local task advances a
+progress tick once per second. Serve probes from a separate thread, as in the
+example below, and the probe can answer even when the event loop is blocked:
+`live` turns `False` with `event_loop_stalled` once the tick is older than
+`event_loop_stall_seconds` (default 30). A probe handler that runs on the
+worker's own event loop cannot report its own stall.
+
+```python
+status = worker.status()
+status.ready, status.ready_reason   # (True, "ready")
+status.live, status.live_reason     # (True, "live")
+status.as_dict()                    # JSON-serializable; identity and progress only
+```
+
+The snapshot contains the worker ID, namespace, task queue, state, reasons,
+ages in seconds, and per-role progress. It never contains payloads, task
+arguments, exception messages (only exception class names) or credentials.
+
+## Kubernetes example
+
+`examples/worker_health_probe.py` runs a worker with a stdlib HTTP server on its
+own thread, serving `/readyz`, `/livez` and `/status`. Each answers 200 or 503
+with the snapshot as JSON. Bind it to the pod network only.
+
+```yaml
+containers:
+  - name: worker
+    image: example/python-worker:1.0.0
+    env:
+      - name: DW_HEALTH_HOST
+        value: "0.0.0.0"
+    ports:
+      - name: health
+        containerPort: 8081
+    startupProbe:
+      httpGet: {path: /readyz, port: health}
+      periodSeconds: 2
+      failureThreshold: 60
+    readinessProbe:
+      httpGet: {path: /readyz, port: health}
+      periodSeconds: 5
+      failureThreshold: 2
+    livenessProbe:
+      httpGet: {path: /livez, port: health}
+      periodSeconds: 10
+      failureThreshold: 6
+terminationGracePeriodSeconds: 45
+```
+
+Give the liveness probe a failure window longer than ordinary event-loop pauses,
+such as a large replay. Restarting a worker that is only unready because the
+Server is unreachable does not help, so don't probe liveness with `/readyz`.
+
+## Shutdown
+
+Three different timers apply. Keep them distinct:
+
+- `Worker(shutdown_timeout=...)` is how long `stop()` drains in-flight work
+  before cancelling it and removing the registration.
+- The orchestrator's stop grace (`terminationGracePeriodSeconds`) is how long
+  the process may run after SIGTERM before it is killed. Make it longer than
+  `shutdown_timeout`, plus time for deregistration.
+- Handler termination is separate again. Cancelling an await cannot stop a
+  running synchronous activity thread; if one is still running at the
+  deadline, `stop()` raises and the status becomes `failed:shutdown`.
+
+On SIGTERM, call `worker.stop()` and await the run task. While draining, the
+worker reports `draining` (not ready, still live), so traffic moves away
+without a liveness restart interrupting the drain. See
+[Lifecycle and shutdown](../reference/worker.md#lifecycle-and-shutdown).

docs/reference/worker.md

@@ -197,4 +197,15 @@ before its reply reaches the SDK. A worker that never receives that reply has
 not begun polling and becomes stale under the Server's heartbeat policy.
 Use the existing workflow handle and queue when bringing up a replacement.
 
+## Readiness and liveness
+
+`Worker.status()` returns a read-only snapshot that separates readiness
+(registered, every configured poller running, latest Server exchange of each
+role succeeded) from liveness (event loop progressing, no poller exited, no
+request outliving its bound). It performs no network I/O and can be called from
+another thread. See [Worker readiness and liveness](../guides/worker-health.md)
+for states, probe configuration and shutdown guidance.
+
 ::: durable_workflow.worker
+
+::: durable_workflow.worker_status

examples/worker_health_probe.py

@@ -0,0 +1,114 @@
+"""Serve worker readiness and liveness probes next to a long-running worker.
+
+Run against a Server:
+
+    DURABLE_WORKFLOW_SERVER_URL=http://localhost:8080 \\
+    DURABLE_WORKFLOW_AUTH_TOKEN=dev-token-123 \\
+    python examples/worker_health_probe.py
+
+Then probe it:
+
+    curl -i http://127.0.0.1:8081/readyz
+    curl -i http://127.0.0.1:8081/livez
+
+The probe server runs on its own thread. ``Worker.status()`` performs no
+network I/O and does not need the worker's event loop, so ``/livez`` still
+answers, with 503, when that loop is stalled. Bind the probe port only to the
+pod or host network your orchestrator uses; the response carries worker
+identity and progress, not payloads or credentials.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import os
+import signal
+import threading
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+
+from durable_workflow import Client, Worker, activity, workflow
+
+
+@activity.defn(name="greet")
+def greet(name: str) -> str:
+    return f"hello, {name}"
+
+
+@workflow.defn(name="greeter")
+class GreeterWorkflow:
+    def run(self, ctx, name):  # type: ignore[no-untyped-def]
+        result = yield ctx.schedule_activity("greet", [name])
+        return result
+
+
+def start_probe_server(worker: Worker, host: str, port: int) -> ThreadingHTTPServer:
+    class ProbeHandler(BaseHTTPRequestHandler):
+        def do_GET(self) -> None:  # noqa: N802 - http.server API
+            status = worker.status()
+            if self.path == "/readyz":
+                healthy = status.ready
+            elif self.path == "/livez":
+                healthy = status.live
+            elif self.path == "/status":
+                healthy = True
+            else:
+                self.send_error(404)
+                return
+            body = json.dumps(status.as_dict()).encode()
+            self.send_response(200 if healthy else 503)
+            self.send_header("Content-Type", "application/json")
+            self.send_header("Content-Length", str(len(body)))
+            self.end_headers()
+            self.wfile.write(body)
+
+        def log_message(self, format: str, *args: object) -> None:  # noqa: A002 - http.server API
+            return
+
+    server = ThreadingHTTPServer((host, port), ProbeHandler)
+    threading.Thread(target=server.serve_forever, name="worker-health-probe", daemon=True).start()
+    return server
+
+
+async def main() -> None:
+    async with Client(
+        os.environ.get("DURABLE_WORKFLOW_SERVER_URL", "http://localhost:8080"),
+        token=os.environ.get("DURABLE_WORKFLOW_AUTH_TOKEN"),
+        namespace=os.environ.get("DURABLE_WORKFLOW_NAMESPACE", "default"),
+    ) as client:
+        worker = Worker(
+            client,
+            task_queue=os.environ.get("DURABLE_WORKFLOW_TASK_QUEUE", "python-workers"),
+            workflows=[GreeterWorkflow],
+            activities=[greet],
+            # Keep this below the orchestrator's termination grace period.
+            shutdown_timeout=25.0,
+        )
+        probe = start_probe_server(
+            worker,
+            os.environ.get("DW_HEALTH_HOST", "127.0.0.1"),
+            int(os.environ.get("DW_HEALTH_PORT", "8081")),
+        )
+        shutdown_requested = asyncio.Event()
+        loop = asyncio.get_running_loop()
+        for signum in (signal.SIGTERM, signal.SIGINT):
+            loop.add_signal_handler(signum, shutdown_requested.set)
+
+        run_task = asyncio.create_task(worker.run())
+        try:
+            await asyncio.wait(
+                {run_task, asyncio.create_task(shutdown_requested.wait())},
+                return_when=asyncio.FIRST_COMPLETED,
+            )
+        finally:
+            try:
+                await worker.stop()
+            finally:
+                try:
+                    await run_task
+                finally:
+                    probe.shutdown()
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

mkdocs.yml

@@ -79,6 +79,7 @@ nav:
   - Execution telemetry: telemetry.md
   - Sticky execution: guides/sticky-execution.md
   - Separate worker processes: guides/separate-workers.md
+  - Worker readiness and liveness: guides/worker-health.md
   - Query deadlines: query-task-authority.md
   - SDK guide: https://durable-workflow.com/docs/2.0/polyglot/python/
   - PyPI: https://pypi.org/project/durable-workflow/

scripts/smoke-built-package.py

@@ -112,6 +112,7 @@ def completed(result):
     "durable_workflow.sync",
     "durable_workflow.testing",
     "durable_workflow.worker",
+    "durable_workflow.worker_status",
     "durable_workflow.workflow",
     "durable_workflow.workflow_updates_conformance",
 ]

src/durable_workflow/__init__.py

@@ -225,6 +225,7 @@
 )
 from .worker import Worker
 from .worker_session import WorkerSession, WorkerSessionOptions
+from .worker_status import WorkerRoleStatus, WorkerState, WorkerStatus
 from .workflow import (
     ActivityRetryPolicy,
     ChildWorkflowRetryPolicy,
@@ -314,6 +315,9 @@
     "Worker",
     "WorkerSession",
     "WorkerSessionOptions",
+    "WorkerRoleStatus",
+    "WorkerState",
+    "WorkerStatus",
     "WorkerInterceptor",
     "WorkflowCommandResult",
     "WorkflowExecution",