Feat: WT-1718 improve webtrit components monitoring

[Avareco]

WT-1718: expose Prometheus metrics from the adapter

Why

In WT-1717 the instance stopped serving requests while staying healthy as far as the cluster was concerned: the connection pool to PortaBilling ran dry, requests piled up — "902 requests in / 0 out" — with zero restarts, /api/health-check still returning its constant, and not one alert firing. We heard about it from the customer.

The numbers that would have shown this are unreachable from outside the process by construction: the httpx pool is a structure in the application's own memory, and Kubernetes can only see that the container is alive.

This PR closes point 2 of the ticket.

What it adds

/metrics on a dedicated port (METRICS_PORT, 9568 by default) rather than the application's, so that it is never reachable through the public ingress.

Our own metrics (app/metrics.py):

metric	what it answers
httpx_pool_connections{verify,host,state}	how many connections are held, by destination and state
httpx_pool_max_connections{verify}	the ceiling (PORTASWITCH_MAX_CONNECTIONS, 100 by default)
httpx_pool_queued_requests{verify}	how many requests are waiting for a free connection
httpx_pool_timeouts_total{host}	the ones that gave up waiting
request_deadline_expired_total{method}	calls killed outright by REQUEST_DEADLINE

The third row is the important one. A busy pool is not a problem — that is what a pool is for. The problem starts when the queue for it stops draining.

Saturation reads as connections == max_connections and queued_requests > 0 at the same time.

Both counters are deliberately threshold-free: any increment already means a live user got nothing back.

From prometheus-fastapi-instrumentator

metric	what it answers
http_requests_inprogress{handler,method}	how many requests are being served right now — the direct reading of WT-1717
http_requests_total{handler,method,status}	the 5xx share per endpoint
http_request_duration_seconds{handler}	latency per endpoint
http_request_size_bytes, http_response_size_bytes	payload sizes

The library leaves requests_inprogress off by default; it is enabled explicitly here, because it is precisely the metric that was missing.

How it is enabled

METRICS_PORT in the environment; left unset, the 9568 default applies. A failure to start the metrics server is logged and swallowed — a port already in use must not turn into an outage.

Worth a reviewer's attention

Middleware rather than RouteWithLogging. The existing route class only wraps routes registered on the routers, and /api/health-check is declared straight on app, so it would have been missing from the metrics.

The pool is read at scrape time, not on a timer, so the values are never stale. connections hands out a copy of the list and len() on a list cannot tear, so reading from the scrape thread is safe. The numbers may be a moment behind the event loop, which does not matter for a gauge.

Every private httpcore attribute is read defensively. _max_connections, _requests, _origin, is_idle() — all through getattr with a fallback. A future version that renames them costs the metric, not the scrape. There is a test for exactly that.

Single worker. The metrics server runs in a daemon thread of this process, which is correct for the one uvicorn worker the image starts. --workers 4 would give each worker its own registry and a scrape would report just one of the four; that would need the multiprocess mode of prometheus_client. The limitation is written into the module docstring.

record_pool_timeout is called by hand from except PoolTimeout. In the Elixir components Finch emits this as a telemetry event on its own; in httpx a pool timeout is an exception, so it can only be counted where it is caught. The import is lazy, inside the function, so the connector stays importable and working without the metrics stack.

Label cardinality is bounded. handler is the route template (/user/voicemails/{message_id}), never the resolved path, and statuses are grouped into 2xx/3xx/4xx/5xx. Neither a user id nor a resource id can reach a label.

prometheus-fastapi-instrumentator is capped at 7.1.* — 8.x requires a starlette newer than the one FastAPI is pinned to here.

The branch carries a merge commit — bringing main in to pick up WT-1774. It contributes no changes of its own.

Verified

• the 10 tests in the new tests/test_35_metrics.py pass inside the built image
• the image builds
• black is clean on app/metrics.py
• main merged in, no conflicts

The tests are unit tests against a fake pool — no server, no network. What they exercise is the reading of the pool, because that is where all the fragility is: in another library's private attributes.

Not in this PR

• the chart wiring — that lives in webtrit_deploy_internal as its own PR;
• alert rules — thresholds cannot be invented before the metrics have run on dev against real traffic; separate ticket;
• point 1 of the ticket (exit on error / readiness) — moved to WT-1867.