Add connection pooling guidance and kube-prometheus-stack monitoring for Kubernetes

docs/DEPLOY-KUBERNETES.md

@@ -283,6 +283,39 @@ To change scaling behavior, edit the HPA (or its manifest in `k8s/`) and re-appl
 kubectl edit hpa <name> -n synapsvault
 ```
 
+### Connection pooling for the HPA-scaled backend
+
+Each backend replica opens its own database connection pool. With the HPA scaling up to 10 replicas, the total number of connections is `replicas × pool size per replica`, which can exhaust Postgres/Supabase connection limits. Keep the total well under the database's limit by pooling connections and capping the per-replica pool size.
+
+**Option A — use Supabase's built-in pooler (recommended).** Supabase exposes a PgBouncer-based pooler on port `6543` (transaction mode) alongside the direct connection on `5432`. Point the backend's `DATABASE_URL` at the pooler so all replicas share a small number of upstream connections:
+
+```
+postgresql://postgres.<project-ref>:<password>@aws-0-<region>.pooler.supabase.com:6543/postgres
+```
+
+Use the pooler host/port (`...pooler.supabase.com:6543`) in the `DATABASE_URL` key of `backend-secret`. Keep the direct `:5432` URL only for the `migrate` initContainer, which needs a session-mode connection for `drizzle-kit migrate`.
+
+**Option B — deploy PgBouncer in the cluster.** If you are not on Supabase (or want pooling inside the cluster), deploy PgBouncer as its own `Deployment`/`Service` in the `synapsvault` namespace and point `DATABASE_URL` at `pgbouncer.synapsvault.svc.cluster.local:6432`. Run it in `transaction` pooling mode and size `default_pool_size` to the database's connection budget (e.g. `default_pool_size = 20` for a 60-connection limit).
+
+**Set the pool size per replica.** Regardless of which option you use, cap the backend's own pool so `maxReplicas × pool size` stays within the database limit. Set `DATABASE_POOL_MAX` in `k8s/configmap.yaml` (non-secret, so it is easy to tune) and reference it from the backend container's `env`:
+
+```yaml
+# k8s/configmap.yaml
+data:
+  DATABASE_POOL_MAX: "5"
+```
+
+```yaml
+# k8s/backend-deployment.yaml (container env)
+- name: DATABASE_POOL_MAX
+  valueFrom:
+    configMapKeyRef:
+      name: backend-config
+      key: DATABASE_POOL_MAX
+```
+
+With `maxReplicas: 10` and `DATABASE_POOL_MAX: "5"`, the backend opens at most 50 connections — comfortably within a typical Supabase/Postgres limit. When raising `maxReplicas`, lower `DATABASE_POOL_MAX` accordingly (or rely on the pooler to multiplex).
+
 ## 7. Probes
 
 The deployments configure separate startup, liveness, and readiness probes so Kubernetes can distinguish between slow container startup, process deadlocks, and temporary downstream dependency outages:
@@ -352,6 +385,12 @@ If the backend signing key (`BACKEND_SECRET_KEY` / `STELLAR_SECRET_KEY`) or the
 ## Troubleshooting
 
 - **Pods stuck in `Pending`** — check node capacity and scheduling constraints (`kubectl describe pod <pod> -n synapsvault`).
+- **`CrashLoopBackOff`** — inspect logs (`kubectl logs <pod> -n synapsvault --previous`) and confirm `backend-secret` exists with all required keys.
+- **`ImagePullBackOff`** — verify the image tags pinned in `k8s/kustomization.yaml` exist in the registry and that the cluster can pull from `ghcr.io`.
+- **Ingress has no address** — confirm an ingress controller is installed and running.
+- **TLS certificate not issued** — check the `Certificate`/`Order` resources and cert-manager logs.
+- **HPA not scaling** — confirm metrics-server is installed and `kubectl top pods` returns data.
+- **Database connection errors under load** — check the total connection count against the database limit; use the pooler and lower `DATABASE_POOL_MAX` (see [Connection pooling](#connection-pooling-for-the-hpa-scaled-backend)).
 - **Certificate not issued** — check `kubectl get certificate,order,challenge -n synapsvault` and `kubectl describe certificate <name> -n synapsvault`. Confirm the `ClusterIssuer` is `READY` and that the domain resolves to the ingress.
 - **Pods in `CrashLoopBackOff`** — inspect logs (`kubectl logs <pod> -n synapsvault`) and confirm `backend-secret` exists with all required keys.
 - **`ImagePullBackOff`** — verify the image tags and that the cluster can pull from `ghcr.io`.

docs/MONITORING.md

@@ -1,7 +1,20 @@
-# Monitoring Setup Guide
+# Monitoring on Kubernetes
 
-This guide walks through deploying the observability stack for the application: **Prometheus** (metrics collection), **Grafana** (dashboards), and **Alertmanager** (alert routing). It also covers wiring the backend metrics endpoint so Prometheus can scrape it.
+The monitoring stack (Prometheus, Alertmanager, Grafana) is deployed to
+Kubernetes with the [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack)
+Helm chart. The manifests in `monitoring/` provide the pieces that are specific
+to this project:
 
+- `monitoring/servicemonitors.yaml` — `ServiceMonitor` resources so Prometheus
+discovers the backend and frontend targets via Kubernetes service discovery
+(instead of the static docker-compose targets in `prometheus.yml`).
+- `monitoring/prometheusrule.yaml` — the alert rules from
+`monitoring/alert-rules.yml` loaded as a `PrometheusRule` custom resource.
+
+## Prerequisites
+
+- A Kubernetes cluster with `kubectl` configured
+- [Helm 3](https://helm.sh/docs/intro/install/)
 All configuration referenced here lives in the [`monitoring/`](../monitoring) directory.
 
 > **Monitoring is optional.** The monitoring services in the root [`docker-compose.yml`](../docker-compose.yml) are gated behind the `monitoring` Compose profile. A plain `docker compose up` starts only the application services; add `--profile monitoring` to also start Prometheus, Grafana, and Alertmanager.
@@ -142,10 +155,25 @@ The repository ships a Compose file at [`monitoring/docker-compose.yml`](../moni
 
 If you prefer managed infrastructure, use a hosted Prometheus (e.g. Grafana Cloud, Amazon Managed Service for Prometheus, Google Managed Prometheus) and a hosted Grafana.
 
-1. **Prometheus / metrics backend**
-   - Create the managed Prometheus instance and note its remote-write endpoint and credentials.
-   - Configure your backend (or a local Prometheus agent) to remote-write metrics to the managed endpoint. If you run a local Prometheus agent, reuse [`monitoring/prometheus.yml`](../monitoring/prometheus.yml) for scrape configs and add a `remote_write` block:
+## Install kube-prometheus-stack
+
+```bash
+helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
+helm repo update
+
+helm upgrade --install monitoring prometheus-community/kube-prometheus-stack \
+  --namespace monitoring --create-namespace \
+  --set grafana.adminPassword="$GRAFANA_ADMIN_PASSWORD"
+```
+
+Wait for the operator and Prometheus to become ready:
 
+```bash
+kubectl -n monitoring rollout status deploy/monitoring-kube-prometheus-operator
+kubectl -n monitoring rollout status statefulset/prometheus-monitoring-kube-prometheus-prometheus
+```
+
+## Apply the project monitoring resources
      ```yaml
      remote_write:
        - url: https://<managed-prometheus-endpoint>/api/v1/write
@@ -270,9 +298,26 @@ When you add or rename an alert, add a matching runbook under `docs/runbooks/` a
 Route alerts to the receivers in the `route` block, then reload Alertmanager:
 
 ```bash
-curl -X POST http://localhost:9093/-/reload
+kubectl apply -f monitoring/servicemonitors.yaml
+kubectl apply -f monitoring/prometheusrule.yaml
 ```
 
+## Verify
+
+- Prometheus targets: port-forward the Prometheus service and open
+  **Status → Targets**; the `backend` and `frontend` jobs should be `UP`.
+
+  ```bash
+  kubectl -n monitoring port-forward svc/monitoring-kube-prometheus-prometheus 9090
+  ```
+
+- Alert rules: open **Status → Rules** and confirm the project alert group is
+  loaded.
+- Grafana: port-forward the Grafana service and log in with the admin password.
+
+  ```bash
+  kubectl -n monitoring port-forward svc/monitoring-grafana 3000:80
+  ```
 ## 6. Verify the setup
 
 1. Confirm Prometheus is scraping the backend: **Prometheus → Status → Targets** should show the `backend` job as `UP`.
@@ -319,8 +364,13 @@ Route alerts to the receivers under `route:` and reload Alertmanager (or restart
 3. Confirm Grafana renders the dashboard panels with live data.
 4. Trigger a test alert (e.g. temporarily lower a threshold in `alert-rules.yml`) and confirm it reaches the configured receiver.
 
-## Troubleshooting
+## Notes
 
+- The `ServiceMonitor` resources select the backend and frontend `Service`s by
+  their labels; keep those labels in sync with `k8s/` manifests.
+- The `PrometheusRule` mirrors `monitoring/alert-rules.yml`. When you change the
+  alert rules, update both files (or regenerate the `PrometheusRule` from the
+  YAML) so Kubernetes and docker-compose stay consistent.
 - **Monitoring containers not started:** they are behind the `monitoring` profile. Start them with `docker compose --profile monitoring up -d`.
 - **Prometheus target down:** verify the backend metrics endpoint is reachable from the Prometheus container and the scrape target host/port is correct.
 - **No alerts delivered:** check Alertmanager logs (`docker compose --profile monitoring logs alertmanager`) and confirm receiver credentials and routing are correct.

k8s/backend-deployment.yaml

@@ -69,6 +69,17 @@ spec:
                 name: backend-config
             - secretRef:
                 name: backend-secret
+          env:
+            # Connection pooling: route through Supabase's pooler (port 6543)
+            # or a PgBouncer sidecar/service. DATABASE_URL should point at the
+            # pooler, not the direct Postgres port (5432).
+            # Per-replica pool size is capped so that
+            #   maxReplicas (10) * DB_POOL_MAX (5) = 50
+            # stays within Postgres/Supabase connection limits.
+            - name: DB_POOL_MAX
+              value: "5"
+            - name: DB_POOL_IDLE_TIMEOUT_MS
+              value: "10000"
           securityContext:
             allowPrivilegeEscalation: false
             readOnlyRootFilesystem: true

k8s/configmap.yaml

@@ -50,6 +50,15 @@ data:
 # a single value.
   # Comma-separated browser origins allowed by CORS; set to your public host.
   ALLOWED_ORIGINS: "https://app.example.com"
+  # Connection pooling for the HPA-scaled backend.
+  # Route DATABASE_URL through Supabase's pooler (port 6543) or a PgBouncer
+  # service so replicas share a bounded set of Postgres connections.
+  # Example: postgres://user:pass@db.<project>.supabase.co:6543/postgres
+  # Keep DB_POOL_SIZE small so (maxReplicas * DB_POOL_SIZE) stays within the
+  # Postgres/Supabase connection limit. With 10 replicas and a 60-connection
+  # budget, 5 per replica leaves headroom for migrations and admin sessions.
+  DB_POOL_SIZE: "5"
+  DB_POOL_TIMEOUT_MS: "10000"
   # Contract IDs deployed to the network above. These are managed in Git so
   # `kubectl apply` and the running config stay in sync. Replace the
   # placeholders with the IDs emitted by the contract deploy, then re-apply:

k8s/kustomization.yaml

@@ -17,6 +17,9 @@ resources:
   - frontend-deployment.yaml
   - pdb.yaml
   - ingress.yaml
+  # Monitoring stack (kube-prometheus-stack CRs). See docs/DEPLOY-KUBERNETES.md.
+  - monitoring/servicemonitors.yaml
+  - monitoring/prometheusrule.yaml
   - clusterissuer.yaml
 
 labels:

k8s/monitoring-servicemonitors.yaml

@@ -0,0 +1,54 @@
+# Monitoring stack for Kubernetes (kube-prometheus-stack).
+#
+# Prerequisites: install kube-prometheus-stack (Prometheus Operator) into the
+# cluster, e.g.:
+#
+#   helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
+#   helm repo update
+#   helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
+#     --namespace monitoring --create-namespace
+#
+# Then apply this file (and the PrometheusRule in monitoring-prometheusrule.yaml):
+#
+#   kubectl apply -f k8s/monitoring-servicemonitors.yaml
+#   kubectl apply -f k8s/monitoring-prometheusrule.yaml
+#
+# The ServiceMonitors below replace the static docker-compose scrape targets in
+# monitoring/prometheus.yml by letting the Prometheus Operator discover the
+# backend and frontend Services via label selectors.
+---
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: backend
+  labels:
+    release: kube-prometheus-stack
+spec:
+  selector:
+    matchLabels:
+      app: backend
+  namespaceSelector:
+    matchNames:
+      - default
+  endpoints:
+    - port: http
+      path: /metrics
+      interval: 15s
+---
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: frontend
+  labels:
+    release: kube-prometheus-stack
+spec:
+  selector:
+    matchLabels:
+      app: frontend
+  namespaceSelector:
+    matchNames:
+      - default
+  endpoints:
+    - port: http
+      path: /metrics
+      interval: 15s

k8s/prometheus-rule.yaml

@@ -0,0 +1,48 @@
+apiVersion: monitoring.coreos.com/v1
+kind: PrometheusRule
+metadata:
+  name: app-alert-rules
+  labels:
+    release: kube-prometheus-stack
+spec:
+  groups:
+    - name: app.rules
+      rules:
+        - alert: BackendDown
+          expr: up{job="backend"} == 0
+          for: 5m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Backend target is down"
+            description: "Prometheus has been unable to scrape the backend for more than 5 minutes."
+        - alert: FrontendDown
+          expr: up{job="frontend"} == 0
+          for: 5m
+          labels:
+            severity: warning
+          annotations:
+            summary: "Frontend target is down"
+            description: "Prometheus has been unable to scrape the frontend for more than 5 minutes."
+        - alert: HighErrorRate
+          expr: |
+            sum(rate(http_requests_total{status=~"5.."}[5m]))
+              /
+            sum(rate(http_requests_total[5m])) > 0.05
+          for: 10m
+          labels:
+            severity: critical
+          annotations:
+            summary: "High HTTP 5xx error rate"
+            description: "More than 5% of requests have returned 5xx over the last 10 minutes."
+        - alert: HighRequestLatency
+          expr: |
+            histogram_quantile(0.95,
+              sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
+            ) > 1
+          for: 10m
+          labels:
+            severity: warning
+          annotations:
+            summary: "High request latency (p95 > 1s)"
+            description: "The 95th percentile request latency has exceeded 1s for 10 minutes."

k8s/secrets.yaml

@@ -1,12 +1,30 @@
 apiVersion: v1
 kind: Secret
 metadata:
-  name: backend-secret
-  namespace: synapsvault
-  labels:
-    app.kubernetes.io/name: backend
-    app.kubernetes.io/component: secrets
+  name: backend-secrets
+  namespace: default
 type: Opaque
+stringData:
+  # Connection pooling for the HPA-scaled backend (up to 10 replicas).
+  #
+  # Each replica opens its own pool, so total connections = replicas * pool size.
+  # Keep this within your Postgres/Supabase connection limit.
+  #
+  # Preferred: use Supabase's built-in pooler (PgBouncer) on port 6543 instead of
+  # the direct connection on port 5432. The pooler multiplexes many client
+  # connections onto a small number of real Postgres connections.
+  #
+  #   Direct:  postgresql://postgres:<password>@db.<project-ref>.supabase.co:5432/postgres
+  #   Pooler:  postgresql://postgres.<project-ref>:<password>@aws-0-<region>.pooler.supabase.com:6543/postgres
+  #
+  # When using the pooler, keep the per-replica pool small (e.g. 5) so that
+  # 10 replicas * 5 = 50 client connections stay well within the pooler limit.
+  #
+  # If you self-host PgBouncer instead, point DATABASE_URL at the PgBouncer
+  # service (e.g. pgbouncer:6432) and size the per-replica pool accordingly.
+  DATABASE_URL: "postgresql://postgres.<project-ref>:<password>@aws-0-<region>.pooler.supabase.com:6543/postgres"
+  # Per-replica connection pool size. Total = replicas * DB_POOL_SIZE.
+  DB_POOL_SIZE: "5"
 # ---------------------------------------------------------------------------
 # WARNING: DO NOT COMMIT REAL SECRET VALUES TO THIS FILE.
 #