diff --git a/docs/DEPLOY-KUBERNETES.md b/docs/DEPLOY-KUBERNETES.md index a36cc4a..dc76599 100644 --- a/docs/DEPLOY-KUBERNETES.md +++ b/docs/DEPLOY-KUBERNETES.md @@ -283,6 +283,39 @@ To change scaling behavior, edit the HPA (or its manifest in `k8s/`) and re-appl kubectl edit hpa -n synapsvault ``` +### Connection pooling for the HPA-scaled backend + +Each backend replica opens its own database connection pool. With the HPA scaling up to 10 replicas, the total number of connections is `replicas × pool size per replica`, which can exhaust Postgres/Supabase connection limits. Keep the total well under the database's limit by pooling connections and capping the per-replica pool size. + +**Option A — use Supabase's built-in pooler (recommended).** Supabase exposes a PgBouncer-based pooler on port `6543` (transaction mode) alongside the direct connection on `5432`. Point the backend's `DATABASE_URL` at the pooler so all replicas share a small number of upstream connections: + +``` +postgresql://postgres.:@aws-0-.pooler.supabase.com:6543/postgres +``` + +Use the pooler host/port (`...pooler.supabase.com:6543`) in the `DATABASE_URL` key of `backend-secret`. Keep the direct `:5432` URL only for the `migrate` initContainer, which needs a session-mode connection for `drizzle-kit migrate`. + +**Option B — deploy PgBouncer in the cluster.** If you are not on Supabase (or want pooling inside the cluster), deploy PgBouncer as its own `Deployment`/`Service` in the `synapsvault` namespace and point `DATABASE_URL` at `pgbouncer.synapsvault.svc.cluster.local:6432`. Run it in `transaction` pooling mode and size `default_pool_size` to the database's connection budget (e.g. `default_pool_size = 20` for a 60-connection limit). + +**Set the pool size per replica.** Regardless of which option you use, cap the backend's own pool so `maxReplicas × pool size` stays within the database limit. Set `DATABASE_POOL_MAX` in `k8s/configmap.yaml` (non-secret, so it is easy to tune) and reference it from the backend container's `env`: + +```yaml +# k8s/configmap.yaml +data: + DATABASE_POOL_MAX: "5" +``` + +```yaml +# k8s/backend-deployment.yaml (container env) +- name: DATABASE_POOL_MAX + valueFrom: + configMapKeyRef: + name: backend-config + key: DATABASE_POOL_MAX +``` + +With `maxReplicas: 10` and `DATABASE_POOL_MAX: "5"`, the backend opens at most 50 connections — comfortably within a typical Supabase/Postgres limit. When raising `maxReplicas`, lower `DATABASE_POOL_MAX` accordingly (or rely on the pooler to multiplex). + ## 7. Probes The deployments configure separate startup, liveness, and readiness probes so Kubernetes can distinguish between slow container startup, process deadlocks, and temporary downstream dependency outages: @@ -352,6 +385,12 @@ If the backend signing key (`BACKEND_SECRET_KEY` / `STELLAR_SECRET_KEY`) or the ## Troubleshooting - **Pods stuck in `Pending`** — check node capacity and scheduling constraints (`kubectl describe pod -n synapsvault`). +- **`CrashLoopBackOff`** — inspect logs (`kubectl logs -n synapsvault --previous`) and confirm `backend-secret` exists with all required keys. +- **`ImagePullBackOff`** — verify the image tags pinned in `k8s/kustomization.yaml` exist in the registry and that the cluster can pull from `ghcr.io`. +- **Ingress has no address** — confirm an ingress controller is installed and running. +- **TLS certificate not issued** — check the `Certificate`/`Order` resources and cert-manager logs. +- **HPA not scaling** — confirm metrics-server is installed and `kubectl top pods` returns data. +- **Database connection errors under load** — check the total connection count against the database limit; use the pooler and lower `DATABASE_POOL_MAX` (see [Connection pooling](#connection-pooling-for-the-hpa-scaled-backend)). - **Certificate not issued** — check `kubectl get certificate,order,challenge -n synapsvault` and `kubectl describe certificate -n synapsvault`. Confirm the `ClusterIssuer` is `READY` and that the domain resolves to the ingress. - **Pods in `CrashLoopBackOff`** — inspect logs (`kubectl logs -n synapsvault`) and confirm `backend-secret` exists with all required keys. - **`ImagePullBackOff`** — verify the image tags and that the cluster can pull from `ghcr.io`. diff --git a/docs/MONITORING.md b/docs/MONITORING.md index 7178adb..64fa604 100644 --- a/docs/MONITORING.md +++ b/docs/MONITORING.md @@ -1,7 +1,20 @@ -# Monitoring Setup Guide +# Monitoring on Kubernetes -This guide walks through deploying the observability stack for the application: **Prometheus** (metrics collection), **Grafana** (dashboards), and **Alertmanager** (alert routing). It also covers wiring the backend metrics endpoint so Prometheus can scrape it. +The monitoring stack (Prometheus, Alertmanager, Grafana) is deployed to +Kubernetes with the [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) +Helm chart. The manifests in `monitoring/` provide the pieces that are specific +to this project: +- `monitoring/servicemonitors.yaml` — `ServiceMonitor` resources so Prometheus +discovers the backend and frontend targets via Kubernetes service discovery +(instead of the static docker-compose targets in `prometheus.yml`). +- `monitoring/prometheusrule.yaml` — the alert rules from +`monitoring/alert-rules.yml` loaded as a `PrometheusRule` custom resource. + +## Prerequisites + +- A Kubernetes cluster with `kubectl` configured +- [Helm 3](https://helm.sh/docs/intro/install/) All configuration referenced here lives in the [`monitoring/`](../monitoring) directory. > **Monitoring is optional.** The monitoring services in the root [`docker-compose.yml`](../docker-compose.yml) are gated behind the `monitoring` Compose profile. A plain `docker compose up` starts only the application services; add `--profile monitoring` to also start Prometheus, Grafana, and Alertmanager. @@ -142,10 +155,25 @@ The repository ships a Compose file at [`monitoring/docker-compose.yml`](../moni If you prefer managed infrastructure, use a hosted Prometheus (e.g. Grafana Cloud, Amazon Managed Service for Prometheus, Google Managed Prometheus) and a hosted Grafana. -1. **Prometheus / metrics backend** - - Create the managed Prometheus instance and note its remote-write endpoint and credentials. - - Configure your backend (or a local Prometheus agent) to remote-write metrics to the managed endpoint. If you run a local Prometheus agent, reuse [`monitoring/prometheus.yml`](../monitoring/prometheus.yml) for scrape configs and add a `remote_write` block: +## Install kube-prometheus-stack + +```bash +helm repo add prometheus-community https://prometheus-community.github.io/helm-charts +helm repo update + +helm upgrade --install monitoring prometheus-community/kube-prometheus-stack \ + --namespace monitoring --create-namespace \ + --set grafana.adminPassword="$GRAFANA_ADMIN_PASSWORD" +``` + +Wait for the operator and Prometheus to become ready: +```bash +kubectl -n monitoring rollout status deploy/monitoring-kube-prometheus-operator +kubectl -n monitoring rollout status statefulset/prometheus-monitoring-kube-prometheus-prometheus +``` + +## Apply the project monitoring resources ```yaml remote_write: - url: https:///api/v1/write @@ -270,9 +298,26 @@ When you add or rename an alert, add a matching runbook under `docs/runbooks/` a Route alerts to the receivers in the `route` block, then reload Alertmanager: ```bash -curl -X POST http://localhost:9093/-/reload +kubectl apply -f monitoring/servicemonitors.yaml +kubectl apply -f monitoring/prometheusrule.yaml ``` +## Verify + +- Prometheus targets: port-forward the Prometheus service and open + **Status → Targets**; the `backend` and `frontend` jobs should be `UP`. + + ```bash + kubectl -n monitoring port-forward svc/monitoring-kube-prometheus-prometheus 9090 + ``` + +- Alert rules: open **Status → Rules** and confirm the project alert group is + loaded. +- Grafana: port-forward the Grafana service and log in with the admin password. + + ```bash + kubectl -n monitoring port-forward svc/monitoring-grafana 3000:80 + ``` ## 6. Verify the setup 1. Confirm Prometheus is scraping the backend: **Prometheus → Status → Targets** should show the `backend` job as `UP`. @@ -319,8 +364,13 @@ Route alerts to the receivers under `route:` and reload Alertmanager (or restart 3. Confirm Grafana renders the dashboard panels with live data. 4. Trigger a test alert (e.g. temporarily lower a threshold in `alert-rules.yml`) and confirm it reaches the configured receiver. -## Troubleshooting +## Notes +- The `ServiceMonitor` resources select the backend and frontend `Service`s by + their labels; keep those labels in sync with `k8s/` manifests. +- The `PrometheusRule` mirrors `monitoring/alert-rules.yml`. When you change the + alert rules, update both files (or regenerate the `PrometheusRule` from the + YAML) so Kubernetes and docker-compose stay consistent. - **Monitoring containers not started:** they are behind the `monitoring` profile. Start them with `docker compose --profile monitoring up -d`. - **Prometheus target down:** verify the backend metrics endpoint is reachable from the Prometheus container and the scrape target host/port is correct. - **No alerts delivered:** check Alertmanager logs (`docker compose --profile monitoring logs alertmanager`) and confirm receiver credentials and routing are correct. diff --git a/k8s/backend-deployment.yaml b/k8s/backend-deployment.yaml index 680ec66..da8edb1 100644 --- a/k8s/backend-deployment.yaml +++ b/k8s/backend-deployment.yaml @@ -69,6 +69,17 @@ spec: name: backend-config - secretRef: name: backend-secret + env: + # Connection pooling: route through Supabase's pooler (port 6543) + # or a PgBouncer sidecar/service. DATABASE_URL should point at the + # pooler, not the direct Postgres port (5432). + # Per-replica pool size is capped so that + # maxReplicas (10) * DB_POOL_MAX (5) = 50 + # stays within Postgres/Supabase connection limits. + - name: DB_POOL_MAX + value: "5" + - name: DB_POOL_IDLE_TIMEOUT_MS + value: "10000" securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true diff --git a/k8s/configmap.yaml b/k8s/configmap.yaml index 05446e2..997febd 100644 --- a/k8s/configmap.yaml +++ b/k8s/configmap.yaml @@ -50,6 +50,15 @@ data: # a single value. # Comma-separated browser origins allowed by CORS; set to your public host. ALLOWED_ORIGINS: "https://app.example.com" + # Connection pooling for the HPA-scaled backend. + # Route DATABASE_URL through Supabase's pooler (port 6543) or a PgBouncer + # service so replicas share a bounded set of Postgres connections. + # Example: postgres://user:pass@db..supabase.co:6543/postgres + # Keep DB_POOL_SIZE small so (maxReplicas * DB_POOL_SIZE) stays within the + # Postgres/Supabase connection limit. With 10 replicas and a 60-connection + # budget, 5 per replica leaves headroom for migrations and admin sessions. + DB_POOL_SIZE: "5" + DB_POOL_TIMEOUT_MS: "10000" # Contract IDs deployed to the network above. These are managed in Git so # `kubectl apply` and the running config stay in sync. Replace the # placeholders with the IDs emitted by the contract deploy, then re-apply: diff --git a/k8s/kustomization.yaml b/k8s/kustomization.yaml index d184adc..f6d0126 100644 --- a/k8s/kustomization.yaml +++ b/k8s/kustomization.yaml @@ -17,6 +17,9 @@ resources: - frontend-deployment.yaml - pdb.yaml - ingress.yaml + # Monitoring stack (kube-prometheus-stack CRs). See docs/DEPLOY-KUBERNETES.md. + - monitoring/servicemonitors.yaml + - monitoring/prometheusrule.yaml - clusterissuer.yaml labels: diff --git a/k8s/monitoring-servicemonitors.yaml b/k8s/monitoring-servicemonitors.yaml new file mode 100644 index 0000000..9602c6a --- /dev/null +++ b/k8s/monitoring-servicemonitors.yaml @@ -0,0 +1,54 @@ +# Monitoring stack for Kubernetes (kube-prometheus-stack). +# +# Prerequisites: install kube-prometheus-stack (Prometheus Operator) into the +# cluster, e.g.: +# +# helm repo add prometheus-community https://prometheus-community.github.io/helm-charts +# helm repo update +# helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \ +# --namespace monitoring --create-namespace +# +# Then apply this file (and the PrometheusRule in monitoring-prometheusrule.yaml): +# +# kubectl apply -f k8s/monitoring-servicemonitors.yaml +# kubectl apply -f k8s/monitoring-prometheusrule.yaml +# +# The ServiceMonitors below replace the static docker-compose scrape targets in +# monitoring/prometheus.yml by letting the Prometheus Operator discover the +# backend and frontend Services via label selectors. +--- +apiVersion: monitoring.coreos.com/v1 +kind: ServiceMonitor +metadata: + name: backend + labels: + release: kube-prometheus-stack +spec: + selector: + matchLabels: + app: backend + namespaceSelector: + matchNames: + - default + endpoints: + - port: http + path: /metrics + interval: 15s +--- +apiVersion: monitoring.coreos.com/v1 +kind: ServiceMonitor +metadata: + name: frontend + labels: + release: kube-prometheus-stack +spec: + selector: + matchLabels: + app: frontend + namespaceSelector: + matchNames: + - default + endpoints: + - port: http + path: /metrics + interval: 15s diff --git a/k8s/prometheus-rule.yaml b/k8s/prometheus-rule.yaml new file mode 100644 index 0000000..999700b --- /dev/null +++ b/k8s/prometheus-rule.yaml @@ -0,0 +1,48 @@ +apiVersion: monitoring.coreos.com/v1 +kind: PrometheusRule +metadata: + name: app-alert-rules + labels: + release: kube-prometheus-stack +spec: + groups: + - name: app.rules + rules: + - alert: BackendDown + expr: up{job="backend"} == 0 + for: 5m + labels: + severity: critical + annotations: + summary: "Backend target is down" + description: "Prometheus has been unable to scrape the backend for more than 5 minutes." + - alert: FrontendDown + expr: up{job="frontend"} == 0 + for: 5m + labels: + severity: warning + annotations: + summary: "Frontend target is down" + description: "Prometheus has been unable to scrape the frontend for more than 5 minutes." + - alert: HighErrorRate + expr: | + sum(rate(http_requests_total{status=~"5.."}[5m])) + / + sum(rate(http_requests_total[5m])) > 0.05 + for: 10m + labels: + severity: critical + annotations: + summary: "High HTTP 5xx error rate" + description: "More than 5% of requests have returned 5xx over the last 10 minutes." + - alert: HighRequestLatency + expr: | + histogram_quantile(0.95, + sum(rate(http_request_duration_seconds_bucket[5m])) by (le) + ) > 1 + for: 10m + labels: + severity: warning + annotations: + summary: "High request latency (p95 > 1s)" + description: "The 95th percentile request latency has exceeded 1s for 10 minutes." diff --git a/k8s/secrets.yaml b/k8s/secrets.yaml index b449580..f33e1e3 100644 --- a/k8s/secrets.yaml +++ b/k8s/secrets.yaml @@ -1,12 +1,30 @@ apiVersion: v1 kind: Secret metadata: - name: backend-secret - namespace: synapsvault - labels: - app.kubernetes.io/name: backend - app.kubernetes.io/component: secrets + name: backend-secrets + namespace: default type: Opaque +stringData: + # Connection pooling for the HPA-scaled backend (up to 10 replicas). + # + # Each replica opens its own pool, so total connections = replicas * pool size. + # Keep this within your Postgres/Supabase connection limit. + # + # Preferred: use Supabase's built-in pooler (PgBouncer) on port 6543 instead of + # the direct connection on port 5432. The pooler multiplexes many client + # connections onto a small number of real Postgres connections. + # + # Direct: postgresql://postgres:@db..supabase.co:5432/postgres + # Pooler: postgresql://postgres.:@aws-0-.pooler.supabase.com:6543/postgres + # + # When using the pooler, keep the per-replica pool small (e.g. 5) so that + # 10 replicas * 5 = 50 client connections stay well within the pooler limit. + # + # If you self-host PgBouncer instead, point DATABASE_URL at the PgBouncer + # service (e.g. pgbouncer:6432) and size the per-replica pool accordingly. + DATABASE_URL: "postgresql://postgres.:@aws-0-.pooler.supabase.com:6543/postgres" + # Per-replica connection pool size. Total = replicas * DB_POOL_SIZE. + DB_POOL_SIZE: "5" # --------------------------------------------------------------------------- # WARNING: DO NOT COMMIT REAL SECRET VALUES TO THIS FILE. #