diff --git a/DECISIONS.md b/DECISIONS.md index 9acd6315..0a7eb873 100644 --- a/DECISIONS.md +++ b/DECISIONS.md @@ -1537,3 +1537,8 @@ The disclosure input parses a form's body as JSON, as it does for a JSON body. **Decision**: For Overslash-managed sign-in (no `org_idp_configs` row claims the provider), login uses the deployment's `{PROVIDER}_AUTH_*` app even when the org has saved OAuth App Credentials (`OAUTH_{PROVIDER}_CLIENT_ID/SECRET` org secrets) for that provider. Those credentials still override env for service connections (SPEC §7). An admin can opt a provider in: a row in `org_oauth_signin_overrides (org_id, provider_key)`, off by default, makes the org's credentials serve sign-in too. It is set from the *Use for sign-in* action on a Google/GitHub row in Org Settings → OAuth App Credentials (`PATCH /v1/org-oauth-credentials/{provider} {override_signin}`), shown only when the org override exists and the env sign-in app is configured, or from the same checkbox on the save form (`PUT … {override_signin}`). Deleting the credentials clears the opt-in. Dedicated IdP rows, including those with `use_org_credentials`, are unchanged: they are already an explicit sign-in choice. There is no backfill, so orgs that had saved Google/GitHub credentials go back to the built-in sign-in app until an admin opts in. `services::org_signin::resolve_org_signin_credentials`; tests in `tests/org_oauth_credentials.rs`. **Rationale**: Admins save org OAuth credentials to brand or scope service connections, and the old precedence quietly moved every member's login onto that app as well. That app may have a narrower consent screen, test-user limits or unverified status, so login could break for reasons unrelated to sign-in. Making the sign-in takeover a separate, visible opt-in keeps "configure connections" from changing "how people log in". +## D-NEXT: `/health` fails when Postgres is down; Cloud Run probes are TCP + +**Date**: 2026-10-27 +**Decision**: `GET /health` returns 503 `status: degraded` when its bounded `SELECT 1` fails, and 200 `status: ok` otherwise. `/ready` keeps the same gate. The Cloud Run startup and liveness probes on the API are `tcp_socket` probes on the container port, not HTTP probes on `/health`. The GCM uptime check behind `[P0] API Down` and the Better Stack monitor stay on `/health`, so a database outage pages in prod. Dev's nightly Cloud SQL stop therefore fires dev's `API Down` every night, and that is accepted: it emails only, and it proves the path end to end. `routes/health.rs`, `infra/modules/cloud-run/main.tf`. +**Rationale**: `/health` was always-200 so that a Cloud SQL blip would not make the probes restart containers. That never worked. The probe's default 1 s timeout is shorter than `/health`'s 2 s database timeout, so with the database gone the liveness probe failed anyway. Dev's instance was shut down about 90 s after every nightly stop, and a replacement cannot boot without the database (`create_app` connects and migrates eagerly), so Cloud Run answered 500 until morning. The design gave neither a surviving container nor an honest status. Splitting the two questions fixes both. Whether the process is up is a TCP connect, which nothing upstream can fail. Whether the API can serve is `/health`, which the monitors read and which no longer restarts anything. A TCP startup probe gives up nothing, because the listener binds only after `create_app` has connected, migrated and preflighted. No new endpoint was added, since a TCP probe needs none. diff --git a/SPEC.md b/SPEC.md index 049431b4..b9978a2a 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1852,7 +1852,7 @@ Every per-IP throttle — these, and the in-handler magic-link, download and upl - **Storage**: Redis/Valkey if available (distributed, accurate across instances); in-memory `DashMap` fallback (single-instance, no external dependency). - **Fail-open**: If Redis becomes unavailable at runtime, requests are allowed through (logged as warning). - **Health endpoint** (`/health`) and **build stamp** (`GET /v1/version`) are exempt from rate limiting. -- **Health vs. readiness**: `/health` is liveness — it reports database reachability in the body (`db`, plus `db_latency_ms` when up; the error itself is logged, never returned, since both endpoints are unauthenticated) but always returns 200, because it backs the Cloud Run startup and liveness probes and a 503 there would restart containers during a database outage. `/ready` runs the same bounded `SELECT 1` and returns **503** when Postgres is unreachable; it is the endpoint to point a load balancer or alerting monitor at. +- **Health vs. liveness**: `/health` is origin health — it runs a bounded `SELECT 1` and returns 200 (`status: ok`) only when Postgres answers, **503** (`status: degraded`) otherwise, reporting `db` (plus `db_latency_ms` when up) in the body; the error itself is logged, never returned, since the endpoint is unauthenticated. It is what the uptime check and the status page probe. `/ready` applies the same gate (`status: ready`). Liveness is not an HTTP endpoint at all: the Cloud Run startup and liveness probes are TCP probes on the container port, so a database outage fails `/health` without restarting containers (D-NEXT). - **Build stamp**: both `/health` and `/ready` also report `version` (release, or the crate version for unreleased builds) and `commit` (full git SHA, or `unknown`). `GET /v1/version` returns the same two values plus `commit_short`, without the database probe — it is unauthenticated for the same reason `/health` is, and is what the dashboard renders in its sidebar footer. The SHA is baked in at compile time (`OVERSLASH_GIT_SHA`, supplied by Cloud Build as `$COMMIT_SHA` or discovered via `git` for local builds). --- diff --git a/STATUS.md b/STATUS.md index 9b8a83ea..a5cadf27 100644 --- a/STATUS.md +++ b/STATUS.md @@ -230,7 +230,7 @@ ### Monitoring & Observability -- OpenTofu module `infra/modules/monitoring/` (PRs #200, #205, #207, #272) deploys 5 GCP dashboards — `overview`, `api-use`, `actions-and-oauth`, `cloudsql-use`, `business` — plus P0/P1/P2 alert policies: API down, API 5xx > 1%, gateway slow-request ratio, Cloud SQL CPU/disk, background-task staleness, OAuth refresh failure ratio, webhook terminal failure ratio, plus uptime checks. +- OpenTofu module `infra/modules/monitoring/` (PRs #200, #205, #207, #272) deploys 5 GCP dashboards — `overview`, `api-use`, `actions-and-oauth`, `cloudsql-use`, `business` — plus P0/P1/P2 alert policies. **Deployed in both envs (measured 2026-10-27, 14 policies + 1 uptime check each):** API down on the `/health` uptime check + API 5xx > 1% (P0); API CPU/memory, gateway slow-request ratio, Cloud SQL CPU/disk, background-task staleness, OAuth refresh failure ratio, upstream error ratio, webhook terminal failure ratio, data integrity, unexpected secret access (P1); API 4xx (P2). Prod P0s page PagerDuty (EU) + email. Dev's API Down fires every night during the Cloud SQL stop, by design. Runbook: [uptime-and-alerts.md](docs/runbooks/uptime-and-alerts.md). - OTel sidecar exports metrics into GMP (instance-label collision fixed in #272). - **Data integrity sweep.** Every metrics-exporter tick also runs `overslash_db::integrity::sweep`: 15 read-only SQL invariants (`crates/overslash-db/src/integrity/*.sql`) over stored references the binding policies (D119, D122) and the relationship gates (#723, #737) would refuse — secret bindings outside the owner's or org vault, bare bindings, pins to a colleague's or another org's connection, pins on org-level instances, BYOC clients across users/orgs/providers, agent-owned secrets, cross-org owners and identity chains, templates from another tier or org, and pending approvals whose resolver is in another org or off the requester's chain. Each emits `custom.googleapis.com/overslash/business/integrity_violations{invariant}` (zeros included) and logs offending rows' ids at WARN; a failing sweep is skipped without losing the business metrics. `[P1] Data Integrity Violation` (any point > 0, or the series absent 30 min) is **gated off** behind `integrity_alert_enabled` until the metric descriptor exists; the pin↔template provider check is not swept (provider isn't in SQL). Runbook: [`docs/runbooks/data-integrity.md`](docs/runbooks/data-integrity.md). - The gateway slow-request alert is a PromQL ratio over diff --git a/TODO.md b/TODO.md index 60035b06..1dc2f424 100644 --- a/TODO.md +++ b/TODO.md @@ -120,7 +120,7 @@ are gaps today. - [ ] Enable container image vulnerability scanning (`containerscanning.googleapis.com`); none of Container Scanning, Container Analysis, Binary Authorization or Security Command Center is enabled. Pairs with the CI-side 6.1.1 work. - [ ] Secret Manager: 16 secrets, none with `rotation`, `next_rotation_time` or `expire_time`. Nothing schedules or alerts on rotation age. (CASA 6.7.1) - [x] **Memorystore has neither AUTH nor transit encryption** — `overslash-prod-valkey` confirmed live: `authEnabled` unset, `transitEncryptionMode: DISABLED`, `BASIC` tier. **In code, pending apply:** `infra/modules/memorystore/main.tf` sets `auth_enabled = true` and `transit_encryption_mode = "SERVER_AUTHENTICATION"`, and publishes the AUTH string and server CA to Secret Manager (`overslash--valkey-auth`, `-valkey-server-ca`), which reach the API (`REDIS_PASSWORD`, `REDIS_CA_CERT`) and oversla-sh (`VALKEY_*`) by `secret_key_ref`. Both clients connect `rediss://` and verify the server against only that CA (`services/valkey.rs`, `oversla-sh/src/storage.rs`). **Operator step still open:** TLS forces a replacement of the instance (downtime, empty keyspace, oversla.sh links lost unless exported) — pre-flight, apply and verify per [memorystore-auth-tls.md](docs/runbooks/memorystore-auth-tls.md). (CASA 6.7.1) -- [ ] **Production has 0 uptime checks and no `[P0] API Down` alert** — 8 alert policies are deployed, which is exactly 12 declared minus 3 disabled in tfvars minus the API-down policy `count`-gated on `api_domain != ""`. The only deployed P0 is `API High 5xx Rate`, which cannot fire when the service returns nothing at all. +- [x] **Production has 0 uptime checks and no `[P0] API Down` alert** — 8 alert policies were deployed (10 on 2026-10-27), which was exactly 12 declared minus 3 disabled in tfvars minus the API-down policy `count`-gated on `api_domain != ""`. The only deployed P0 is `API High 5xx Rate`, which cannot fire when the service returns nothing at all. Root cause: `infra/main.tf` fed the gate `var.domain`, which prod leaves empty because the GCLB serves `api_host_suffix` instead of a domain mapping. It now falls back to `api_host_suffix` when the LB is on, which creates `overslash-prod-api-health` on `api.overslash.com/health` and `[P0] API Down` (PagerDuty + email). The 3 disabled gates (`api_latency`, `oauth_refresh`, `upstream_error`) are on in both envs: their descriptors were measured present on 2026-10-27. The OAuth, upstream and already-deployed webhook ratios clamped a per-second denominator to 1, so at prod volume they could never fire. They now use a `0.001` floor + a minimum-sample guard, like the latency alert. Better Stack (status.overslash.com) still probes the same URL, but it is console-managed and does not page. **Applied 2026-10-27, both envs**, verified per [uptime-and-alerts.md](docs/runbooks/uptime-and-alerts.md): prod has 14 policies (`[P0] API Down` on PagerDuty + email) and `overslash-prod-api-health` passing from all 6 checker regions. Found on the way: dev's `/health` 500'd every night, because the HTTP liveness probe on `/health` (1 s default timeout against a 2 s DB check) killed the instance about 90 s after the Cloud SQL stop, and no replacement could boot. Cloud Run probes are now TCP, and `/health` returns 503 when Postgres is down (D-NEXT), so the prod page covers the database too. **That part is in code, pending a Cloud Run apply + an API deploy.** - [x] **No SSL policy and no Cloud Armor on the production LB** — both `gcloud compute ssl-policies list` and `security-policies list` are empty. TLS 1.2/1.3-only is currently true but unpinned and uncitable. `infra/modules/api-lb/main.tf` now declares `overslash-prod-api-ssl-policy` (TLS 1.2 minimum, `RESTRICTED`) on the HTTPS proxy and `overslash-prod-api-armor` on the backend service — default allow, OWASP CRS rule sets in preview (logged, never blocked) until tuned. **Not applied** — in code, pending a human run of [lb-hardening.md](docs/runbooks/lb-hardening.md), which also lists the evidence commands. (CASA 4.1.1) - [x] Set the org-policy baseline that would make the above guardrails rather than conventions. **Applied 2026-10-27** in both projects. The overspiral.com org now enforces a baseline at org level (measured table in [gcp-posture.md](docs/compliance/casa/gcp-posture.md#supply-chain-and-organization-policy)), and `infra/modules/project-guardrails/` ([project-guardrails.md](docs/runbooks/project-guardrails.md)) pins the Overslash-specific ones per project: `sql.managed.restrictPublicIp` + `restrictAuthorizedNetworks` (enforced in prod, explicitly off on dev's public-IP database), `storage.publicAccessPrevention` and `iam.disableAuditLoggingExemption` (both projects). `iam.managed.disableServiceAccountKeyCreation` is inherited from the org, and dev's hand-set opt-out for the Directory-sync key is adopted into Terraform. Org-wide policies belong to the org repo, not here. - [ ] **`run.allowedIngress`** — unset (allow all) on both projects, measured 2026-10-27. It applies project-wide, so it can only be set to `internal-and-cloud-load-balancing` once overfwd and the shortener stop running with `INGRESS_TRAFFIC_ALL` (prod API is already LB-only). (CASA 6.2.1) diff --git a/crates/overslash-api/src/routes/health.rs b/crates/overslash-api/src/routes/health.rs index bc30e69c..a97c3f42 100644 --- a/crates/overslash-api/src/routes/health.rs +++ b/crates/overslash-api/src/routes/health.rs @@ -10,9 +10,8 @@ use crate::AppState; /// Upper bound on how long a probe may wait for Postgres. /// /// Without this, `PgPool::acquire` blocks for the pool's `acquire_timeout` -/// (30 s by default) when the database is unreachable — long enough for the -/// Cloud Run liveness probe to time out and recycle the container, which is -/// exactly what the always-200 `/health` below exists to prevent. +/// (30 s by default) when the database is unreachable, and the uptime check +/// (10 s timeout) would read a hung database as a timeout rather than a 503. const PROBE_TIMEOUT: Duration = Duration::from_secs(2); // The `version` / `commit` / `sql_policy` fields in both bodies are @@ -87,39 +86,27 @@ async fn probe_db(pool: &PgPool) -> DbProbe { } } -/// Liveness. **Always 200**, even when Postgres is unreachable. +/// Origin health: 200 when the process *and* Postgres are up, 503 otherwise. /// -/// This endpoint backs the Cloud Run startup *and* liveness probes -/// (`infra/modules/cloud-run/main.tf`) plus the Better Stack P0 monitor. If it -/// failed on a Cloud SQL blip, Cloud Run would kill and restart every container -/// mid-outage and the startup probe would block redeploys until the database -/// recovered — the probe would amplify the incident instead of reporting it. -/// -/// So DB state is reported in the body (`db`, plus `db_latency_ms` when up) and -/// never in the status code. For a check that *fails* when the database is -/// down, use [`ready`]. -async fn health(State(state): State) -> Json { - let probe = probe_db(&state.db).await; - let info = build_info(); - let mut body = json!({ - "status": "ok", - "version": info.version, - "commit": info.commit, - "sql_policy": overslash_core::sql_policy::available(), - }); - probe.extend(&mut body); - Json(body) +/// This is what the GCM uptime check (`[P0] API Down`) and the Better Stack +/// monitor probe, so a database outage pages. It used to be always-200 because +/// the Cloud Run probes pointed here; they are TCP probes now +/// (`infra/modules/cloud-run/main.tf`), so nothing restarts a container on a +/// 503 from this endpoint. +async fn health(State(state): State) -> (StatusCode, Json) { + respond(probe_db(&state.db).await, "ok") } -/// Readiness — 503 when Postgres is unreachable. -/// -/// The gated counterpart to [`health`]: this is the one safe to point a -/// load-balancer or an alerting monitor at, because failing it takes the -/// instance out of rotation rather than restarting it. +/// Readiness. Same gate as [`health`]; kept for the callers that use it. async fn ready(State(state): State) -> (StatusCode, Json) { - let probe = probe_db(&state.db).await; + respond(probe_db(&state.db).await, "ready") +} + +/// Shared body + status for both endpoints: `up_status` when the database +/// answered, 503 `degraded` when it did not. +fn respond(probe: DbProbe, up_status: &str) -> (StatusCode, Json) { let (code, status) = if probe.is_up() { - (StatusCode::OK, "ready") + (StatusCode::OK, up_status) } else { (StatusCode::SERVICE_UNAVAILABLE, "degraded") }; @@ -177,6 +164,18 @@ mod tests { ); } + #[test] + fn respond_is_503_degraded_when_db_is_down() { + let (code, Json(body)) = respond(DbProbe::Down, "ok"); + assert_eq!(code, StatusCode::SERVICE_UNAVAILABLE); + assert_eq!(body["status"], "degraded"); + assert_eq!(body["db"], "down"); + + let (code, Json(body)) = respond(DbProbe::Up { latency_ms: 1 }, "ok"); + assert_eq!(code, StatusCode::OK); + assert_eq!(body["status"], "ok"); + } + #[test] fn extend_reports_down_without_detail() { let mut body = json!({ "status": "degraded" }); diff --git a/crates/overslash-api/tests/health.rs b/crates/overslash-api/tests/health.rs index ccb7f0f7..a6bba090 100644 --- a/crates/overslash-api/tests/health.rs +++ b/crates/overslash-api/tests/health.rs @@ -1,14 +1,11 @@ //! Integration tests for `/health` and `/ready`. //! -//! These pin the split that the endpoints exist to encode: +//! Both return 200 when Postgres answers and 503 `degraded` when it does not; +//! `/health` is what the uptime check and Better Stack probe. Cloud Run's own +//! probes are TCP (`infra/modules/cloud-run/main.tf`), so neither endpoint can +//! restart a container. //! -//! 1. `/health` is liveness. It reports database state in the *body* and -//! always returns 200, because Cloud Run's startup and liveness probes -//! point at it — a 503 there would recycle containers mid-outage and -//! block redeploys (see `infra/modules/cloud-run/main.tf`). -//! 2. `/ready` is readiness. It returns 503 when Postgres is unreachable. -//! -//! The unreachable-database branch is covered by a unit test in +//! The unreachable-database branch is covered by unit tests in //! `src/routes/health.rs`, which can point a lazy pool at a dead port without //! disturbing the Postgres instance this suite shares across threads. diff --git a/docs/compliance/casa/gcp-posture.md b/docs/compliance/casa/gcp-posture.md index 431f0b87..22343126 100644 --- a/docs/compliance/casa/gcp-posture.md +++ b/docs/compliance/casa/gcp-posture.md @@ -290,23 +290,52 @@ age, which is the shape of evidence 6.7.1 asks for. ### Monitoring — the count-gate, confirmed +| | 2026-09-22 | 2026-10-27 | +|---|---|---| +| Alert policies deployed (prod) | **8**, all enabled | **10**, all enabled | +| Uptime checks deployed (prod) | **0** | **0** | +| Uptime checks deployed (dev) | 1 | 1 (`overslash-dev-api-health` → `api.dev.overslash.com/health`) | + +The 2026-09-22 arithmetic matched `infra/` exactly: 12 policies declared, minus the 3 +disabled in `infra/env/prod.tfvars` (`api_latency`, `oauth_refresh`, `upstream_error`), +minus `[P0] API Down`, which is `count`-gated on `api_domain != ""` while prod sets +`domain = ""`. **So production has no GCM uptime check and no API-down page**. The only +deployed P0 is `API High 5xx Rate`, which cannot fire when the service is returning +nothing at all. Better Stack ([status-page.md](../../runbooks/status-page.md)) detects the +outage out-of-band, but it is console-managed, outside the reviewable configuration, and +does not page. The two policies added since are `[P1] Data Integrity Violation` and +`[P1] Unexpected Secret Access` (audit-logging module). + +Deployed on 2026-10-27: `[P0] API High 5xx Rate`, `[P1]` API High CPU / API High Memory / +Cloud SQL High CPU / Cloud SQL High Disk / Background Task Stale / Webhook Delivery Failure +Rate / Data Integrity Violation / Unexpected Secret Access, `[P2] API High 4xx Rate`. Both +notification channels exist and are enabled: `overslash-prod PagerDuty (EU)` +(`webhook_tokenauth`) and `overslash-prod Email Alerts`. The 5xx P0 routes to both. + +**The three disabled gates no longer have a reason to stay disabled.** Their descriptors +were measured present in both projects on 2026-10-27: +`overslash_http_request_duration_seconds/histogram`, `overslash_oauth_events_total/counter`, +`overslash_upstream_responses_total/counter`. Reading the queries turned up a second +defect: the OAuth, upstream and (deployed) webhook ratios divided by +`clamp_min(rate, 1)`, which at prod volume (well under 1/s) pins the denominator to 1. They +could not fire, so the deployed webhook policy was counted as coverage it did not provide. + +**Remediation, applied 2026-10-27 on both projects** ([uptime-and-alerts.md](../../runbooks/uptime-and-alerts.md)): +`infra/main.tf` feeds the monitoring module `api_host_suffix` (`api.overslash.com`, served +by the GCLB) when `domain` is empty and the LB is on. That creates the uptime check and +`[P0] API Down` in prod. All three gates are `true` in both tfvars, and the ratio queries +take a `0.001` divide-by-zero floor plus an explicit minimum-sample guard. + +Re-measured after the apply (2026-10-27, prod): + | | | |---|---| -| Alert policies deployed | **8**, all enabled | -| Uptime checks deployed | **0** | - -The arithmetic matches `infra/` exactly: 12 policies declared, minus the 3 disabled at -`infra/env/prod.tfvars:113-119` (`api_latency`, `oauth_refresh`, `upstream_error`), minus -`[P0] API Down`, which is `count`-gated on `api_domain != ""` while prod sets -`domain = ""`. **So production has no GCM uptime check and no API-down page**; the only -deployed P0 is `API High 5xx Rate`, which cannot fire when the service is returning -nothing at all. Better Stack covers the detection out-of-band, but it is console-managed -and outside the reviewable configuration. +| Alert policies deployed | **14**, all enabled. `[P0] API Down` and `[P0] API High 5xx Rate` route to both channels | +| Uptime checks deployed | **1**: `overslash-prod-api-health` → `api.overslash.com/health`, 60s. `check_passed` true from all 6 regions (usa-virginia, usa-oregon, usa-iowa, eur-belgium, apac-singapore, sa-brazil-sao_paulo) | -Deployed: `[P0] API High 5xx Rate`, `[P1]` API High CPU / API High Memory / Cloud SQL High -CPU / Cloud SQL High Disk / Background Task Stale / Webhook Delivery Failure Rate, -`[P2] API High 4xx Rate`. Every one is an availability or capacity signal — there is no -alert on any security event. +Follow-up, in code, pending a Cloud Run apply and an API deploy: `/health` returns 503 +when Postgres is down, and the Cloud Run probes are TCP, so `[P0] API Down` also covers +the database (D-NEXT in DECISIONS.md). ### Storage — production @@ -443,7 +472,7 @@ No verdict is overturned. It does three things: service accounts; the unmanaged `_cloudbuild` bucket holding repository source archives without uniform bucket-level access (fixed and applied 2026-10-23 — [cloud-build-bucket.md](../../runbooks/cloud-build-bucket.md)); and the confirmation that production runs 8 alert - policies and **zero** uptime checks, so the `[P0] API Down` page does not exist. + policies and **zero** uptime checks, so the `[P0] API Down` page does not exist. (Root cause and fix, pending apply: [uptime-and-alerts.md](../../runbooks/uptime-and-alerts.md).) And it supplies better evidence than prose: "Google's own Recommender rates this P2, with a HIGH-severity insight behind it" is a stronger line in a submission than "we read our diff --git a/docs/runbooks/status-page.md b/docs/runbooks/status-page.md index 52aa7952..56bc35b6 100644 --- a/docs/runbooks/status-page.md +++ b/docs/runbooks/status-page.md @@ -8,13 +8,12 @@ Public URL: - **Components**: `api.overslash.com` and `app.overslash.com`, each backed by an HTTPS check on 180s cadence across the `eu`/`us`/`as`/`au` regions. - **Monitors** (managed in the Better Stack console, not IaC): - - `api.overslash.com/health` — plain `status` (HTTP up) check. The body also - carries `"db": "up" | "down"` (plus `db_latency_ms` when up), but - `/health` returns **200 even when `db` is `down`** — by design. It backs the - Cloud Run startup *and* liveness probes, so failing it on a Cloud SQL blip - would recycle every container mid-outage and block redeploys until the - database recovered. Use `/ready` for a check that 503s on an unreachable - database; no probe or monitor points at it yet. + - `api.overslash.com/health` — plain `status` (HTTP up) check. `/health` + returns **503 when the database is unreachable** (`"db": "down"`, + `"status": "degraded"`), so this monitor goes red on a Cloud SQL outage as + well as a dead process. It no longer backs the Cloud Run probes, which are + TCP (`infra/modules/cloud-run/main.tf`), so a 503 here never recycles a + container. - `app.overslash.com` — `keyword` check for `_app/immutable`. The dashboard is a SvelteKit SPA shell with no SSR text, so this marker confirms the real app shell is served rather than a Fastly/router error page (e.g. @@ -27,9 +26,11 @@ Public URL: — if GCM itself has an outage the public page stays truthful. The Better Stack monitors mirror the *target* of the GCM uptime check in -`infra/modules/monitoring/uptime.tf`. The GCM `[P0] api_down` alert -(`infra/modules/monitoring/alerts_p0.tf`) continues to page on-call -independently. +`infra/modules/monitoring/uptime.tf`. The GCM `[P0] API Down` alert +(`infra/modules/monitoring/alerts_p0.tf`) is what pages on-call. Until the change +in [uptime-and-alerts.md](uptime-and-alerts.md) is applied, **prod has neither**: +both were `count`-gated on a `domain` that prod leaves empty, so a total prod outage +shows here and pages no one. ## Who has access @@ -65,9 +66,9 @@ is structural. ## On-call -Page is wired only as a status signal. Paging is via the GCM P0 alert pipeline -(PagerDuty integration tracked in TODO.md §1.4, first bullet). When that lands, -link the rotation schedule from this page. +Page is wired only as a status signal. Paging is via the GCM P0 alert pipeline, +which routes to PagerDuty (EU) and email in prod. The API-down half of that +pipeline is [uptime-and-alerts.md](uptime-and-alerts.md). ## Verifying after changes @@ -78,8 +79,9 @@ To sanity-check the monitor without faking an outage: - The Better Stack monitor page shows the last probe time and HTTP response. During an incident, `/health` distinguishes "the process is wedged" (no -response, or a slow one) from "the process is fine but can't reach Postgres" -(`"db":"down"`) without shelling into the container. The body deliberately +response, a slow one, or Cloud Run's own 500 *"the instance could not start +successfully"*) from "the process is fine but can't reach Postgres" (503 with +`"db":"down"`) without shelling into the container. The body deliberately omits the database error — the endpoint is unauthenticated, and a sqlx error names hosts and roles (CASA 6.2.1) — so read the cause from the API's logs (`health probe: database query failed`). The probe is bounded at 2s, so a hung diff --git a/docs/runbooks/uptime-and-alerts.md b/docs/runbooks/uptime-and-alerts.md new file mode 100644 index 00000000..8290fe0f --- /dev/null +++ b/docs/runbooks/uptime-and-alerts.md @@ -0,0 +1,163 @@ +# Uptime check, `[P0] API Down`, and the PromQL alert gates + +Infra is never auto-applied. This page is the human half of the change that gives +production a GCM uptime check and a paging API-down alert, and switches on the three +PromQL alerts that were off in every environment. + +## What changes + +| | dev | prod | +|---|---|---| +| Uptime check `overslash--api-health` on `https:///health` | already exists (`api.dev.overslash.com`) | **created** on `api.overslash.com` | +| `[P0] overslash- API Down` | already exists | **created**, routed to PagerDuty (EU) + email | +| Overview dashboard uptime tile | already wired | **dashboard replaced** so the tile points at the new check | +| `[P1] API Slow Requests` | **created** | **created** | +| `[P1] OAuth Refresh Failure Rate` | **created** | **created** | +| `[P1] Upstream Error Rate` | **created** | **created** | +| `[P1] Webhook Delivery Failure Rate` | query updated in place | query updated in place | +| Actions & OAuth dashboard | replaced (two ratio charts) | replaced (two ratio charts) | + +### Why prod had no uptime check + +`modules/monitoring` creates the uptime check and `[P0] API Down` only when `api_domain` is +non-empty, and the root module fed it `var.domain`. Prod leaves `domain = ""` on purpose: +the GCLB serves `api.overslash.com` and `*.api.overslash.com` (`api_host_suffix`) instead +of a Cloud Run domain mapping. So both resources were silently dropped. The only P0 left +was `API High 5xx Rate`, which cannot fire when the API returns nothing at all. + +`infra/main.tf` now passes `domain`, or `api_host_suffix` when `domain` is empty and the +LB is on. With neither set, nothing public answers, and the check stays off. + +### Why the three PromQL alerts were off, and why they are on now + +Each one is gated (`*_alert_enabled` in tfvars) because GMP rejects a policy whose metric +descriptor does not exist yet. The descriptors were measured present in **both** projects +on 2026-10-27: + +``` +prometheus.googleapis.com/overslash_http_request_duration_seconds/histogram +prometheus.googleapis.com/overslash_oauth_events_total/counter +prometheus.googleapis.com/overslash_upstream_responses_total/counter +``` + +So all three flags are `true` in both tfvars. **But turning on the OAuth and upstream +alerts as they were would have added two alerts that cannot fire.** They divided by +`clamp_min(sum(rate(...)), 1)`. Prod traffic is well under one event per second, so the +denominator was always clamped to 1, and the "ratio" was really failures per second, +which never reaches 0.10 or 0.20. The deployed webhook alert had the same flaw. All three +now copy the latency alert's pattern: a `0.001` floor that only guards against division +by zero, plus an explicit `and` that requires a minimum sample in the window: + +| Alert | Window | Minimum sample | Fires at | +|---|---|---|---| +| OAuth Refresh Failure Rate | 15m | rate > 0.011/s (~10 refreshes) | > 10% failed | +| Upstream Error Rate | 15m | rate > 0.022/s (~20 responses) | > 20% `5xx` / `error` | +| Webhook Delivery Failure Rate | 30m | rate > 0.0056/s (~10 terminal deliveries) | > 5% failed | + +The two matching dashboard charts on *Actions & OAuth* had the same clamp, so they showed +ratios close to zero. They now use the same `0.001` floor. + +### What `/health` reports, and dev at night + +`/health` returns 503 when Postgres is unreachable (D-NEXT in DECISIONS.md), so +`[P0] API Down` covers the API's database as well as the process. Cloud Run's +own startup and liveness probes are TCP probes and never read `/health`, so a +503 never restarts a container. Before that change, an HTTP liveness probe on +`/health` timed out (1 s default, against a 2 s DB check) once the database was +gone, and it shut the instance down about 90 s after each dev stop. + +Dev stops Cloud SQL from 23:00 to 07:00 Europe/Madrid (`infra-scheduler`), so dev's +check fails every night and `[P0] overslash-dev API Down` emails +alert@overspiral.com nightly. That is deliberate: it tests the alert path +end to end. Prod has no scheduler. + +### Relationship to the status page + +[status.overslash.com](status-page.md) (Better Stack) probes the same `/health` URL from +outside GCP. It is a public status signal, managed in the Better Stack console, and it +does not page. Paging comes from this GCM check: it is in IaC, reviewable, and routed +to PagerDuty. Keep both. + +## Apply + +Dev first. That is where GMP gets to reject a malformed PromQL query before prod sees it. + +```bash +make tofu-plan ENV=dev +# expect: 4 to add, 2 to change, 1 to destroy — all under module.monitoring +make tofu-apply ENV=dev + +make tofu-plan ENV=prod +# expect, under module.monitoring only: +# + google_monitoring_uptime_check_config.api_health[0] +# + google_monitoring_alert_policy.api_down[0] +# + google_monitoring_alert_policy.{api_slow_requests,oauth_refresh_failure_rate,upstream_error_rate}[0] +# ~ google_monitoring_alert_policy.webhook_failure_rate[0] +# -/+ google_monitoring_dashboard.{overview,actions_and_oauth} +# ~ terraform_data.dashboard_hash["overview"], ["actions_and_oauth"] +make tofu-apply ENV=prod +``` + +A full prod plan also shows whatever else is pending apply on `dev`. Read the whole plan, +not only the monitoring lines. To apply only this change: +`tofu plan -var-file=env/prod.tfvars -target=module.monitoring -out=prod.tfplan`. + +The agent sandbox could only make a `-refresh=false -target=module.monitoring` plan: the +module's `depends_on` drags in IAM, which the factory SA cannot read, and the plan stops at +the PagerDuty secret read. Up to that point the prod plan matched the list above, minus +`api_down`, whose channels depend on that secret. + +## Verify + +```bash +# 1. The uptime check exists and targets the apex. +gcloud monitoring uptime list-configs --project=overslash \ + --format="table(displayName,monitoredResource.labels.host,httpCheck.path,period)" +# expect: overslash-prod-api-health api.overslash.com /health 60s + +# 2. It passes from every checker region (give it ~5 minutes after apply). +# Console: Monitoring → Uptime checks → overslash-prod-api-health. +# Or the metric itself (replace with the check's ID, the last segment of `name`): +curl -s -G -H "Authorization: Bearer $(gcloud auth print-access-token)" \ + "https://monitoring.googleapis.com/v3/projects/overslash/timeSeries" \ + --data-urlencode 'filter=metric.type="monitoring.googleapis.com/uptime_check/check_passed" AND metric.labels.check_id=""' \ + --data-urlencode "interval.startTime=$(date -u -d '10 min ago' +%FT%TZ)" \ + --data-urlencode "interval.endTime=$(date -u +%FT%TZ)" \ + | jq -r '.timeSeries[] | [.metric.labels.checker_location, .points[0].value.boolValue] | @tsv' +# expect: one row per region, all true + +# 3. The alert policies exist, are enabled, and P0 pages both channels. +curl -s -H "Authorization: Bearer $(gcloud auth print-access-token)" \ + "https://monitoring.googleapis.com/v3/projects/overslash/alertPolicies?pageSize=100" \ + | jq -r '.alertPolicies[] | [.displayName, .enabled, (.notificationChannels|length)] | @tsv' | sort +# expect 14: the 10 already there + [P0] API Down (2 channels) + API Slow Requests, +# OAuth Refresh Failure Rate, Upstream Error Rate + +# 4. The new dashboard tile resolves: Monitoring → Dashboards → overslash-prod Overview, +# the uptime chart shows data instead of "No data". +``` + +`gcloud alpha monitoring policies list` works too, if the alpha component is installed. + +### Test the page without an outage (optional) + +Do this on dev. It pages the dev email channel only: + +```bash +ID=$(gcloud monitoring uptime list-configs --project=overslash-dev \ + --filter='displayName=overslash-dev-api-health' --format='value(name.basename())') +# Point the check at a path that 404s. This drifts from Terraform on purpose. +gcloud monitoring uptime update "$ID" --project=overslash-dev --path=/health-uptime-drill +# Wait ~3-5 minutes (2 failing 60s probes from ≥2 regions); expect +# "[P0] overslash-dev API Down" to open and an email to alert@overspiral.com. +# Revert, either by hand or with an apply, which puts /health back: +gcloud monitoring uptime update "$ID" --project=overslash-dev --path=/health +``` + +There is no safe way to do it in prod short of an outage. The dev run plus check (3) +above, which shows the prod policy wired to PagerDuty, is the evidence. + +## Evidence for CASA + +Record in [gcp-posture.md](../compliance/casa/gcp-posture.md#monitoring--the-count-gate-confirmed): +the output of steps 1 and 3 with the date, and the policy count going from 10 to 14. diff --git a/infra/env/dev.tfvars b/infra/env/dev.tfvars index c66b7c59..72a8a8fc 100644 --- a/infra/env/dev.tfvars +++ b/infra/env/dev.tfvars @@ -148,15 +148,21 @@ terraform_operators = [ # where is the series named against each flag below. Verify before # flipping, one flag at a time. +# All three descriptors were measured present in both projects on 2026-10-27 +# (`.../overslash_http_request_duration_seconds/histogram`, +# `.../overslash_oauth_events_total/counter`, +# `.../overslash_upstream_responses_total/counter`), so all three are on. +# The two ratio queries were fixed in the same change: they clamped a +# per-second denominator to 1, which at our volume made them unable to fire. +# Runbook: docs/runbooks/uptime-and-alerts.md. + # `overslash_http_request_duration_seconds` must be live as a *histogram*, not # the summary it rendered as before the bucket set landed. -api_latency_alert_enabled = false +api_latency_alert_enabled = true -# `overslash_oauth_events_total` and `overslash_upstream_responses_total`. Both -# have most likely been emitted by now; verify the descriptors and flip these on -# in a follow-up. -oauth_refresh_alert_enabled = false -upstream_error_alert_enabled = false +# `overslash_oauth_events_total` and `overslash_upstream_responses_total`. +oauth_refresh_alert_enabled = true +upstream_error_alert_enabled = true # `custom.googleapis.com/overslash/business/integrity_violations` — written by # the metrics exporter, not GMP, but the descriptor rule is the same. gcloud has diff --git a/infra/env/prod.tfvars b/infra/env/prod.tfvars index 38bdfcdf..49c17b1a 100644 --- a/infra/env/prod.tfvars +++ b/infra/env/prod.tfvars @@ -148,15 +148,21 @@ terraform_operators = [ # where is the series named against each flag below. Verify before # flipping, one flag at a time. +# All three descriptors were measured present in both projects on 2026-10-27 +# (`.../overslash_http_request_duration_seconds/histogram`, +# `.../overslash_oauth_events_total/counter`, +# `.../overslash_upstream_responses_total/counter`), so all three are on. +# The two ratio queries were fixed in the same change: they clamped a +# per-second denominator to 1, which at our volume made them unable to fire. +# Runbook: docs/runbooks/uptime-and-alerts.md. + # `overslash_http_request_duration_seconds` must be live as a *histogram*, not # the summary it rendered as before the bucket set landed. -api_latency_alert_enabled = false +api_latency_alert_enabled = true -# `overslash_oauth_events_total` and `overslash_upstream_responses_total`. Both -# have most likely been emitted by now; verify the descriptors and flip these on -# in a follow-up. -oauth_refresh_alert_enabled = false -upstream_error_alert_enabled = false +# `overslash_oauth_events_total` and `overslash_upstream_responses_total`. +oauth_refresh_alert_enabled = true +upstream_error_alert_enabled = true # `custom.googleapis.com/overslash/business/integrity_violations` — written by # the metrics exporter, not GMP, but the descriptor rule is the same. gcloud has diff --git a/infra/main.tf b/infra/main.tf index 1bd697dc..299c47a8 100644 --- a/infra/main.tf +++ b/infra/main.tf @@ -313,11 +313,16 @@ module "monitoring" { upstream_error_alert_enabled = var.upstream_error_alert_enabled api_latency_alert_enabled = var.api_latency_alert_enabled integrity_alert_enabled = var.integrity_alert_enabled - api_domain = var.domain - api_service_name = module.cloud_run.service_name - cloud_sql_instance_name = module.cloud_sql.instance_name - monthly_budget_usd = var.monthly_budget_usd - billing_account_id = var.billing_account_id + # The host the uptime check probes. Prod leaves `domain` empty (the GCLB + # serves `api_host_suffix` and its wildcard instead of a Cloud Run domain + # mapping), which used to leave this empty too and silently drop both the + # uptime check and `[P0] API Down`. Without the LB and without `domain` + # nothing public answers on `api_host_suffix`, so it stays empty there. + api_domain = var.domain != "" ? var.domain : (var.enable_api_lb ? var.api_host_suffix : "") + api_service_name = module.cloud_run.service_name + cloud_sql_instance_name = module.cloud_sql.instance_name + monthly_budget_usd = var.monthly_budget_usd + billing_account_id = var.billing_account_id depends_on = [ google_project_service.apis, diff --git a/infra/modules/cloud-run/main.tf b/infra/modules/cloud-run/main.tf index 06df2aa8..2545f741 100644 --- a/infra/modules/cloud-run/main.tf +++ b/infra/modules/cloud-run/main.tf @@ -694,9 +694,16 @@ resource "google_cloud_run_v2_service" "api" { mount_path = "/cloudsql" } + # TCP, not `/health`: the probes ask only "is the process up", never + # "are its dependencies up". `/health` checks Postgres and 503s when it + # is down, and an HTTP liveness probe on it killed every instance + # within ~90s of a Cloud SQL stop (dev, nightly) — a replacement then + # cannot boot either, so the probe turned a DB outage into an API one. + # The startup probe loses nothing: the listener binds only after + # `create_app` has connected, migrated and preflighted. + # Dependency health is the uptime check's job (modules/monitoring). startup_probe { - http_get { - path = "/health" + tcp_socket { port = var.container_port } initial_delay_seconds = 5 @@ -705,8 +712,7 @@ resource "google_cloud_run_v2_service" "api" { } liveness_probe { - http_get { - path = "/health" + tcp_socket { port = var.container_port } period_seconds = 30 diff --git a/infra/modules/monitoring/alerts_p1.tf b/infra/modules/monitoring/alerts_p1.tf index 14e00c6f..d618cbcb 100644 --- a/infra/modules/monitoring/alerts_p1.tf +++ b/infra/modules/monitoring/alerts_p1.tf @@ -197,11 +197,19 @@ resource "google_monitoring_alert_policy" "oauth_refresh_failure_rate" { display_name = "OAuth refresh failure ratio > 10%" condition_prometheus_query_language { + # The floor is a divide-by-zero guard only (see `api_slow_requests`). A + # `clamp_min(..., 1)` here, as this used to be, pins the denominator at 1 + # whenever traffic is under 1/s — always, in prod — so the "ratio" was + # failures per second and never reached 0.10. The `and` requires ~10 + # refreshes in the window instead. query = <<-PROMQL - sum(rate(overslash_oauth_events_total{flow="refresh",status="failure"}[15m])) - / - clamp_min(sum(rate(overslash_oauth_events_total{flow="refresh"}[15m])), 1) - > 0.10 + ( + sum(rate(overslash_oauth_events_total{flow="refresh",status="failure"}[15m])) + / + clamp_min(sum(rate(overslash_oauth_events_total{flow="refresh"}[15m])), 0.001) + ) > 0.10 + and + sum(rate(overslash_oauth_events_total{flow="refresh"}[15m])) > 0.011 PROMQL duration = "900s" evaluation_interval = "60s" @@ -235,11 +243,16 @@ resource "google_monitoring_alert_policy" "upstream_error_rate" { display_name = "Upstream error ratio > 20%" condition_prometheus_query_language { + # Same floor + sample guard as the OAuth alert above: ~20 upstream + # responses in the window. query = <<-PROMQL - sum(rate(overslash_upstream_responses_total{job="${var.api_service_name}",status_class=~"5xx|error"}[15m])) - / - clamp_min(sum(rate(overslash_upstream_responses_total{job="${var.api_service_name}"}[15m])), 1) - > 0.20 + ( + sum(rate(overslash_upstream_responses_total{job="${var.api_service_name}",status_class=~"5xx|error"}[15m])) + / + clamp_min(sum(rate(overslash_upstream_responses_total{job="${var.api_service_name}"}[15m])), 0.001) + ) > 0.20 + and + sum(rate(overslash_upstream_responses_total{job="${var.api_service_name}"}[15m])) > 0.022 PROMQL duration = "900s" evaluation_interval = "60s" @@ -270,11 +283,16 @@ resource "google_monitoring_alert_policy" "webhook_failure_rate" { display_name = "Webhook terminal failure ratio > 5%" condition_prometheus_query_language { + # Same floor + sample guard as the OAuth alert above: ~10 terminal + # deliveries in the window. query = <<-PROMQL - sum(rate(overslash_webhook_deliveries_total{status="failed",final="true"}[30m])) - / - clamp_min(sum(rate(overslash_webhook_deliveries_total{final="true"}[30m])), 1) - > 0.05 + ( + sum(rate(overslash_webhook_deliveries_total{status="failed",final="true"}[30m])) + / + clamp_min(sum(rate(overslash_webhook_deliveries_total{final="true"}[30m])), 0.001) + ) > 0.05 + and + sum(rate(overslash_webhook_deliveries_total{final="true"}[30m])) > 0.0056 PROMQL duration = "1800s" evaluation_interval = "60s" @@ -322,11 +340,11 @@ resource "google_monitoring_alert_policy" "api_slow_requests" { display_name = "More than 5% of gateway requests slower than 2.5s" condition_prometheus_query_language { - # Note the departure from this module's `clamp_min(..., 1)` idiom: at prod - # volume the denominator is a per-second rate that is legitimately below - # 1, and clamping there would silently deflate the ratio toward zero. The - # floor is only a divide-by-zero guard; the explicit `and` rate guard is - # what enforces a meaningful sample size, legibly. + # Never `clamp_min(..., 1)` a per-second rate: at prod volume the + # denominator is legitimately below 1, and clamping there would silently + # deflate the ratio toward zero. The floor is only a divide-by-zero + # guard; the explicit `and` rate guard is what enforces a meaningful + # sample size, legibly. Every ratio alert in this file follows it. query = <<-PROMQL ( 1 - ( diff --git a/infra/modules/monitoring/dashboards/actions-and-oauth.json.tftpl b/infra/modules/monitoring/dashboards/actions-and-oauth.json.tftpl index 82566260..2c7dc516 100644 --- a/infra/modules/monitoring/dashboards/actions-and-oauth.json.tftpl +++ b/infra/modules/monitoring/dashboards/actions-and-oauth.json.tftpl @@ -56,7 +56,7 @@ "xyChart": { "dataSets": [{ "timeSeriesQuery": { - "prometheusQuery": "sum by (template_key) (rate(overslash_upstream_responses_total{status_class=~\"5xx|error\"}[15m])) / clamp_min(sum by (template_key) (rate(overslash_upstream_responses_total[15m])), 1)" + "prometheusQuery": "sum by (template_key) (rate(overslash_upstream_responses_total{status_class=~\"5xx|error\"}[15m])) / clamp_min(sum by (template_key) (rate(overslash_upstream_responses_total[15m])), 0.001)" }, "plotType": "LINE" }], @@ -87,7 +87,7 @@ "xyChart": { "dataSets": [{ "timeSeriesQuery": { - "prometheusQuery": "sum by (provider) (rate(overslash_oauth_events_total{flow=\"refresh\",status=\"failure\"}[15m])) / clamp_min(sum by (provider) (rate(overslash_oauth_events_total{flow=\"refresh\"}[15m])), 1)" + "prometheusQuery": "sum by (provider) (rate(overslash_oauth_events_total{flow=\"refresh\",status=\"failure\"}[15m])) / clamp_min(sum by (provider) (rate(overslash_oauth_events_total{flow=\"refresh\"}[15m])), 0.001)" }, "plotType": "LINE" }],