feat(infra): prod uptime check + [P0] API Down; /health fails on DB down, TCP probes

DECISIONS.md

@@ -1537,3 +1537,8 @@ The disclosure input parses a form's body as JSON, as it does for a JSON body.
 **Decision**: For Overslash-managed sign-in (no `org_idp_configs` row claims the provider), login uses the deployment's `{PROVIDER}_AUTH_*` app even when the org has saved OAuth App Credentials (`OAUTH_{PROVIDER}_CLIENT_ID/SECRET` org secrets) for that provider. Those credentials still override env for service connections (SPEC §7). An admin can opt a provider in: a row in `org_oauth_signin_overrides (org_id, provider_key)`, off by default, makes the org's credentials serve sign-in too. It is set from the *Use for sign-in* action on a Google/GitHub row in Org Settings → OAuth App Credentials (`PATCH /v1/org-oauth-credentials/{provider} {override_signin}`), shown only when the org override exists and the env sign-in app is configured, or from the same checkbox on the save form (`PUT … {override_signin}`). Deleting the credentials clears the opt-in. Dedicated IdP rows, including those with `use_org_credentials`, are unchanged: they are already an explicit sign-in choice. There is no backfill, so orgs that had saved Google/GitHub credentials go back to the built-in sign-in app until an admin opts in. `services::org_signin::resolve_org_signin_credentials`; tests in `tests/org_oauth_credentials.rs`.
 **Rationale**: Admins save org OAuth credentials to brand or scope service connections, and the old precedence quietly moved every member's login onto that app as well. That app may have a narrower consent screen, test-user limits or unverified status, so login could break for reasons unrelated to sign-in. Making the sign-in takeover a separate, visible opt-in keeps "configure connections" from changing "how people log in".
 
+## D-NEXT: `/health` fails when Postgres is down; Cloud Run probes are TCP
+
+**Date**: 2026-10-27
+**Decision**: `GET /health` returns 503 `status: degraded` when its bounded `SELECT 1` fails, and 200 `status: ok` otherwise. `/ready` keeps the same gate. The Cloud Run startup and liveness probes on the API are `tcp_socket` probes on the container port, not HTTP probes on `/health`. The GCM uptime check behind `[P0] API Down` and the Better Stack monitor stay on `/health`, so a database outage pages in prod. Dev's nightly Cloud SQL stop therefore fires dev's `API Down` every night, and that is accepted: it emails only, and it proves the path end to end. `routes/health.rs`, `infra/modules/cloud-run/main.tf`.
+**Rationale**: `/health` was always-200 so that a Cloud SQL blip would not make the probes restart containers. That never worked. The probe's default 1 s timeout is shorter than `/health`'s 2 s database timeout, so with the database gone the liveness probe failed anyway. Dev's instance was shut down about 90 s after every nightly stop, and a replacement cannot boot without the database (`create_app` connects and migrates eagerly), so Cloud Run answered 500 until morning. The design gave neither a surviving container nor an honest status. Splitting the two questions fixes both. Whether the process is up is a TCP connect, which nothing upstream can fail. Whether the API can serve is `/health`, which the monitors read and which no longer restarts anything. A TCP startup probe gives up nothing, because the listener binds only after `create_app` has connected, migrated and preflighted. No new endpoint was added, since a TCP probe needs none.

SPEC.md

@@ -1852,7 +1852,7 @@ Every per-IP throttle — these, and the in-handler magic-link, download and upl
 - **Storage**: Redis/Valkey if available (distributed, accurate across instances); in-memory `DashMap` fallback (single-instance, no external dependency).
 - **Fail-open**: If Redis becomes unavailable at runtime, requests are allowed through (logged as warning).
 - **Health endpoint** (`/health`) and **build stamp** (`GET /v1/version`) are exempt from rate limiting.
-- **Health vs. readiness**: `/health` is liveness — it reports database reachability in the body (`db`, plus `db_latency_ms` when up; the error itself is logged, never returned, since both endpoints are unauthenticated) but always returns 200, because it backs the Cloud Run startup and liveness probes and a 503 there would restart containers during a database outage. `/ready` runs the same bounded `SELECT 1` and returns **503** when Postgres is unreachable; it is the endpoint to point a load balancer or alerting monitor at.
+- **Health vs. liveness**: `/health` is origin health — it runs a bounded `SELECT 1` and returns 200 (`status: ok`) only when Postgres answers, **503** (`status: degraded`) otherwise, reporting `db` (plus `db_latency_ms` when up) in the body; the error itself is logged, never returned, since the endpoint is unauthenticated. It is what the uptime check and the status page probe. `/ready` applies the same gate (`status: ready`). Liveness is not an HTTP endpoint at all: the Cloud Run startup and liveness probes are TCP probes on the container port, so a database outage fails `/health` without restarting containers (D-NEXT).
 - **Build stamp**: both `/health` and `/ready` also report `version` (release, or the crate version for unreleased builds) and `commit` (full git SHA, or `unknown`). `GET /v1/version` returns the same two values plus `commit_short`, without the database probe — it is unauthenticated for the same reason `/health` is, and is what the dashboard renders in its sidebar footer. The SHA is baked in at compile time (`OVERSLASH_GIT_SHA`, supplied by Cloud Build as `$COMMIT_SHA` or discovered via `git` for local builds).
 
 ---

STATUS.md

@@ -230,7 +230,7 @@
 
 ### Monitoring & Observability
 
-- OpenTofu module `infra/modules/monitoring/` (PRs #200, #205, #207, #272) deploys 5 GCP dashboards — `overview`, `api-use`, `actions-and-oauth`, `cloudsql-use`, `business` — plus P0/P1/P2 alert policies: API down, API 5xx > 1%, gateway slow-request ratio, Cloud SQL CPU/disk, background-task staleness, OAuth refresh failure ratio, webhook terminal failure ratio, plus uptime checks.
+- OpenTofu module `infra/modules/monitoring/` (PRs #200, #205, #207, #272) deploys 5 GCP dashboards — `overview`, `api-use`, `actions-and-oauth`, `cloudsql-use`, `business` — plus P0/P1/P2 alert policies. **Deployed in both envs (measured 2026-10-27, 14 policies + 1 uptime check each):** API down on the `/health` uptime check + API 5xx > 1% (P0); API CPU/memory, gateway slow-request ratio, Cloud SQL CPU/disk, background-task staleness, OAuth refresh failure ratio, upstream error ratio, webhook terminal failure ratio, data integrity, unexpected secret access (P1); API 4xx (P2). Prod P0s page PagerDuty (EU) + email. Dev's API Down fires every night during the Cloud SQL stop, by design. Runbook: [uptime-and-alerts.md](docs/runbooks/uptime-and-alerts.md).
 - OTel sidecar exports metrics into GMP (instance-label collision fixed in #272).
 - **Data integrity sweep.** Every metrics-exporter tick also runs `overslash_db::integrity::sweep`: 15 read-only SQL invariants (`crates/overslash-db/src/integrity/*.sql`) over stored references the binding policies (D119, D122) and the relationship gates (#723, #737) would refuse — secret bindings outside the owner's or org vault, bare bindings, pins to a colleague's or another org's connection, pins on org-level instances, BYOC clients across users/orgs/providers, agent-owned secrets, cross-org owners and identity chains, templates from another tier or org, and pending approvals whose resolver is in another org or off the requester's chain. Each emits `custom.googleapis.com/overslash/business/integrity_violations{invariant}` (zeros included) and logs offending rows' ids at WARN; a failing sweep is skipped without losing the business metrics. `[P1] Data Integrity Violation` (any point > 0, or the series absent 30 min) is **gated off** behind `integrity_alert_enabled` until the metric descriptor exists; the pin↔template provider check is not swept (provider isn't in SQL). Runbook: [`docs/runbooks/data-integrity.md`](docs/runbooks/data-integrity.md).
 - The gateway slow-request alert is a PromQL ratio over

TODO.md

@@ -120,7 +120,7 @@ are gaps today.
 - [ ] Enable container image vulnerability scanning (`containerscanning.googleapis.com`); none of Container Scanning, Container Analysis, Binary Authorization or Security Command Center is enabled. Pairs with the CI-side 6.1.1 work.
 - [ ] Secret Manager: 16 secrets, none with `rotation`, `next_rotation_time` or `expire_time`. Nothing schedules or alerts on rotation age. (CASA 6.7.1)
 - [x] **Memorystore has neither AUTH nor transit encryption** — `overslash-prod-valkey` confirmed live: `authEnabled` unset, `transitEncryptionMode: DISABLED`, `BASIC` tier. **In code, pending apply:** `infra/modules/memorystore/main.tf` sets `auth_enabled = true` and `transit_encryption_mode = "SERVER_AUTHENTICATION"`, and publishes the AUTH string and server CA to Secret Manager (`overslash-<env>-valkey-auth`, `-valkey-server-ca`), which reach the API (`REDIS_PASSWORD`, `REDIS_CA_CERT`) and oversla-sh (`VALKEY_*`) by `secret_key_ref`. Both clients connect `rediss://` and verify the server against only that CA (`services/valkey.rs`, `oversla-sh/src/storage.rs`). **Operator step still open:** TLS forces a replacement of the instance (downtime, empty keyspace, oversla.sh links lost unless exported) — pre-flight, apply and verify per [memorystore-auth-tls.md](docs/runbooks/memorystore-auth-tls.md). (CASA 6.7.1)
-- [ ] **Production has 0 uptime checks and no `[P0] API Down` alert** — 8 alert policies are deployed, which is exactly 12 declared minus 3 disabled in tfvars minus the API-down policy `count`-gated on `api_domain != ""`. The only deployed P0 is `API High 5xx Rate`, which cannot fire when the service returns nothing at all.
+- [x] **Production has 0 uptime checks and no `[P0] API Down` alert** — 8 alert policies were deployed (10 on 2026-10-27), which was exactly 12 declared minus 3 disabled in tfvars minus the API-down policy `count`-gated on `api_domain != ""`. The only deployed P0 is `API High 5xx Rate`, which cannot fire when the service returns nothing at all. Root cause: `infra/main.tf` fed the gate `var.domain`, which prod leaves empty because the GCLB serves `api_host_suffix` instead of a domain mapping. It now falls back to `api_host_suffix` when the LB is on, which creates `overslash-prod-api-health` on `api.overslash.com/health` and `[P0] API Down` (PagerDuty + email). The 3 disabled gates (`api_latency`, `oauth_refresh`, `upstream_error`) are on in both envs: their descriptors were measured present on 2026-10-27. The OAuth, upstream and already-deployed webhook ratios clamped a per-second denominator to 1, so at prod volume they could never fire. They now use a `0.001` floor + a minimum-sample guard, like the latency alert. Better Stack (status.overslash.com) still probes the same URL, but it is console-managed and does not page. **Applied 2026-10-27, both envs**, verified per [uptime-and-alerts.md](docs/runbooks/uptime-and-alerts.md): prod has 14 policies (`[P0] API Down` on PagerDuty + email) and `overslash-prod-api-health` passing from all 6 checker regions. Found on the way: dev's `/health` 500'd every night, because the HTTP liveness probe on `/health` (1 s default timeout against a 2 s DB check) killed the instance about 90 s after the Cloud SQL stop, and no replacement could boot. Cloud Run probes are now TCP, and `/health` returns 503 when Postgres is down (D-NEXT), so the prod page covers the database too. **That part is in code, pending a Cloud Run apply + an API deploy.**
 - [x] **No SSL policy and no Cloud Armor on the production LB** — both `gcloud compute ssl-policies list` and `security-policies list` are empty. TLS 1.2/1.3-only is currently true but unpinned and uncitable. `infra/modules/api-lb/main.tf` now declares `overslash-prod-api-ssl-policy` (TLS 1.2 minimum, `RESTRICTED`) on the HTTPS proxy and `overslash-prod-api-armor` on the backend service — default allow, OWASP CRS rule sets in preview (logged, never blocked) until tuned. **Not applied** — in code, pending a human run of [lb-hardening.md](docs/runbooks/lb-hardening.md), which also lists the evidence commands. (CASA 4.1.1)
 - [x] Set the org-policy baseline that would make the above guardrails rather than conventions. **Applied 2026-10-27** in both projects. The overspiral.com org now enforces a baseline at org level (measured table in [gcp-posture.md](docs/compliance/casa/gcp-posture.md#supply-chain-and-organization-policy)), and `infra/modules/project-guardrails/` ([project-guardrails.md](docs/runbooks/project-guardrails.md)) pins the Overslash-specific ones per project: `sql.managed.restrictPublicIp` + `restrictAuthorizedNetworks` (enforced in prod, explicitly off on dev's public-IP database), `storage.publicAccessPrevention` and `iam.disableAuditLoggingExemption` (both projects). `iam.managed.disableServiceAccountKeyCreation` is inherited from the org, and dev's hand-set opt-out for the Directory-sync key is adopted into Terraform. Org-wide policies belong to the org repo, not here.
 - [ ] **`run.allowedIngress`** — unset (allow all) on both projects, measured 2026-10-27. It applies project-wide, so it can only be set to `internal-and-cloud-load-balancing` once overfwd and the shortener stop running with `INGRESS_TRAFFIC_ALL` (prod API is already LB-only). (CASA 6.2.1)

crates/overslash-api/src/routes/health.rs

@@ -10,9 +10,8 @@ use crate::AppState;
 /// Upper bound on how long a probe may wait for Postgres.
 ///
 /// Without this, `PgPool::acquire` blocks for the pool's `acquire_timeout`
-/// (30 s by default) when the database is unreachable — long enough for the
-/// Cloud Run liveness probe to time out and recycle the container, which is
-/// exactly what the always-200 `/health` below exists to prevent.
+/// (30 s by default) when the database is unreachable, and the uptime check
+/// (10 s timeout) would read a hung database as a timeout rather than a 503.
 const PROBE_TIMEOUT: Duration = Duration::from_secs(2);
 
 // The `version` / `commit` / `sql_policy` fields in both bodies are
@@ -87,39 +86,27 @@ async fn probe_db(pool: &PgPool) -> DbProbe {
     }
 }
 
-/// Liveness. **Always 200**, even when Postgres is unreachable.
+/// Origin health: 200 when the process *and* Postgres are up, 503 otherwise.
 ///
-/// This endpoint backs the Cloud Run startup *and* liveness probes
-/// (`infra/modules/cloud-run/main.tf`) plus the Better Stack P0 monitor. If it
-/// failed on a Cloud SQL blip, Cloud Run would kill and restart every container
-/// mid-outage and the startup probe would block redeploys until the database
-/// recovered — the probe would amplify the incident instead of reporting it.
-///
-/// So DB state is reported in the body (`db`, plus `db_latency_ms` when up) and
-/// never in the status code. For a check that *fails* when the database is
-/// down, use [`ready`].
-async fn health(State(state): State<AppState>) -> Json<Value> {
-    let probe = probe_db(&state.db).await;
-    let info = build_info();
-    let mut body = json!({
-        "status": "ok",
-        "version": info.version,
-        "commit": info.commit,
-        "sql_policy": overslash_core::sql_policy::available(),
-    });
-    probe.extend(&mut body);
-    Json(body)
+/// This is what the GCM uptime check (`[P0] API Down`) and the Better Stack
+/// monitor probe, so a database outage pages. It used to be always-200 because
+/// the Cloud Run probes pointed here; they are TCP probes now
+/// (`infra/modules/cloud-run/main.tf`), so nothing restarts a container on a
+/// 503 from this endpoint.
+async fn health(State(state): State<AppState>) -> (StatusCode, Json<Value>) {
+    respond(probe_db(&state.db).await, "ok")
 }
 
-/// Readiness — 503 when Postgres is unreachable.
-///
-/// The gated counterpart to [`health`]: this is the one safe to point a
-/// load-balancer or an alerting monitor at, because failing it takes the
-/// instance out of rotation rather than restarting it.
+/// Readiness. Same gate as [`health`]; kept for the callers that use it.
 async fn ready(State(state): State<AppState>) -> (StatusCode, Json<Value>) {
-    let probe = probe_db(&state.db).await;
+    respond(probe_db(&state.db).await, "ready")
+}
+
+/// Shared body + status for both endpoints: `up_status` when the database
+/// answered, 503 `degraded` when it did not.
+fn respond(probe: DbProbe, up_status: &str) -> (StatusCode, Json<Value>) {
     let (code, status) = if probe.is_up() {
-        (StatusCode::OK, "ready")
+        (StatusCode::OK, up_status)
     } else {
         (StatusCode::SERVICE_UNAVAILABLE, "degraded")
     };
@@ -177,6 +164,18 @@ mod tests {
         );
     }
 
+    #[test]
+    fn respond_is_503_degraded_when_db_is_down() {
+        let (code, Json(body)) = respond(DbProbe::Down, "ok");
+        assert_eq!(code, StatusCode::SERVICE_UNAVAILABLE);
+        assert_eq!(body["status"], "degraded");
+        assert_eq!(body["db"], "down");
+
+        let (code, Json(body)) = respond(DbProbe::Up { latency_ms: 1 }, "ok");
+        assert_eq!(code, StatusCode::OK);
+        assert_eq!(body["status"], "ok");
+    }
+
     #[test]
     fn extend_reports_down_without_detail() {
         let mut body = json!({ "status": "degraded" });

crates/overslash-api/tests/health.rs

@@ -1,14 +1,11 @@
 //! Integration tests for `/health` and `/ready`.
 //!
-//! These pin the split that the endpoints exist to encode:
+//! Both return 200 when Postgres answers and 503 `degraded` when it does not;
+//! `/health` is what the uptime check and Better Stack probe. Cloud Run's own
+//! probes are TCP (`infra/modules/cloud-run/main.tf`), so neither endpoint can
+//! restart a container.
 //!
-//!   1. `/health` is liveness. It reports database state in the *body* and
-//!      always returns 200, because Cloud Run's startup and liveness probes
-//!      point at it — a 503 there would recycle containers mid-outage and
-//!      block redeploys (see `infra/modules/cloud-run/main.tf`).
-//!   2. `/ready` is readiness. It returns 503 when Postgres is unreachable.
-//!
-//! The unreachable-database branch is covered by a unit test in
+//! The unreachable-database branch is covered by unit tests in
 //! `src/routes/health.rs`, which can point a lazy pool at a dead port without
 //! disturbing the Postgres instance this suite shares across threads.