[angel-manuel]

Why

Prod has 0 uptime checks and no [P0] API Down (TODO §1.6). The only P0 is API High 5xx Rate, which can't fire when the API returns nothing. status.overslash.com (Better Stack) does catch an outage, but it's console-managed and doesn't page. status-page.md also claimed the GCM API-down alert "continues to page on-call", which was false.

Root cause

modules/monitoring gates both the uptime check and api_down on api_domain != "", and infra/main.tf passed var.domain. Prod sets domain = "" on purpose, because the GCLB serves api_host_suffix. Now: domain, or api_host_suffix when domain is empty and enable_api_lb is on. The LB host rule and cert already cover the apex, curl https://api.overslash.com/health → 200, and Cloud Armor has no rate rules that would hit the checkers.

The 3 disabled prod gates

All three descriptors exist in both projects (measured 2026-10-27 via the Monitoring REST API):
overslash_http_request_duration_seconds/histogram, overslash_oauth_events_total/counter, overslash_upstream_responses_total/counter. So api_latency, oauth_refresh and upstream_error are true in dev and prod.

Turning them on unchanged would have added alerts that can't fire. The OAuth and upstream ratios, and the already-deployed webhook ratio, divided by clamp_min(rate, 1). Prod traffic is under 1/s, so the denominator was always 1 and the "ratio" was failures per second. They now follow the latency alert: a 0.001 divide-by-zero floor plus an explicit minimum-sample and guard (~10 refreshes / ~20 upstream responses / ~10 terminal webhook deliveries per window). The two Actions & OAuth dashboard ratio charts had the same clamp and are fixed too.

Plan (read-only, from the agent sandbox)

A full plan isn't possible here: module.monitoring's depends_on pulls in IAM, which the factory SA can't read, and the providers' user_project_override needs serviceusage on the project. I ran -refresh=false -lock=false -target=module.monitoring with a temporary, uncommitted provider override:

• dev: 4 to add, 2 to change, 1 to destroy. Slow/OAuth/upstream policies created, webhook updated in place, actions_and_oauth dashboard replaced. Uptime and API Down unchanged, since dev already sets domain.
• prod: uptime check created, the same 3 policies, webhook update, overview + actions_and_oauth dashboards replaced (the overview's uptime tile now gets a real check ID). The plan then stopped on the PagerDuty secret read (the SA is correctly denied), so api_down didn't render. Its only input past that point is the P0 channel list.

tofu validate and tofu fmt -check pass.

Not applied

Infra is never auto-applied. Apply dev first, then prod, and verify per docs/runbooks/uptime-and-alerts.md (gcloud monitoring uptime list-configs, alert-policy listing via REST, plus an optional dev drill that points the check at a 404 path). Expected in prod after apply: 14 policies, 1 uptime check (10 today).

Docs: gcp-posture.md monitoring section re-measured (10 policies today, both channels present) and carries the remediation; TODO item checked as in code, pending apply; STATUS.md monitoring line corrected to what's actually deployed; status-page.md no longer claims a page that doesn't exist.

Applied (commit 1), verified 2026-10-27

Both envs. Prod: 14 policies, [P0] API Down → PagerDuty + email; overslash-prod-api-health check_passed true from all 6 regions.

Follow-up (commit 2): /health 503s on DB down, Cloud Run probes are TCP

Found while checking dev's nightly behaviour. Dev's uptime check failed 100% every night: the HTTP liveness probe on /health (1s default timeout, against a 2s DB check) shut the instance down ~90s after the Cloud SQL stop (LIVENESS HTTP probe failed 3 times consecutively … The instance has been shut down, 2026-10-26 22:01:38Z). No replacement could boot (eager DB connect), so Cloud Run returned 500 all night. The always-200 /health existed only to protect those probes, and it didn't.

• infra/modules/cloud-run/main.tf: startup + liveness → tcp_socket on the container port. The listener binds only after create_app has connected, migrated and preflighted, so startup readiness is unchanged.
• routes/health.rs: /health returns 503 degraded when the DB probe fails (same gate as /ready), so prod's [P0] API Down and Better Stack cover the database. No new endpoint.
• Dev's API Down firing nightly is accepted (email only).
• Decision D-NEXT, SPEC §health, status-page/uptime runbooks updated. Tests: routes::health unit tests (new 503 case) + tests/health.rs/version/security_headers pass.

Needs: a Cloud Run apply (module.cloud_run) + an API release, both envs.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
overslash	Ready	Preview	Oct 27, 2026 10:54pm UTC

[github-actions]

Code diff size

+18 / −31 across 1 file (net -13)

Area	Files	Added	Removed	Net
overslash-api	1	+18	−31	-13

_{Source files under src/ only (.cjs, .go, .js, .jsx, .mjs, .py, .rs, .svelte, .ts, .tsx); test files and inline #[cfg(test)] modules excluded. 14 other changed files not counted.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!