feat: add independent Gatus monitoring and healthcheck integration

README.md

@@ -446,3 +446,96 @@ For multi-user Syncthing with per-user server folder isolation and VPN-only acce
   - backend CORS allowlist/pattern does not include current frontend origin.
 - WebCrypto unavailable:
   - app opened via plain HTTP or insecure context.
+
+## 10) Independent Runtime Monitoring (Gatus + Heartbeats)
+
+Gatus runs as a **host-level systemd service**, not in `docker-compose.deploy.yml`. This keeps monitoring independent from the application stack: a Compose outage does not also take down the monitor.
+
+### Network and privacy model
+
+- Install Gatus on the deployment host (or another independent host) and keep its dashboard bound to `127.0.0.1:8082`. Access it with an SSH tunnel (`ssh -L 8082:127.0.0.1:8082 <server>`) or an authenticated VPN-only proxy.
+- The monitoring host must have the same VPN connectivity and Split-DNS access as a normal user for VPN-only endpoints. For example, `vault.example.com` should resolve to the VPN address on that host.
+- Do not add public DNS records, firewall openings, or public port forwards for VPN-only services just to make a check pass.
+- Configure every endpoint with the URLs users actually visit through the frontend/reverse proxy. Do not point checks at Docker-only service names or internal container ports.
+
+### Install Gatus
+
+From the repository root on the server:
+
+```bash
+cd /opt/deploy
+sudo GATUS_VERSION=v5.15.0 ./monitoring/gatus/install.sh
+sudoedit /etc/gatus/gatus.env
+```
+
+Set the URLs in `/etc/gatus/gatus.env`. The example file is [monitoring/gatus/gatus.env.example](./monitoring/gatus/gatus.env.example). It includes the user-facing Vault Web and Cloud Page UI URLs, a Cloud Page backend probe, Vaultwarden, Vault Habits and the WebSocket route `wss://vault.example.com/ws-chat`, matching `docker/nginx/default.conf`. Replace every example host with the real route in your installation.
+
+The Cloud Page backend probe deliberately requests a nonexistent Secure Send token through `/cloud-api/public/secure-sends/<probe-token>/meta`. A healthy backend returns the expected `404` response body `Secure Send link is unavailable`; this proves the request reached the Cloud Page application rather than merely receiving Angular's SPA shell. If the backend or its database is unavailable, the expected status/body check fails.
+
+To verify the backend monitor on the deployment host, stop only the Cloud Page backend and wait for the next Gatus interval:
+
+```bash
+cd /opt/deploy
+sudo docker compose -f docker-compose.deploy.yml stop cloud-page-backend
+# The "Cloud Page backend" endpoint should become unhealthy.
+sudo docker compose -f docker-compose.deploy.yml start cloud-page-backend
+```
+
+After the backend is healthy again, Gatus should report the endpoint as healthy on a subsequent check.
+
+Enable the service:
+
+```bash
+sudo systemctl enable --now gatus
+sudo systemctl status gatus --no-pager
+sudo journalctl -u gatus -n 100 --no-pager
+# From your workstation, open an SSH tunnel:
+ssh -L 8082:127.0.0.1:8082 <server>
+# Then visit http://127.0.0.1:8082 in your browser.
+```
+
+Gatus restarts on failure and runs checks every minute. A failed condition marks the endpoint unhealthy. Review the Gatus status page/logs and investigate the affected reverse proxy, VPN/DNS route or application. Do not expose private services to the public internet to satisfy a check.
+
+### Optional external host and backup heartbeats
+
+Create separate monitors at [healthchecks.io](https://healthchecks.io/) for host uptime and the backup job. Keep generated UUIDs secret. The scripts in [monitoring/healthchecks](./monitoring/healthchecks) are no-ops when their UUID environment variable is absent, so this integration remains optional.
+
+Create `/etc/default/vault-web-heartbeats` with mode `0600` and set:
+
+```bash
+HC_HOST_UUID=replace-with-host-check-uuid
+HC_BACKUP_UUID=replace-with-backup-check-uuid
+```
+
+Install the host timer using the example units:
+
+```bash
+sudo install -m 0755 /opt/deploy/monitoring/healthchecks/host-heartbeat.sh /usr/local/sbin/vault-web-host-heartbeat
+sudo install -m 0644 /opt/deploy/monitoring/healthchecks/host-heartbeat.service.example /etc/systemd/system/vault-web-host-heartbeat.service
+sudo install -m 0644 /opt/deploy/monitoring/healthchecks/host-heartbeat.timer.example /etc/systemd/system/vault-web-host-heartbeat.timer
+sudo systemctl daemon-reload
+sudo systemctl enable --now vault-web-host-heartbeat.timer
+```
+
+For backup monitoring, install the hook and attach it to the existing systemd backup service with an `ExecStartPost` drop-in. systemd runs `ExecStartPost` only after the backup command exits successfully, so failed backups never send a success heartbeat:
+
+```bash
+sudo install -m 0755 /opt/deploy/monitoring/healthchecks/backup-heartbeat.sh /usr/local/sbin/vault-web-backup-heartbeat
+sudo install -d -m 0755 /etc/systemd/system/backup-vault-users.service.d
+sudo tee /etc/systemd/system/backup-vault-users.service.d/healthchecks.conf >/dev/null <<'EOF'
+[Service]
+EnvironmentFile=/etc/default/vault-web-heartbeats
+ExecStartPost=/usr/local/sbin/vault-web-backup-heartbeat
+EOF
+sudo systemctl daemon-reload
+```
+
+The required `EnvironmentFile` passes `HC_BACKUP_UUID` from `/etc/default/vault-web-heartbeats` into the hook. The hook runs after the existing backup script, including its retention and verification steps. Do not call it at backup start: a later backup failure must not be reported as success. Test the configured path with a successful backup run, then confirm the backup monitor's last-ping time updates in healthchecks.io. Set its grace period longer than the normal backup duration; missed pings trigger its configured email/webhook alerts.
+
+### Troubleshooting
+
+- **All VPN endpoints fail:** check VPN connectivity and Split-DNS (`getent hosts vault.example.com`).
+- **One endpoint fails:** open the same user-facing URL from a VPN client and inspect proxy/application logs; confirm the expected HTTP status.
+- **WebSocket fails:** verify the route supports an HTTP Upgrade handshake and `/ws-chat` matches the frontend proxy.
+- **Heartbeat is late:** inspect the systemd timer/service, outbound HTTPS connectivity and UUID configuration.
+

monitoring/gatus/config.yaml

@@ -0,0 +1,61 @@
+# Gatus runs on the host, outside the application Compose project.
+# Keep the dashboard local; access it through SSH tunnelling or an authenticated proxy.
+web:
+  address: 127.0.0.1
+  port: 8082
+
+# Set variables in /etc/gatus/gatus.env to the same routes users visit.
+endpoints:
+  - name: Vault Web
+    group: Vault Web
+    url: "${VAULT_WEB_URL}"
+    interval: 1m
+    conditions:
+      - "[STATUS] == 200"
+      - "[RESPONSE_TIME] < 5000"
+
+  - name: Cloud Page
+    group: Vault Web
+    url: "${CLOUD_PAGE_URL}"
+    interval: 1m
+    conditions:
+      - "[STATUS] == 200"
+      - "[RESPONSE_TIME] < 5000"
+
+  # A deliberately unknown public Secure Send token exercises the Cloud Page
+  # backend (and its database) through the same /cloud-api/ reverse-proxy route.
+  # The expected 404 and response text prove the request reached the application;
+  # a stopped backend returns a proxy error/connection failure instead.
+  - name: Cloud Page backend
+    group: Vault Web
+    url: "${CLOUD_PAGE_BACKEND_URL}"
+    interval: 1m
+    conditions:
+      - "[STATUS] == 404"
+      - "[BODY] == pat(*Secure Send link is unavailable*)"
+      - "[RESPONSE_TIME] < 5000"
+
+  - name: Vaultwarden
+    group: Supporting services
+    url: "${VAULTWARDEN_URL}"
+    interval: 1m
+    conditions:
+      - "[STATUS] == 200"
+      - "[RESPONSE_TIME] < 5000"
+
+  - name: Vault Habits
+    group: Supporting services
+    url: "${VAULT_HABITS_URL}"
+    interval: 1m
+    conditions:
+      - "[STATUS] == 200"
+      - "[RESPONSE_TIME] < 5000"
+
+  # Gatus natively performs a WebSocket handshake for ws:// and wss:// URLs.
+  - name: Vault Web chat WebSocket
+    group: Realtime
+    url: "${VAULT_WEBSOCKET_URL}"
+    interval: 1m
+    conditions:
+      - "[CONNECTED] == true"
+      - "[RESPONSE_TIME] < 5000"

monitoring/gatus/gatus.env.example

@@ -0,0 +1,10 @@
+# Copy to /etc/gatus/gatus.env and replace example URLs.
+# The host running Gatus must have the same VPN/DNS access as its users.
+VAULT_WEB_URL=https://vault.example.com/
+CLOUD_PAGE_URL=https://vault.example.com/cloud/
+# This deliberately probes a nonexistent Secure Send token. The Cloud Page
+# backend should return 404 with "Secure Send link is unavailable".
+CLOUD_PAGE_BACKEND_URL=https://vault.example.com/cloud-api/public/secure-sends/gatus-health-probe/meta
+VAULTWARDEN_URL=https://vaultwarden.example.internal/
+VAULT_HABITS_URL=https://habits.example.com/
+VAULT_WEBSOCKET_URL=wss://vault.example.com/ws-chat

monitoring/gatus/gatus.service

@@ -0,0 +1,28 @@
+[Unit]
+Description=Gatus external service health monitoring
+Documentation=https://github.com/TwiN/gatus
+Wants=network-online.target
+After=network-online.target
+# When tailscaled provides the private route, optionally add:
+# Wants=tailscaled.service
+# After=tailscaled.service
+
+[Service]
+Type=simple
+User=gatus
+Group=gatus
+EnvironmentFile=/etc/gatus/gatus.env
+Environment=GATUS_CONFIG_PATH=/etc/gatus/config.yaml
+ExecStart=/usr/local/bin/gatus
+Restart=on-failure
+RestartSec=5s
+NoNewPrivileges=true
+PrivateTmp=true
+ProtectSystem=strict
+ProtectHome=true
+ReadWritePaths=/var/lib/gatus
+StateDirectory=gatus
+UMask=0077
+
+[Install]
+WantedBy=multi-user.target

monitoring/gatus/install.sh

@@ -0,0 +1,61 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Install the official Gatus binary from its pinned Go module version.
+GATUS_VERSION="${GATUS_VERSION:-v5.15.0}"
+MIN_GO_VERSION="1.23.3"
+
+if [[ "$(id -u)" -ne 0 ]]; then
+  echo "Run this script with sudo/root." >&2
+  exit 1
+fi
+
+if ! command -v go >/dev/null 2>&1; then
+  echo "Go >= ${MIN_GO_VERSION} is required to install Gatus." >&2
+  exit 1
+fi
+
+GO_VERSION="$(go version | awk '{print $3}' | sed 's/^go//')"
+if [[ "$(printf '%s\n' "$MIN_GO_VERSION" "$GO_VERSION" | sort -V | head -n1)" != "$MIN_GO_VERSION" ]]; then
+  echo "Go >= ${MIN_GO_VERSION} is required; found ${GO_VERSION}." >&2
+  exit 1
+fi
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+TMP_DIR="$(mktemp -d)"
+trap 'rm -rf "$TMP_DIR"' EXIT
+umask 077
+
+mkdir -p "$TMP_DIR/bin"
+
+CGO_ENABLED=0 GOBIN="$TMP_DIR/bin" \
+  go install "github.com/TwiN/gatus/v5@${GATUS_VERSION}"
+
+if [[ ! -x "$TMP_DIR/bin/gatus" ]]; then
+  echo "Gatus binary installation failed." >&2
+  exit 1
+fi
+
+install -o root -g root -m 0755 "$TMP_DIR/bin/gatus" /usr/local/bin/gatus
+
+if ! getent group gatus >/dev/null; then
+  groupadd --system gatus
+fi
+if ! id gatus >/dev/null 2>&1; then
+  useradd --system --gid gatus --home-dir /var/lib/gatus --no-create-home --shell /usr/sbin/nologin gatus
+fi
+
+install -d -o root -g gatus -m 0750 /etc/gatus
+install -d -o gatus -g gatus -m 0750 /var/lib/gatus
+install -o root -g root -m 0644 "$SCRIPT_DIR/config.yaml" /etc/gatus/config.yaml
+install -o root -g root -m 0644 "$SCRIPT_DIR/gatus.service" /etc/systemd/system/gatus.service
+
+if [[ ! -f /etc/gatus/gatus.env ]]; then
+  install -o root -g gatus -m 0640 "$SCRIPT_DIR/gatus.env.example" /etc/gatus/gatus.env
+  echo "Created /etc/gatus/gatus.env. Set your real user-facing URLs."
+else
+  echo "Keeping existing /etc/gatus/gatus.env"
+fi
+
+systemctl daemon-reload
+echo "Gatus installed. Edit /etc/gatus/gatus.env, then run: systemctl enable --now gatus"

monitoring/healthchecks/backup-heartbeat.sh

@@ -0,0 +1,9 @@
+#!/usr/bin/env bash
+set -euo pipefail
+# Called by backup-vault-users.service only after the backup command succeeds.
+if [[ -z "${HC_BACKUP_UUID:-}" ]]; then
+  echo "HC_BACKUP_UUID is unset; skipping optional backup heartbeat." >&2
+  exit 0
+fi
+curl --fail --silent --show-error --max-time 10 "https://hc-ping.com/$HC_BACKUP_UUID" >/dev/null
+echo "Backup heartbeat sent to healthchecks.io."

monitoring/healthchecks/host-heartbeat.service.example

@@ -0,0 +1,7 @@
+[Unit]
+Description=Optional healthchecks.io host heartbeat
+
+[Service]
+Type=oneshot
+EnvironmentFile=-/etc/default/vault-web-heartbeats
+ExecStart=/usr/local/sbin/vault-web-host-heartbeat

monitoring/healthchecks/host-heartbeat.sh

@@ -0,0 +1,5 @@
+#!/usr/bin/env bash
+set -euo pipefail
+# Optional healthchecks.io heartbeat. Missing UUID is intentionally a no-op.
+[[ -n "${HC_HOST_UUID:-}" ]] || exit 0
+curl --fail --silent --show-error --max-time 10 "https://hc-ping.com/$HC_HOST_UUID" >/dev/null

monitoring/healthchecks/host-heartbeat.timer.example

@@ -0,0 +1,10 @@
+[Unit]
+Description=Send host uptime heartbeat every five minutes
+
+[Timer]
+OnBootSec=2min
+OnUnitActiveSec=5min
+Persistent=true
+
+[Install]
+WantedBy=timers.target