diff --git a/README.md b/README.md index c0e0acb..af3d958 100644 --- a/README.md +++ b/README.md @@ -446,3 +446,96 @@ For multi-user Syncthing with per-user server folder isolation and VPN-only acce - backend CORS allowlist/pattern does not include current frontend origin. - WebCrypto unavailable: - app opened via plain HTTP or insecure context. + +## 10) Independent Runtime Monitoring (Gatus + Heartbeats) + +Gatus runs as a **host-level systemd service**, not in `docker-compose.deploy.yml`. This keeps monitoring independent from the application stack: a Compose outage does not also take down the monitor. + +### Network and privacy model + +- Install Gatus on the deployment host (or another independent host) and keep its dashboard bound to `127.0.0.1:8082`. Access it with an SSH tunnel (`ssh -L 8082:127.0.0.1:8082 `) or an authenticated VPN-only proxy. +- The monitoring host must have the same VPN connectivity and Split-DNS access as a normal user for VPN-only endpoints. For example, `vault.example.com` should resolve to the VPN address on that host. +- Do not add public DNS records, firewall openings, or public port forwards for VPN-only services just to make a check pass. +- Configure every endpoint with the URLs users actually visit through the frontend/reverse proxy. Do not point checks at Docker-only service names or internal container ports. + +### Install Gatus + +From the repository root on the server: + +```bash +cd /opt/deploy +sudo GATUS_VERSION=v5.15.0 ./monitoring/gatus/install.sh +sudoedit /etc/gatus/gatus.env +``` + +Set the URLs in `/etc/gatus/gatus.env`. The example file is [monitoring/gatus/gatus.env.example](./monitoring/gatus/gatus.env.example). It includes the user-facing Vault Web and Cloud Page UI URLs, a Cloud Page backend probe, Vaultwarden, Vault Habits and the WebSocket route `wss://vault.example.com/ws-chat`, matching `docker/nginx/default.conf`. Replace every example host with the real route in your installation. + +The Cloud Page backend probe deliberately requests a nonexistent Secure Send token through `/cloud-api/public/secure-sends//meta`. A healthy backend returns the expected `404` response body `Secure Send link is unavailable`; this proves the request reached the Cloud Page application rather than merely receiving Angular's SPA shell. If the backend or its database is unavailable, the expected status/body check fails. + +To verify the backend monitor on the deployment host, stop only the Cloud Page backend and wait for the next Gatus interval: + +```bash +cd /opt/deploy +sudo docker compose -f docker-compose.deploy.yml stop cloud-page-backend +# The "Cloud Page backend" endpoint should become unhealthy. +sudo docker compose -f docker-compose.deploy.yml start cloud-page-backend +``` + +After the backend is healthy again, Gatus should report the endpoint as healthy on a subsequent check. + +Enable the service: + +```bash +sudo systemctl enable --now gatus +sudo systemctl status gatus --no-pager +sudo journalctl -u gatus -n 100 --no-pager +# From your workstation, open an SSH tunnel: +ssh -L 8082:127.0.0.1:8082 +# Then visit http://127.0.0.1:8082 in your browser. +``` + +Gatus restarts on failure and runs checks every minute. A failed condition marks the endpoint unhealthy. Review the Gatus status page/logs and investigate the affected reverse proxy, VPN/DNS route or application. Do not expose private services to the public internet to satisfy a check. + +### Optional external host and backup heartbeats + +Create separate monitors at [healthchecks.io](https://healthchecks.io/) for host uptime and the backup job. Keep generated UUIDs secret. The scripts in [monitoring/healthchecks](./monitoring/healthchecks) are no-ops when their UUID environment variable is absent, so this integration remains optional. + +Create `/etc/default/vault-web-heartbeats` with mode `0600` and set: + +```bash +HC_HOST_UUID=replace-with-host-check-uuid +HC_BACKUP_UUID=replace-with-backup-check-uuid +``` + +Install the host timer using the example units: + +```bash +sudo install -m 0755 /opt/deploy/monitoring/healthchecks/host-heartbeat.sh /usr/local/sbin/vault-web-host-heartbeat +sudo install -m 0644 /opt/deploy/monitoring/healthchecks/host-heartbeat.service.example /etc/systemd/system/vault-web-host-heartbeat.service +sudo install -m 0644 /opt/deploy/monitoring/healthchecks/host-heartbeat.timer.example /etc/systemd/system/vault-web-host-heartbeat.timer +sudo systemctl daemon-reload +sudo systemctl enable --now vault-web-host-heartbeat.timer +``` + +For backup monitoring, install the hook and attach it to the existing systemd backup service with an `ExecStartPost` drop-in. systemd runs `ExecStartPost` only after the backup command exits successfully, so failed backups never send a success heartbeat: + +```bash +sudo install -m 0755 /opt/deploy/monitoring/healthchecks/backup-heartbeat.sh /usr/local/sbin/vault-web-backup-heartbeat +sudo install -d -m 0755 /etc/systemd/system/backup-vault-users.service.d +sudo tee /etc/systemd/system/backup-vault-users.service.d/healthchecks.conf >/dev/null <<'EOF' +[Service] +EnvironmentFile=/etc/default/vault-web-heartbeats +ExecStartPost=/usr/local/sbin/vault-web-backup-heartbeat +EOF +sudo systemctl daemon-reload +``` + +The required `EnvironmentFile` passes `HC_BACKUP_UUID` from `/etc/default/vault-web-heartbeats` into the hook. The hook runs after the existing backup script, including its retention and verification steps. Do not call it at backup start: a later backup failure must not be reported as success. Test the configured path with a successful backup run, then confirm the backup monitor's last-ping time updates in healthchecks.io. Set its grace period longer than the normal backup duration; missed pings trigger its configured email/webhook alerts. + +### Troubleshooting + +- **All VPN endpoints fail:** check VPN connectivity and Split-DNS (`getent hosts vault.example.com`). +- **One endpoint fails:** open the same user-facing URL from a VPN client and inspect proxy/application logs; confirm the expected HTTP status. +- **WebSocket fails:** verify the route supports an HTTP Upgrade handshake and `/ws-chat` matches the frontend proxy. +- **Heartbeat is late:** inspect the systemd timer/service, outbound HTTPS connectivity and UUID configuration. + diff --git a/monitoring/gatus/config.yaml b/monitoring/gatus/config.yaml new file mode 100644 index 0000000..36de129 --- /dev/null +++ b/monitoring/gatus/config.yaml @@ -0,0 +1,61 @@ +# Gatus runs on the host, outside the application Compose project. +# Keep the dashboard local; access it through SSH tunnelling or an authenticated proxy. +web: + address: 127.0.0.1 + port: 8082 + +# Set variables in /etc/gatus/gatus.env to the same routes users visit. +endpoints: + - name: Vault Web + group: Vault Web + url: "${VAULT_WEB_URL}" + interval: 1m + conditions: + - "[STATUS] == 200" + - "[RESPONSE_TIME] < 5000" + + - name: Cloud Page + group: Vault Web + url: "${CLOUD_PAGE_URL}" + interval: 1m + conditions: + - "[STATUS] == 200" + - "[RESPONSE_TIME] < 5000" + + # A deliberately unknown public Secure Send token exercises the Cloud Page + # backend (and its database) through the same /cloud-api/ reverse-proxy route. + # The expected 404 and response text prove the request reached the application; + # a stopped backend returns a proxy error/connection failure instead. + - name: Cloud Page backend + group: Vault Web + url: "${CLOUD_PAGE_BACKEND_URL}" + interval: 1m + conditions: + - "[STATUS] == 404" + - "[BODY] == pat(*Secure Send link is unavailable*)" + - "[RESPONSE_TIME] < 5000" + + - name: Vaultwarden + group: Supporting services + url: "${VAULTWARDEN_URL}" + interval: 1m + conditions: + - "[STATUS] == 200" + - "[RESPONSE_TIME] < 5000" + + - name: Vault Habits + group: Supporting services + url: "${VAULT_HABITS_URL}" + interval: 1m + conditions: + - "[STATUS] == 200" + - "[RESPONSE_TIME] < 5000" + + # Gatus natively performs a WebSocket handshake for ws:// and wss:// URLs. + - name: Vault Web chat WebSocket + group: Realtime + url: "${VAULT_WEBSOCKET_URL}" + interval: 1m + conditions: + - "[CONNECTED] == true" + - "[RESPONSE_TIME] < 5000" diff --git a/monitoring/gatus/gatus.env.example b/monitoring/gatus/gatus.env.example new file mode 100644 index 0000000..101cfa7 --- /dev/null +++ b/monitoring/gatus/gatus.env.example @@ -0,0 +1,10 @@ +# Copy to /etc/gatus/gatus.env and replace example URLs. +# The host running Gatus must have the same VPN/DNS access as its users. +VAULT_WEB_URL=https://vault.example.com/ +CLOUD_PAGE_URL=https://vault.example.com/cloud/ +# This deliberately probes a nonexistent Secure Send token. The Cloud Page +# backend should return 404 with "Secure Send link is unavailable". +CLOUD_PAGE_BACKEND_URL=https://vault.example.com/cloud-api/public/secure-sends/gatus-health-probe/meta +VAULTWARDEN_URL=https://vaultwarden.example.internal/ +VAULT_HABITS_URL=https://habits.example.com/ +VAULT_WEBSOCKET_URL=wss://vault.example.com/ws-chat diff --git a/monitoring/gatus/gatus.service b/monitoring/gatus/gatus.service new file mode 100644 index 0000000..feb3e42 --- /dev/null +++ b/monitoring/gatus/gatus.service @@ -0,0 +1,28 @@ +[Unit] +Description=Gatus external service health monitoring +Documentation=https://github.com/TwiN/gatus +Wants=network-online.target +After=network-online.target +# When tailscaled provides the private route, optionally add: +# Wants=tailscaled.service +# After=tailscaled.service + +[Service] +Type=simple +User=gatus +Group=gatus +EnvironmentFile=/etc/gatus/gatus.env +Environment=GATUS_CONFIG_PATH=/etc/gatus/config.yaml +ExecStart=/usr/local/bin/gatus +Restart=on-failure +RestartSec=5s +NoNewPrivileges=true +PrivateTmp=true +ProtectSystem=strict +ProtectHome=true +ReadWritePaths=/var/lib/gatus +StateDirectory=gatus +UMask=0077 + +[Install] +WantedBy=multi-user.target diff --git a/monitoring/gatus/install.sh b/monitoring/gatus/install.sh new file mode 100644 index 0000000..9ee0296 --- /dev/null +++ b/monitoring/gatus/install.sh @@ -0,0 +1,61 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Install the official Gatus binary from its pinned Go module version. +GATUS_VERSION="${GATUS_VERSION:-v5.15.0}" +MIN_GO_VERSION="1.23.3" + +if [[ "$(id -u)" -ne 0 ]]; then + echo "Run this script with sudo/root." >&2 + exit 1 +fi + +if ! command -v go >/dev/null 2>&1; then + echo "Go >= ${MIN_GO_VERSION} is required to install Gatus." >&2 + exit 1 +fi + +GO_VERSION="$(go version | awk '{print $3}' | sed 's/^go//')" +if [[ "$(printf '%s\n' "$MIN_GO_VERSION" "$GO_VERSION" | sort -V | head -n1)" != "$MIN_GO_VERSION" ]]; then + echo "Go >= ${MIN_GO_VERSION} is required; found ${GO_VERSION}." >&2 + exit 1 +fi + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +TMP_DIR="$(mktemp -d)" +trap 'rm -rf "$TMP_DIR"' EXIT +umask 077 + +mkdir -p "$TMP_DIR/bin" + +CGO_ENABLED=0 GOBIN="$TMP_DIR/bin" \ + go install "github.com/TwiN/gatus/v5@${GATUS_VERSION}" + +if [[ ! -x "$TMP_DIR/bin/gatus" ]]; then + echo "Gatus binary installation failed." >&2 + exit 1 +fi + +install -o root -g root -m 0755 "$TMP_DIR/bin/gatus" /usr/local/bin/gatus + +if ! getent group gatus >/dev/null; then + groupadd --system gatus +fi +if ! id gatus >/dev/null 2>&1; then + useradd --system --gid gatus --home-dir /var/lib/gatus --no-create-home --shell /usr/sbin/nologin gatus +fi + +install -d -o root -g gatus -m 0750 /etc/gatus +install -d -o gatus -g gatus -m 0750 /var/lib/gatus +install -o root -g root -m 0644 "$SCRIPT_DIR/config.yaml" /etc/gatus/config.yaml +install -o root -g root -m 0644 "$SCRIPT_DIR/gatus.service" /etc/systemd/system/gatus.service + +if [[ ! -f /etc/gatus/gatus.env ]]; then + install -o root -g gatus -m 0640 "$SCRIPT_DIR/gatus.env.example" /etc/gatus/gatus.env + echo "Created /etc/gatus/gatus.env. Set your real user-facing URLs." +else + echo "Keeping existing /etc/gatus/gatus.env" +fi + +systemctl daemon-reload +echo "Gatus installed. Edit /etc/gatus/gatus.env, then run: systemctl enable --now gatus" diff --git a/monitoring/healthchecks/backup-heartbeat.sh b/monitoring/healthchecks/backup-heartbeat.sh new file mode 100644 index 0000000..ae920de --- /dev/null +++ b/monitoring/healthchecks/backup-heartbeat.sh @@ -0,0 +1,9 @@ +#!/usr/bin/env bash +set -euo pipefail +# Called by backup-vault-users.service only after the backup command succeeds. +if [[ -z "${HC_BACKUP_UUID:-}" ]]; then + echo "HC_BACKUP_UUID is unset; skipping optional backup heartbeat." >&2 + exit 0 +fi +curl --fail --silent --show-error --max-time 10 "https://hc-ping.com/$HC_BACKUP_UUID" >/dev/null +echo "Backup heartbeat sent to healthchecks.io." diff --git a/monitoring/healthchecks/host-heartbeat.service.example b/monitoring/healthchecks/host-heartbeat.service.example new file mode 100644 index 0000000..07c353e --- /dev/null +++ b/monitoring/healthchecks/host-heartbeat.service.example @@ -0,0 +1,7 @@ +[Unit] +Description=Optional healthchecks.io host heartbeat + +[Service] +Type=oneshot +EnvironmentFile=-/etc/default/vault-web-heartbeats +ExecStart=/usr/local/sbin/vault-web-host-heartbeat diff --git a/monitoring/healthchecks/host-heartbeat.sh b/monitoring/healthchecks/host-heartbeat.sh new file mode 100644 index 0000000..34b0b51 --- /dev/null +++ b/monitoring/healthchecks/host-heartbeat.sh @@ -0,0 +1,5 @@ +#!/usr/bin/env bash +set -euo pipefail +# Optional healthchecks.io heartbeat. Missing UUID is intentionally a no-op. +[[ -n "${HC_HOST_UUID:-}" ]] || exit 0 +curl --fail --silent --show-error --max-time 10 "https://hc-ping.com/$HC_HOST_UUID" >/dev/null diff --git a/monitoring/healthchecks/host-heartbeat.timer.example b/monitoring/healthchecks/host-heartbeat.timer.example new file mode 100644 index 0000000..ac5f6ed --- /dev/null +++ b/monitoring/healthchecks/host-heartbeat.timer.example @@ -0,0 +1,10 @@ +[Unit] +Description=Send host uptime heartbeat every five minutes + +[Timer] +OnBootSec=2min +OnUnitActiveSec=5min +Persistent=true + +[Install] +WantedBy=timers.target