From e24bf7739ca41e0ef1a7c30cd75425bcf36ee158 Mon Sep 17 00:00:00 2001 From: Prashant Maurya Date: Tue, 3 Nov 2026 01:03:45 +0530 Subject: [PATCH 01/21] feat(monitoring): add independent Gatus and host heartbeats --- README.md | 64 +++++++++++++++++++ monitoring/gatus/config.yaml | 46 +++++++++++++ monitoring/gatus/gatus.env.example | 7 ++ monitoring/gatus/gatus.service | 28 ++++++++ monitoring/gatus/install.sh | 47 ++++++++++++++ monitoring/healthchecks/backup-heartbeat.sh | 5 ++ .../host-heartbeat.service.example | 7 ++ monitoring/healthchecks/host-heartbeat.sh | 5 ++ .../healthchecks/host-heartbeat.timer.example | 10 +++ 9 files changed, 219 insertions(+) create mode 100644 monitoring/gatus/config.yaml create mode 100644 monitoring/gatus/gatus.env.example create mode 100644 monitoring/gatus/gatus.service create mode 100644 monitoring/gatus/install.sh create mode 100644 monitoring/healthchecks/backup-heartbeat.sh create mode 100644 monitoring/healthchecks/host-heartbeat.service.example create mode 100644 monitoring/healthchecks/host-heartbeat.sh create mode 100644 monitoring/healthchecks/host-heartbeat.timer.example diff --git a/README.md b/README.md index c0e0acb..06192c8 100644 --- a/README.md +++ b/README.md @@ -446,3 +446,67 @@ For multi-user Syncthing with per-user server folder isolation and VPN-only acce - backend CORS allowlist/pattern does not include current frontend origin. - WebCrypto unavailable: - app opened via plain HTTP or insecure context. + +## 10) Independent Runtime Monitoring (Gatus + Heartbeats) + +Gatus runs as a **host-level systemd service**, not in `docker-compose.deploy.yml`. This keeps monitoring independent from the application stack: a Compose outage does not also take down the monitor. + +### Network and privacy model + +- Install Gatus on the deployment host (or another independent host) and keep its dashboard bound to loopback unless you deliberately protect it behind an authenticated VPN-only proxy. +- The monitoring host must have the same VPN connectivity and Split-DNS access as a normal user for VPN-only endpoints. For example, `vault.example.com` should resolve to the VPN address on that host. +- Do not add public DNS records, firewall openings, or public port forwards for VPN-only services just to make a check pass. +- Configure every endpoint with the URLs users actually visit through the frontend/reverse proxy. Do not point checks at Docker-only service names or internal container ports. + +### Install Gatus + +From the repository root on the server: + +```bash +cd /opt/deploy +sudo GATUS_VERSION=v5.15.0 ./monitoring/gatus/install.sh +sudoedit /etc/gatus/gatus.env +``` + +Set the URLs in `/etc/gatus/gatus.env`. The example file is [monitoring/gatus/gatus.env.example](./monitoring/gatus/gatus.env.example). It includes Vault Web, Cloud Page, Vaultwarden, Vault Habits and the WebSocket route `wss://vault.example.com/ws-chat`, matching `docker/nginx/default.conf`. Replace every example with the real user-facing route in your installation. + +Enable the service: + +```bash +sudo systemctl enable --now gatus +sudo systemctl status gatus --no-pager +sudo journalctl -u gatus -n 100 --no-pager +``` + +Gatus restarts on failure and runs checks every minute. A failed condition marks the endpoint unhealthy. Review the Gatus status page/logs and investigate the affected reverse proxy, VPN/DNS route or application. Do not expose private services to the public internet to satisfy a check. + +### Optional external host and backup heartbeats + +Create separate monitors at [healthchecks.io](https://healthchecks.io/) for host uptime and the backup job. Keep generated UUIDs secret. The scripts in [monitoring/healthchecks](./monitoring/healthchecks) are no-ops when their UUID environment variable is absent, so this integration remains optional. + +Create `/etc/default/vault-web-heartbeats` with mode `0600` and set: + +```bash +HC_HOST_UUID=replace-with-host-check-uuid +HC_BACKUP_UUID=replace-with-backup-check-uuid +``` + +Install the host timer using the example units: + +```bash +sudo install -m 0755 /opt/deploy/monitoring/healthchecks/host-heartbeat.sh /usr/local/sbin/vault-web-host-heartbeat +sudo install -m 0644 /opt/deploy/monitoring/healthchecks/host-heartbeat.service.example /etc/systemd/system/vault-web-host-heartbeat.service +sudo install -m 0644 /opt/deploy/monitoring/healthchecks/host-heartbeat.timer.example /etc/systemd/system/vault-web-host-heartbeat.timer +sudo systemctl daemon-reload +sudo systemctl enable --now vault-web-host-heartbeat.timer +``` + +For backup monitoring, invoke `backup-heartbeat.sh` only after the backup and verification succeed. Install it at a stable path such as `/usr/local/sbin/vault-web-backup-heartbeat` and add that command at the end of the successful backup path in `backup-vault-users.sh`. Never ping at backup start, because a later failure would be incorrectly reported as success. Set healthchecks.io grace periods to exceed the normal job duration; missed pings trigger the monitor's configured email/webhook alerts. + +### Troubleshooting + +- **All VPN endpoints fail:** check VPN connectivity and Split-DNS (`getent hosts vault.example.com`). +- **One endpoint fails:** open the same user-facing URL from a VPN client and inspect proxy/application logs; confirm the expected HTTP status. +- **WebSocket fails:** verify the route supports an HTTP Upgrade handshake and `/ws-chat` matches the frontend proxy. +- **Heartbeat is late:** inspect the systemd timer/service, outbound HTTPS connectivity and UUID configuration. + diff --git a/monitoring/gatus/config.yaml b/monitoring/gatus/config.yaml new file mode 100644 index 0000000..5b7d16a --- /dev/null +++ b/monitoring/gatus/config.yaml @@ -0,0 +1,46 @@ +# Gatus runs on the host, outside the application Compose project. +# Configure these URLs to match the same VPN/reverse-proxy routes used by users. +endpoints: + - name: Vault Web + group: Vault Web + url: "${VAULT_WEB_URL}" + interval: 1m + conditions: + - "[STATUS] == 200" + - "[RESPONSE_TIME] < 5000" + + - name: Cloud Page + group: Vault Web + url: "${CLOUD_PAGE_URL}" + interval: 1m + conditions: + - "[STATUS] == 200" + - "[RESPONSE_TIME] < 5000" + + - name: Vaultwarden + group: Supporting services + url: "${VAULTWARDEN_URL}" + interval: 1m + conditions: + - "[STATUS] == 200" + - "[RESPONSE_TIME] < 5000" + + - name: Vault Habits + group: Supporting services + url: "${VAULT_HABITS_URL}" + interval: 1m + conditions: + - "[STATUS] == 200" + - "[RESPONSE_TIME] < 5000" + + # Gatus checks the WebSocket handshake on the same VPN-facing route as clients. + - name: Vault Web chat WebSocket + group: Realtime + url: "${VAULT_WEBSOCKET_URL}" + interval: 1m + conditions: + - "[CONNECTED] == true" + - "[RESPONSE_TIME] < 5000" + client: + websocket: + heartbeat_interval: 30s diff --git a/monitoring/gatus/gatus.env.example b/monitoring/gatus/gatus.env.example new file mode 100644 index 0000000..e669a58 --- /dev/null +++ b/monitoring/gatus/gatus.env.example @@ -0,0 +1,7 @@ +# Copy to /etc/gatus/gatus.env and replace example URLs. +# The host running Gatus must have the same VPN/DNS access as its users. +VAULT_WEB_URL=https://vault.example.com/ +CLOUD_PAGE_URL=https://vault.example.com/cloud/ +VAULTWARDEN_URL=https://vaultwarden.example.internal/ +VAULT_HABITS_URL=https://habits.example.com/ +VAULT_WEBSOCKET_URL=wss://vault.example.com/ws-chat diff --git a/monitoring/gatus/gatus.service b/monitoring/gatus/gatus.service new file mode 100644 index 0000000..feb3e42 --- /dev/null +++ b/monitoring/gatus/gatus.service @@ -0,0 +1,28 @@ +[Unit] +Description=Gatus external service health monitoring +Documentation=https://github.com/TwiN/gatus +Wants=network-online.target +After=network-online.target +# When tailscaled provides the private route, optionally add: +# Wants=tailscaled.service +# After=tailscaled.service + +[Service] +Type=simple +User=gatus +Group=gatus +EnvironmentFile=/etc/gatus/gatus.env +Environment=GATUS_CONFIG_PATH=/etc/gatus/config.yaml +ExecStart=/usr/local/bin/gatus +Restart=on-failure +RestartSec=5s +NoNewPrivileges=true +PrivateTmp=true +ProtectSystem=strict +ProtectHome=true +ReadWritePaths=/var/lib/gatus +StateDirectory=gatus +UMask=0077 + +[Install] +WantedBy=multi-user.target diff --git a/monitoring/gatus/install.sh b/monitoring/gatus/install.sh new file mode 100644 index 0000000..2179d1a --- /dev/null +++ b/monitoring/gatus/install.sh @@ -0,0 +1,47 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Installs the official Gatus release binary. Pin GATUS_VERSION explicitly. +GATUS_VERSION="${GATUS_VERSION:-v5.15.0}" +ARCH="$(uname -m)" +case "$ARCH" in + x86_64) ASSET_ARCH="amd64" ;; + aarch64|arm64) ASSET_ARCH="arm64" ;; + *) echo "Unsupported architecture: $ARCH" >&2; exit 1 ;; +esac + +if [[ "$(id -u)" -ne 0 ]]; then + echo "Run this script with sudo/root." >&2 + exit 1 +fi + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +TMP_DIR="$(mktemp -d)" +trap 'rm -rf "$TMP_DIR"' EXIT +ASSET="gatus-linux-$ASSET_ARCH" +URL="https://github.com/TwiN/gatus/releases/download/$GATUS_VERSION/$ASSET" + +curl --fail --location --silent --show-error "$URL" -o "$TMP_DIR/gatus" +install -o root -g root -m 0755 "$TMP_DIR/gatus" /usr/local/bin/gatus + +if ! getent group gatus >/dev/null; then + groupadd --system gatus +fi +if ! id gatus >/dev/null 2>&1; then + useradd --system --gid gatus --home-dir /var/lib/gatus --no-create-home --shell /usr/sbin/nologin gatus +fi + +install -d -o root -g gatus -m 0750 /etc/gatus +install -d -o gatus -g gatus -m 0750 /var/lib/gatus +install -o root -g root -m 0644 "$SCRIPT_DIR/config.yaml" /etc/gatus/config.yaml +install -o root -g root -m 0644 "$SCRIPT_DIR/gatus.service" /etc/systemd/system/gatus.service + +if [[ ! -f /etc/gatus/gatus.env ]]; then + install -o root -g gatus -m 0640 "$SCRIPT_DIR/gatus.env.example" /etc/gatus/gatus.env + echo "Created /etc/gatus/gatus.env. Set your real user-facing URLs." +else + echo "Keeping existing /etc/gatus/gatus.env" +fi + +systemctl daemon-reload +echo "Gatus installed. Edit /etc/gatus/gatus.env, then run: systemctl enable --now gatus" diff --git a/monitoring/healthchecks/backup-heartbeat.sh b/monitoring/healthchecks/backup-heartbeat.sh new file mode 100644 index 0000000..7b19585 --- /dev/null +++ b/monitoring/healthchecks/backup-heartbeat.sh @@ -0,0 +1,5 @@ +#!/usr/bin/env bash +set -euo pipefail +# Call this only after the backup and its verification have succeeded. +[[ -n "${HC_BACKUP_UUID:-}" ]] || exit 0 +curl --fail --silent --show-error --max-time 10 "https://hc-ping.com/$HC_BACKUP_UUID" >/dev/null diff --git a/monitoring/healthchecks/host-heartbeat.service.example b/monitoring/healthchecks/host-heartbeat.service.example new file mode 100644 index 0000000..07c353e --- /dev/null +++ b/monitoring/healthchecks/host-heartbeat.service.example @@ -0,0 +1,7 @@ +[Unit] +Description=Optional healthchecks.io host heartbeat + +[Service] +Type=oneshot +EnvironmentFile=-/etc/default/vault-web-heartbeats +ExecStart=/usr/local/sbin/vault-web-host-heartbeat diff --git a/monitoring/healthchecks/host-heartbeat.sh b/monitoring/healthchecks/host-heartbeat.sh new file mode 100644 index 0000000..34b0b51 --- /dev/null +++ b/monitoring/healthchecks/host-heartbeat.sh @@ -0,0 +1,5 @@ +#!/usr/bin/env bash +set -euo pipefail +# Optional healthchecks.io heartbeat. Missing UUID is intentionally a no-op. +[[ -n "${HC_HOST_UUID:-}" ]] || exit 0 +curl --fail --silent --show-error --max-time 10 "https://hc-ping.com/$HC_HOST_UUID" >/dev/null diff --git a/monitoring/healthchecks/host-heartbeat.timer.example b/monitoring/healthchecks/host-heartbeat.timer.example new file mode 100644 index 0000000..ac5f6ed --- /dev/null +++ b/monitoring/healthchecks/host-heartbeat.timer.example @@ -0,0 +1,10 @@ +[Unit] +Description=Send host uptime heartbeat every five minutes + +[Timer] +OnBootSec=2min +OnUnitActiveSec=5min +Persistent=true + +[Install] +WantedBy=timers.target From 99396ee843b6c6d52d65422cd41a32a8ca6e80ff Mon Sep 17 00:00:00 2001 From: Prashant Maurya Date: Tue, 3 Nov 2026 01:04:05 +0530 Subject: [PATCH 02/21] fix(monitoring): align WebSocket checks with Gatus config --- monitoring/gatus/config.yaml | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/monitoring/gatus/config.yaml b/monitoring/gatus/config.yaml index 5b7d16a..4af2597 100644 --- a/monitoring/gatus/config.yaml +++ b/monitoring/gatus/config.yaml @@ -33,7 +33,7 @@ endpoints: - "[STATUS] == 200" - "[RESPONSE_TIME] < 5000" - # Gatus checks the WebSocket handshake on the same VPN-facing route as clients. + # Gatus supports ws:// and wss:// endpoint schemes directly. - name: Vault Web chat WebSocket group: Realtime url: "${VAULT_WEBSOCKET_URL}" @@ -41,6 +41,3 @@ endpoints: conditions: - "[CONNECTED] == true" - "[RESPONSE_TIME] < 5000" - client: - websocket: - heartbeat_interval: 30s From ec2b2d63cc3e3b6942ac36a6d184cf11b5ab9558 Mon Sep 17 00:00:00 2001 From: Prashant Maurya Date: Tue, 3 Nov 2026 01:04:08 +0530 Subject: [PATCH 03/21] fix(monitoring): align WebSocket checks with Gatus config From 52aa1e2a8209b0b1cbf5485514a2e9aef403298c Mon Sep 17 00:00:00 2001 From: Prashant Maurya Date: Tue, 3 Nov 2026 01:04:11 +0530 Subject: [PATCH 04/21] fix(monitoring): align WebSocket checks with Gatus config From 75164ead13493ee54041bfb70a3e7d0c3b29277d Mon Sep 17 00:00:00 2001 From: Prashant Maurya Date: Tue, 3 Nov 2026 01:04:14 +0530 Subject: [PATCH 05/21] fix(monitoring): align WebSocket checks with Gatus config From ba937ec28faab13ad2268997cae87273b9bb41bd Mon Sep 17 00:00:00 2001 From: Prashant Maurya Date: Tue, 3 Nov 2026 01:04:24 +0530 Subject: [PATCH 06/21] fix(monitoring): align websocket checks with Gatus From 059df46f14bb3b28cc721270f749f8af27d17232 Mon Sep 17 00:00:00 2001 From: Prashant Maurya Date: Tue, 3 Nov 2026 01:04:28 +0530 Subject: [PATCH 07/21] fix(monitoring): align websocket checks with Gatus From 92ecb7e3639570f6a4f8dbe323f91e29ab412f76 Mon Sep 17 00:00:00 2001 From: Prashant Maurya Date: Tue, 3 Nov 2026 01:04:31 +0530 Subject: [PATCH 08/21] fix(monitoring): align websocket checks with Gatus From 8d26ece6282c9b231d714b14728c513bfed0e3dd Mon Sep 17 00:00:00 2001 From: Prashant Maurya Date: Tue, 3 Nov 2026 01:04:34 +0530 Subject: [PATCH 09/21] fix(monitoring): align websocket checks with Gatus From 538c8e6bc93474a05b39388aaea3d6d19e762560 Mon Sep 17 00:00:00 2001 From: Prashant Maurya Date: Tue, 3 Nov 2026 01:04:41 +0530 Subject: [PATCH 10/21] docs(monitoring): clarify backup heartbeat installation --- README.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 06192c8..0aa2a69 100644 --- a/README.md +++ b/README.md @@ -501,7 +501,13 @@ sudo systemctl daemon-reload sudo systemctl enable --now vault-web-host-heartbeat.timer ``` -For backup monitoring, invoke `backup-heartbeat.sh` only after the backup and verification succeed. Install it at a stable path such as `/usr/local/sbin/vault-web-backup-heartbeat` and add that command at the end of the successful backup path in `backup-vault-users.sh`. Never ping at backup start, because a later failure would be incorrectly reported as success. Set healthchecks.io grace periods to exceed the normal job duration; missed pings trigger the monitor's configured email/webhook alerts. +For backup monitoring, install the hook at a stable path and run it only after the backup and verification succeed: + +```bash +sudo install -m 0755 /opt/deploy/monitoring/healthchecks/backup-heartbeat.sh /usr/local/sbin/vault-web-backup-heartbeat +``` + +Then add `/usr/local/sbin/vault-web-backup-heartbeat` as the final command in the successful path of `/usr/local/sbin/backup-vault-users.sh`. Keep it after rsync, retention cleanup and any verification steps. The heartbeat hook is not automatically wired into the pre-existing backup script, so configure this host-side step explicitly. Never ping at backup start, because a later failure would be incorrectly reported as success. Set healthchecks.io grace periods to exceed the normal job duration; missed pings trigger its configured email/webhook alerts. ### Troubleshooting From 9bb46bf9108e11a6477c6bccb6c77d46ee2c1221 Mon Sep 17 00:00:00 2001 From: Prashant Maurya Date: Tue, 3 Nov 2026 01:05:02 +0530 Subject: [PATCH 11/21] fix(monitoring): load host heartbeat credentials From 0b6c4aa739e70e1a4358e862f5bac7d2d1ef0b24 Mon Sep 17 00:00:00 2001 From: Prashant Maurya Date: Tue, 3 Nov 2026 01:05:22 +0530 Subject: [PATCH 12/21] fix(monitoring): extract Gatus release archive --- monitoring/gatus/install.sh | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/monitoring/gatus/install.sh b/monitoring/gatus/install.sh index 2179d1a..3cc8d5f 100644 --- a/monitoring/gatus/install.sh +++ b/monitoring/gatus/install.sh @@ -18,10 +18,16 @@ fi SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" TMP_DIR="$(mktemp -d)" trap 'rm -rf "$TMP_DIR"' EXIT +# Release assets are distributed as a tar.gz archive, not a bare executable. ASSET="gatus-linux-$ASSET_ARCH" -URL="https://github.com/TwiN/gatus/releases/download/$GATUS_VERSION/$ASSET" +URL="https://github.com/TwiN/gatus/releases/download/$GATUS_VERSION/$ASSET.tar.gz" -curl --fail --location --silent --show-error "$URL" -o "$TMP_DIR/gatus" +curl --fail --location --silent --show-error "$URL" -o "$TMP_DIR/gatus.tar.gz" +tar -xzf "$TMP_DIR/gatus.tar.gz" -C "$TMP_DIR" +if [[ ! -f "$TMP_DIR/gatus" ]]; then + echo "The release archive did not contain the expected gatus binary." >&2 + exit 1 +fi install -o root -g root -m 0755 "$TMP_DIR/gatus" /usr/local/bin/gatus if ! getent group gatus >/dev/null; then From 36b4aef20d9b32193bad7c46601670e33c6139f3 Mon Sep 17 00:00:00 2001 From: Prashant Maurya Date: Tue, 3 Nov 2026 01:05:29 +0530 Subject: [PATCH 13/21] fix(monitoring): restrict installer temporary file permissions --- monitoring/gatus/install.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/monitoring/gatus/install.sh b/monitoring/gatus/install.sh index 3cc8d5f..38bfc44 100644 --- a/monitoring/gatus/install.sh +++ b/monitoring/gatus/install.sh @@ -18,6 +18,7 @@ fi SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" TMP_DIR="$(mktemp -d)" trap 'rm -rf "$TMP_DIR"' EXIT +umask 077 # Release assets are distributed as a tar.gz archive, not a bare executable. ASSET="gatus-linux-$ASSET_ARCH" URL="https://github.com/TwiN/gatus/releases/download/$GATUS_VERSION/$ASSET.tar.gz" From a1f5ed12de20de62cac40c4ddfb5b77ac167e3d6 Mon Sep 17 00:00:00 2001 From: Prashant Maurya Date: Tue, 3 Nov 2026 01:05:41 +0530 Subject: [PATCH 14/21] fix(monitoring): normalize Gatus configuration --- monitoring/gatus/config.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/monitoring/gatus/config.yaml b/monitoring/gatus/config.yaml index 4af2597..e6eebca 100644 --- a/monitoring/gatus/config.yaml +++ b/monitoring/gatus/config.yaml @@ -1,5 +1,5 @@ # Gatus runs on the host, outside the application Compose project. -# Configure these URLs to match the same VPN/reverse-proxy routes used by users. +# Set variables in /etc/gatus/gatus.env to the same routes users visit. endpoints: - name: Vault Web group: Vault Web @@ -33,7 +33,7 @@ endpoints: - "[STATUS] == 200" - "[RESPONSE_TIME] < 5000" - # Gatus supports ws:// and wss:// endpoint schemes directly. + # Gatus natively performs a WebSocket handshake for ws:// and wss:// URLs. - name: Vault Web chat WebSocket group: Realtime url: "${VAULT_WEBSOCKET_URL}" From fed1d5df16617cae441849dddea4b8de20c0f0d1 Mon Sep 17 00:00:00 2001 From: Prashant Maurya Date: Tue, 3 Nov 2026 01:56:52 +0530 Subject: [PATCH 15/21] fix(monitoring): install Gatus from Go module --- monitoring/gatus/install.sh | 39 ++++++++++++++++++++++--------------- 1 file changed, 23 insertions(+), 16 deletions(-) diff --git a/monitoring/gatus/install.sh b/monitoring/gatus/install.sh index 38bfc44..9ee0296 100644 --- a/monitoring/gatus/install.sh +++ b/monitoring/gatus/install.sh @@ -1,35 +1,42 @@ #!/usr/bin/env bash set -euo pipefail -# Installs the official Gatus release binary. Pin GATUS_VERSION explicitly. +# Install the official Gatus binary from its pinned Go module version. GATUS_VERSION="${GATUS_VERSION:-v5.15.0}" -ARCH="$(uname -m)" -case "$ARCH" in - x86_64) ASSET_ARCH="amd64" ;; - aarch64|arm64) ASSET_ARCH="arm64" ;; - *) echo "Unsupported architecture: $ARCH" >&2; exit 1 ;; -esac +MIN_GO_VERSION="1.23.3" if [[ "$(id -u)" -ne 0 ]]; then echo "Run this script with sudo/root." >&2 exit 1 fi +if ! command -v go >/dev/null 2>&1; then + echo "Go >= ${MIN_GO_VERSION} is required to install Gatus." >&2 + exit 1 +fi + +GO_VERSION="$(go version | awk '{print $3}' | sed 's/^go//')" +if [[ "$(printf '%s\n' "$MIN_GO_VERSION" "$GO_VERSION" | sort -V | head -n1)" != "$MIN_GO_VERSION" ]]; then + echo "Go >= ${MIN_GO_VERSION} is required; found ${GO_VERSION}." >&2 + exit 1 +fi + SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" TMP_DIR="$(mktemp -d)" trap 'rm -rf "$TMP_DIR"' EXIT umask 077 -# Release assets are distributed as a tar.gz archive, not a bare executable. -ASSET="gatus-linux-$ASSET_ARCH" -URL="https://github.com/TwiN/gatus/releases/download/$GATUS_VERSION/$ASSET.tar.gz" - -curl --fail --location --silent --show-error "$URL" -o "$TMP_DIR/gatus.tar.gz" -tar -xzf "$TMP_DIR/gatus.tar.gz" -C "$TMP_DIR" -if [[ ! -f "$TMP_DIR/gatus" ]]; then - echo "The release archive did not contain the expected gatus binary." >&2 + +mkdir -p "$TMP_DIR/bin" + +CGO_ENABLED=0 GOBIN="$TMP_DIR/bin" \ + go install "github.com/TwiN/gatus/v5@${GATUS_VERSION}" + +if [[ ! -x "$TMP_DIR/bin/gatus" ]]; then + echo "Gatus binary installation failed." >&2 exit 1 fi -install -o root -g root -m 0755 "$TMP_DIR/gatus" /usr/local/bin/gatus + +install -o root -g root -m 0755 "$TMP_DIR/bin/gatus" /usr/local/bin/gatus if ! getent group gatus >/dev/null; then groupadd --system gatus From 9f531b724c67dec536a251ced25e744c899dd958 Mon Sep 17 00:00:00 2001 From: Prashant Maurya Date: Tue, 3 Nov 2026 02:06:02 +0530 Subject: [PATCH 16/21] fix(monitoring): bind Gatus dashboard to loopback --- monitoring/gatus/config.yaml | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/monitoring/gatus/config.yaml b/monitoring/gatus/config.yaml index e6eebca..a2de53e 100644 --- a/monitoring/gatus/config.yaml +++ b/monitoring/gatus/config.yaml @@ -1,4 +1,9 @@ # Gatus runs on the host, outside the application Compose project. +# Keep the dashboard local; access it through SSH tunnelling or an authenticated proxy. +web: + address: 127.0.0.1 + port: 8082 + # Set variables in /etc/gatus/gatus.env to the same routes users visit. endpoints: - name: Vault Web From 1488e6997d6bccd85c7a87e59cf573fbf4825bd7 Mon Sep 17 00:00:00 2001 From: Prashant Maurya Date: Tue, 3 Nov 2026 02:06:05 +0530 Subject: [PATCH 17/21] docs(monitoring): document local Gatus dashboard access --- README.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 0aa2a69..0fc8c6f 100644 --- a/README.md +++ b/README.md @@ -453,7 +453,7 @@ Gatus runs as a **host-level systemd service**, not in `docker-compose.deploy.ym ### Network and privacy model -- Install Gatus on the deployment host (or another independent host) and keep its dashboard bound to loopback unless you deliberately protect it behind an authenticated VPN-only proxy. +- Install Gatus on the deployment host (or another independent host) and keep its dashboard bound to `127.0.0.1:8082`. Access it with an SSH tunnel (`ssh -L 8082:127.0.0.1:8082 `) or an authenticated VPN-only proxy. - The monitoring host must have the same VPN connectivity and Split-DNS access as a normal user for VPN-only endpoints. For example, `vault.example.com` should resolve to the VPN address on that host. - Do not add public DNS records, firewall openings, or public port forwards for VPN-only services just to make a check pass. - Configure every endpoint with the URLs users actually visit through the frontend/reverse proxy. Do not point checks at Docker-only service names or internal container ports. @@ -476,6 +476,9 @@ Enable the service: sudo systemctl enable --now gatus sudo systemctl status gatus --no-pager sudo journalctl -u gatus -n 100 --no-pager +# From your workstation, open an SSH tunnel: +ssh -L 8082:127.0.0.1:8082 +# Then visit http://127.0.0.1:8082 in your browser. ``` Gatus restarts on failure and runs checks every minute. A failed condition marks the endpoint unhealthy. Review the Gatus status page/logs and investigate the affected reverse proxy, VPN/DNS route or application. Do not expose private services to the public internet to satisfy a check. From d6ade4d34cd0f0f3fda4e4dce83856d39c096d79 Mon Sep 17 00:00:00 2001 From: Prashant Maurya Date: Wed, 4 Nov 2026 00:33:45 +0530 Subject: [PATCH 18/21] fix(monitoring): add Cloud Page backend probe URL --- monitoring/gatus/gatus.env.example | 3 +++ 1 file changed, 3 insertions(+) diff --git a/monitoring/gatus/gatus.env.example b/monitoring/gatus/gatus.env.example index e669a58..101cfa7 100644 --- a/monitoring/gatus/gatus.env.example +++ b/monitoring/gatus/gatus.env.example @@ -2,6 +2,9 @@ # The host running Gatus must have the same VPN/DNS access as its users. VAULT_WEB_URL=https://vault.example.com/ CLOUD_PAGE_URL=https://vault.example.com/cloud/ +# This deliberately probes a nonexistent Secure Send token. The Cloud Page +# backend should return 404 with "Secure Send link is unavailable". +CLOUD_PAGE_BACKEND_URL=https://vault.example.com/cloud-api/public/secure-sends/gatus-health-probe/meta VAULTWARDEN_URL=https://vaultwarden.example.internal/ VAULT_HABITS_URL=https://habits.example.com/ VAULT_WEBSOCKET_URL=wss://vault.example.com/ws-chat From c3aee1663f4de6f592f655b24b098a5ad505dbdb Mon Sep 17 00:00:00 2001 From: Prashant Maurya Date: Wed, 4 Nov 2026 00:33:47 +0530 Subject: [PATCH 19/21] fix(monitoring): monitor Cloud Page backend health --- monitoring/gatus/config.yaml | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/monitoring/gatus/config.yaml b/monitoring/gatus/config.yaml index a2de53e..36de129 100644 --- a/monitoring/gatus/config.yaml +++ b/monitoring/gatus/config.yaml @@ -22,6 +22,19 @@ endpoints: - "[STATUS] == 200" - "[RESPONSE_TIME] < 5000" + # A deliberately unknown public Secure Send token exercises the Cloud Page + # backend (and its database) through the same /cloud-api/ reverse-proxy route. + # The expected 404 and response text prove the request reached the application; + # a stopped backend returns a proxy error/connection failure instead. + - name: Cloud Page backend + group: Vault Web + url: "${CLOUD_PAGE_BACKEND_URL}" + interval: 1m + conditions: + - "[STATUS] == 404" + - "[BODY] == pat(*Secure Send link is unavailable*)" + - "[RESPONSE_TIME] < 5000" + - name: Vaultwarden group: Supporting services url: "${VAULTWARDEN_URL}" From dd2e6da129817aa289e244070f7f6f0b41a63abf Mon Sep 17 00:00:00 2001 From: Prashant Maurya Date: Wed, 4 Nov 2026 00:33:49 +0530 Subject: [PATCH 20/21] fix(monitoring): report configured backup heartbeat --- monitoring/healthchecks/backup-heartbeat.sh | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/monitoring/healthchecks/backup-heartbeat.sh b/monitoring/healthchecks/backup-heartbeat.sh index 7b19585..ae920de 100644 --- a/monitoring/healthchecks/backup-heartbeat.sh +++ b/monitoring/healthchecks/backup-heartbeat.sh @@ -1,5 +1,9 @@ #!/usr/bin/env bash set -euo pipefail -# Call this only after the backup and its verification have succeeded. -[[ -n "${HC_BACKUP_UUID:-}" ]] || exit 0 +# Called by backup-vault-users.service only after the backup command succeeds. +if [[ -z "${HC_BACKUP_UUID:-}" ]]; then + echo "HC_BACKUP_UUID is unset; skipping optional backup heartbeat." >&2 + exit 0 +fi curl --fail --silent --show-error --max-time 10 "https://hc-ping.com/$HC_BACKUP_UUID" >/dev/null +echo "Backup heartbeat sent to healthchecks.io." From 6914197fabe7e50dbe52f62d269b667afc62fb47 Mon Sep 17 00:00:00 2001 From: Prashant Maurya Date: Wed, 4 Nov 2026 00:33:53 +0530 Subject: [PATCH 21/21] docs(monitoring): wire backup heartbeat after successful backup --- README.md | 26 +++++++++++++++++++++++--- 1 file changed, 23 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 0fc8c6f..af3d958 100644 --- a/README.md +++ b/README.md @@ -468,7 +468,20 @@ sudo GATUS_VERSION=v5.15.0 ./monitoring/gatus/install.sh sudoedit /etc/gatus/gatus.env ``` -Set the URLs in `/etc/gatus/gatus.env`. The example file is [monitoring/gatus/gatus.env.example](./monitoring/gatus/gatus.env.example). It includes Vault Web, Cloud Page, Vaultwarden, Vault Habits and the WebSocket route `wss://vault.example.com/ws-chat`, matching `docker/nginx/default.conf`. Replace every example with the real user-facing route in your installation. +Set the URLs in `/etc/gatus/gatus.env`. The example file is [monitoring/gatus/gatus.env.example](./monitoring/gatus/gatus.env.example). It includes the user-facing Vault Web and Cloud Page UI URLs, a Cloud Page backend probe, Vaultwarden, Vault Habits and the WebSocket route `wss://vault.example.com/ws-chat`, matching `docker/nginx/default.conf`. Replace every example host with the real route in your installation. + +The Cloud Page backend probe deliberately requests a nonexistent Secure Send token through `/cloud-api/public/secure-sends//meta`. A healthy backend returns the expected `404` response body `Secure Send link is unavailable`; this proves the request reached the Cloud Page application rather than merely receiving Angular's SPA shell. If the backend or its database is unavailable, the expected status/body check fails. + +To verify the backend monitor on the deployment host, stop only the Cloud Page backend and wait for the next Gatus interval: + +```bash +cd /opt/deploy +sudo docker compose -f docker-compose.deploy.yml stop cloud-page-backend +# The "Cloud Page backend" endpoint should become unhealthy. +sudo docker compose -f docker-compose.deploy.yml start cloud-page-backend +``` + +After the backend is healthy again, Gatus should report the endpoint as healthy on a subsequent check. Enable the service: @@ -504,13 +517,20 @@ sudo systemctl daemon-reload sudo systemctl enable --now vault-web-host-heartbeat.timer ``` -For backup monitoring, install the hook at a stable path and run it only after the backup and verification succeed: +For backup monitoring, install the hook and attach it to the existing systemd backup service with an `ExecStartPost` drop-in. systemd runs `ExecStartPost` only after the backup command exits successfully, so failed backups never send a success heartbeat: ```bash sudo install -m 0755 /opt/deploy/monitoring/healthchecks/backup-heartbeat.sh /usr/local/sbin/vault-web-backup-heartbeat +sudo install -d -m 0755 /etc/systemd/system/backup-vault-users.service.d +sudo tee /etc/systemd/system/backup-vault-users.service.d/healthchecks.conf >/dev/null <<'EOF' +[Service] +EnvironmentFile=/etc/default/vault-web-heartbeats +ExecStartPost=/usr/local/sbin/vault-web-backup-heartbeat +EOF +sudo systemctl daemon-reload ``` -Then add `/usr/local/sbin/vault-web-backup-heartbeat` as the final command in the successful path of `/usr/local/sbin/backup-vault-users.sh`. Keep it after rsync, retention cleanup and any verification steps. The heartbeat hook is not automatically wired into the pre-existing backup script, so configure this host-side step explicitly. Never ping at backup start, because a later failure would be incorrectly reported as success. Set healthchecks.io grace periods to exceed the normal job duration; missed pings trigger its configured email/webhook alerts. +The required `EnvironmentFile` passes `HC_BACKUP_UUID` from `/etc/default/vault-web-heartbeats` into the hook. The hook runs after the existing backup script, including its retention and verification steps. Do not call it at backup start: a later backup failure must not be reported as success. Test the configured path with a successful backup run, then confirm the backup monitor's last-ping time updates in healthchecks.io. Set its grace period longer than the normal backup duration; missed pings trigger its configured email/webhook alerts. ### Troubleshooting