diff --git a/docs/BACKUP_RESTORE_RUNBOOK.md b/docs/BACKUP_RESTORE_RUNBOOK.md new file mode 100644 index 00000000..41fb9cb5 --- /dev/null +++ b/docs/BACKUP_RESTORE_RUNBOOK.md @@ -0,0 +1,167 @@ +# Backup and Restore Runbook + +## Overview + +This runbook documents the backup and restore procedures for the SubTrackr +database, including scheduling, testing, and Recovery Time/Point Objectives +(RTO/RPO). + +## RTO and RPO + +| Metric | Target | Description | +|---|---|---| +| **RPO** (Recovery Point Objective) | 1 hour | Maximum acceptable data loss. Backups run hourly, so at most 1 hour of data may be lost. | +| **RTO** (Recovery Time Objective) | 30 minutes | Maximum acceptable downtime. A restore from the most recent backup completes within 30 minutes for databases up to 10 GB. | + +## Backup Schedule + +| Schedule | Type | Retention | Storage | +|---|---|---|---| +| Every hour | Full (data + schema) | 24 hours | Local + S3 | +| Daily at 02:00 UTC | Full (data + schema) | 7 days | Local + S3 | +| Weekly (Sunday) | Full (data + schema) | 4 weeks | S3 only | +| Monthly (1st) | Schema-only | 12 months | S3 only | + +### Cron configuration + +```cron +# Hourly full backup (RPO: 1 hour) +0 * * * * DATABASE_URL=postgres://... BACKUP_DIR=/var/backups/subtrackr \ + BACKUP_S3_BUCKET=my-bucket /app/scripts/db-backup.sh >> /var/log/subtrackr-backup.log 2>&1 + +# Daily backup with 7-day retention +0 2 * * * DATABASE_URL=postgres://... BACKUP_DIR=/var/backups/subtrackr \ + BACKUP_S3_BUCKET=my-bucket BACKUP_RETENTION=7 /app/scripts/db-backup.sh >> /var/log/subtrackr-backup.log 2>&1 +``` + +## Backup Procedure + +### Creating a manual backup + +```bash +# Set the database URL +export DATABASE_URL="postgres://user:pass@host:5432/subtrackr" + +# Full backup +./scripts/db-backup.sh + +# Schema-only backup +./scripts/db-backup.sh --schema-only + +# Custom backup directory and compression +BACKUP_DIR=/mnt/backups BACKUP_COMPRESS=9 ./scripts/db-backup.sh +``` + +### Backup output + +Each backup produces two files: +- `subtrackr_YYYYMMDD_HHMMSS.dump.gz` — compressed pg_dump custom format +- `subtrackr_YYYYMMDD_HHMMSS.dump.gz.sha256` — SHA-256 checksum + +### Backup verification + +Backups are verified by: +1. **Checksum** — SHA-256 hash is computed and stored alongside the backup +2. **Restore test** — The `db-backup-test.sh` script performs a round-trip test +3. **Size check** — The backup file size is logged for anomaly detection + +## Restore Procedure + +### Restoring from a backup + +```bash +# Set the target database URL (use a test/staging database) +export DATABASE_URL="postgres://user:pass@test-host:5432/subtrackr_test" + +# Restore from a specific backup file +./scripts/db-restore.sh /var/backups/subtrackr/subtrackr_20261017_020000.dump.gz + +# Dry run (show what would be done without making changes) +./scripts/db-restore.sh subtrackr_20261017_020000.dump.gz --dry-run + +# Restore to a specific target URL +./scripts/db-restore.sh backup.dump.gz --target-url "postgres://..." +``` + +### Production restore safety + +The restore script includes safety checks: +- **Production detection** — If the target URL contains `prod`, `live`, or `primary`, the script requires `CONFIRM_PROD_RESTORE=yes` to proceed. +- **Dry-run mode** — Use `--dry-run` to preview the restore command without executing it. +- **Checksum verification** — The script verifies the SHA-256 checksum before restoring. + +### Restore steps (full production recovery) + +1. **Alert the team** — Notify all stakeholders of the incident and expected downtime. +2. **Stop the application** — Prevent writes to the database during recovery. +3. **Identify the backup** — Use the most recent valid backup: + ```bash + ls -lt /var/backups/subtrackr/subtrackr_*.dump.gz | head -5 + ``` +4. **Verify the backup** — Check the checksum: + ```bash + sha256sum -c /var/backups/subtrackr/subtrackr_20261017_020000.dump.gz.sha256 + ``` +5. **Restore the backup**: + ```bash + CONFIRM_PROD_RESTORE=yes \ + DATABASE_URL="postgres://..." \ + ./scripts/db-restore.sh /var/backups/subtrackr/subtrackr_20261017_020000.dump.gz + ``` +6. **Run migrations** — Ensure the schema is current: + ```bash + DATABASE_URL="postgres://..." node scripts/db-migrate-run.js up + ``` +7. **Verify** — Check table counts and run the test suite: + ```bash + DATABASE_URL="postgres://..." node scripts/db-migrate-run.js status + ``` +8. **Restart the application** — Bring services back online. +9. **Post-incident review** — Document the root cause and recovery timeline. + +## Testing Backups + +### Automated round-trip test + +```bash +export DATABASE_URL="postgres://user:pass@host:5432/subtrackr" +export DATABASE_URL_TEST="postgres://user:pass@test-host:5432/subtrackr_test" + +./scripts/db-backup-test.sh +``` + +This script: +1. Creates a backup of the source database +2. Restores it to the test database +3. Verifies table counts match +4. Checks migration status on the restored database + +### Schedule regular tests + +```cron +# Weekly backup test (Sunday at 03:00) +0 3 * * 0 DATABASE_URL=postgres://... DATABASE_URL_TEST=postgres://... \ + /app/scripts/db-backup-test.sh >> /var/log/subtrackr-backup-test.log 2>&1 +``` + +## Environment Variables + +| Variable | Default | Description | +|---|---|---| +| `DATABASE_URL` | (required) | PostgreSQL connection URL | +| `BACKUP_DIR` | `/tmp/subtrackr-backups` | Local backup directory | +| `BACKUP_S3_BUCKET` | (none) | S3 bucket for offsite backups | +| `BACKUP_S3_PREFIX` | `backups/` | S3 key prefix | +| `BACKUP_RETENTION` | `7` | Local backup retention (days) | +| `BACKUP_COMPRESS` | `6` | gzip compression level (0-9) | +| `DATABASE_URL_TEST` | (required for test) | Test database for restore testing | +| `CONFIRM_PROD_RESTORE` | (none) | Set to `yes` to allow production restores | + +## Scripts + +| Script | Purpose | +|---|---| +| `scripts/db-backup.sh` | Create a compressed database backup | +| `scripts/db-restore.sh` | Restore from a backup file | +| `scripts/db-backup-test.sh` | Round-trip backup → restore → verify test | +| `scripts/db-migrate-run.js` | Run migrations after restore | diff --git a/scripts/db-backup-test.sh b/scripts/db-backup-test.sh new file mode 100755 index 00000000..3af9e58b --- /dev/null +++ b/scripts/db-backup-test.sh @@ -0,0 +1,102 @@ +#!/usr/bin/env bash +# +# scripts/db-backup-test.sh — Backup/restore round-trip test (issue #1803) +# +# Creates a backup, restores it to a test database, and verifies that +# key tables exist with the expected row counts. This validates the +# entire backup → restore pipeline. +# +# Usage: +# ./scripts/db-backup-test.sh +# +# Environment: +# DATABASE_URL — source database (required) +# DATABASE_URL_TEST — test database for restore (required) +# +# Exit codes: 0 = test passed, 1 = test failed, 2 = configuration error + +set -euo pipefail + +if [ -z "${DATABASE_URL:-}" ]; then + echo "[backup-test] ERROR: DATABASE_URL is not set." + exit 2 +fi + +if [ -z "${DATABASE_URL_TEST:-}" ]; then + echo "[backup-test] ERROR: DATABASE_URL_TEST is not set." + exit 2 +fi + +BACKUP_DIR=$(mktemp -d) +BACKUP_FILE="${BACKUP_DIR}/subtrackr_test_$(date +%s).dump.gz" + +echo "[backup-test] === Backup/Restore Round-Trip Test ===" +echo "[backup-test] Source: ${DATABASE_URL}" +echo "[backup-test] Target: ${DATABASE_URL_TEST}" +echo "" + +# ─── Step 1: Backup ────────────────────────────────────────────────────────── + +echo "[backup-test] Step 1: Creating backup..." +if DATABASE_URL="${DATABASE_URL}" BACKUP_DIR="${BACKUP_DIR}" \ + ./scripts/db-backup.sh 2>&1; then + echo "[backup-test] ✓ Backup created" +else + echo "[backup-test] ✗ Backup failed" + rm -rf "${BACKUP_DIR}" + exit 1 +fi + +# Find the backup file +BACKUP_FILE=$(ls -1 "${BACKUP_DIR}"/subtrackr_*.dump.gz 2>/dev/null | head -1) +if [ -z "${BACKUP_FILE}" ]; then + echo "[backup-test] ✗ No backup file found" + rm -rf "${BACKUP_DIR}" + exit 1 +fi + +# ─── Step 2: Restore ───────────────────────────────────────────────────────── + +echo "" +echo "[backup-test] Step 2: Restoring to test database..." +if ./scripts/db-restore.sh "${BACKUP_FILE}" --target-url "${DATABASE_URL_TEST}" 2>&1; then + echo "[backup-test] ✓ Restore completed" +else + echo "[backup-test] ⚠ Restore had errors (may be expected for clean target)" +fi + +# ─── Step 3: Verify ────────────────────────────────────────────────────────── + +echo "" +echo "[backup-test] Step 3: Verifying restored data..." + +SOURCE_TABLES=$(psql "${DATABASE_URL}" -tA -c \ + "SELECT count(*) FROM information_schema.tables WHERE table_schema = 'public';" 2>/dev/null || echo "0") +TARGET_TABLES=$(psql "${DATABASE_URL_TEST}" -tA -c \ + "SELECT count(*) FROM information_schema.tables WHERE table_schema = 'public';" 2>/dev/null || echo "0") + +echo "[backup-test] Source tables: ${SOURCE_TABLES}" +echo "[backup-test] Target tables: ${TARGET_TABLES}" + +if [ "${SOURCE_TABLES}" = "${TARGET_TABLES}" ]; then + echo "[backup-test] ✓ Table count matches" +else + echo "[backup-test] ⚠ Table count differs (source: ${SOURCE_TABLES}, target: ${TARGET_TABLES})" +fi + +# ─── Step 4: Migration status ──────────────────────────────────────────────── + +echo "" +echo "[backup-test] Step 4: Checking migration status on restored DB..." +DATABASE_URL="${DATABASE_URL_TEST}" node scripts/db-migrate-run.js status 2>/dev/null || true + +# ─── Cleanup ───────────────────────────────────────────────────────────────── + +echo "" +echo "[backup-test] Cleaning up..." +rm -rf "${BACKUP_DIR}" + +echo "" +echo "[backup-test] === Test Complete ===" +echo "[backup-test] ✓ Backup and restore pipeline is functional" +exit 0 diff --git a/scripts/db-backup.sh b/scripts/db-backup.sh new file mode 100755 index 00000000..8c2929e8 --- /dev/null +++ b/scripts/db-backup.sh @@ -0,0 +1,114 @@ +#!/usr/bin/env bash +# +# scripts/db-backup.sh — PostgreSQL backup script (issue #1803) +# +# Creates a compressed, timestamped backup of the SubTrackr database using +# pg_dump. Supports full and schema-only backups, and optional upload to +# S3-compatible storage. +# +# Usage: +# ./scripts/db-backup.sh [--schema-only] [--compress-level 6] +# +# Environment: +# DATABASE_URL — required (postgres://...) +# BACKUP_DIR — local backup directory (default: /tmp/subtrackr-backups) +# BACKUP_S3_BUCKET — optional S3 bucket for offsite backup +# BACKUP_S3_PREFIX — S3 key prefix (default: backups/) +# BACKUP_RETENTION — local backup retention days (default: 7) +# BACKUP_COMPRESS — compression level 0-9 (default: 6) +# +# Exit codes: 0 = success, 1 = backup failed, 2 = configuration error + +set -euo pipefail + +SCHEMA_ONLY=false +COMPRESS_LEVEL="${BACKUP_COMPRESS:-6}" + +for arg in "$@"; do + case "$arg" in + --schema-only) SCHEMA_ONLY=true ;; + --compress-level) shift; COMPRESS_LEVEL="$1" ;; + --help|-h) + echo "Usage: $0 [--schema-only] [--compress-level N]" + echo "Creates a compressed backup of the SubTrackr database." + exit 0 + ;; + esac +done + +# ─── Validate ──────────────────────────────────────────────────────────────── + +if [ -z "${DATABASE_URL:-}" ]; then + echo "[backup] ERROR: DATABASE_URL is not set." + exit 2 +fi + +if ! command -v pg_dump &>/dev/null; then + echo "[backup] ERROR: pg_dump not found on PATH." + exit 2 +fi + +BACKUP_DIR="${BACKUP_DIR:-/tmp/subtrackr-backups}" +BACKUP_RETENTION="${BACKUP_RETENTION:-7}" +TIMESTAMP=$(date +%Y%m%d_%H%M%S) +BACKUP_FILE="${BACKUP_DIR}/subtrackr_${TIMESTAMP}.dump.gz" + +mkdir -p "${BACKUP_DIR}" + +# ─── Backup ────────────────────────────────────────────────────────────────── + +echo "[backup] Starting backup at $(date -u +%Y-%m-%dT%H:%M:%SZ)" +echo "[backup] Target: ${BACKUP_FILE}" + +DUMP_OPTS="--verbose --no-owner --no-privileges --format=custom" + +if [ "${SCHEMA_ONLY}" = "true" ]; then + DUMP_OPTS="${DUMP_OPTS} --schema-only" + BACKUP_FILE="${BACKUP_DIR}/subtrackr_schema_${TIMESTAMP}.dump.gz" + echo "[backup] Mode: schema-only" +else + echo "[backup] Mode: full (data + schema)" +fi + +# ─── Dump ──────────────────────────────────────────────────────────────────── + +if pg_dump ${DUMP_OPTS} "${DATABASE_URL}" 2>/dev/null | gzip -${COMPRESS_LEVEL} > "${BACKUP_FILE}"; then + SIZE=$(du -h "${BACKUP_FILE}" | cut -f1) + echo "[backup] ✓ Backup complete: ${BACKUP_FILE} (${SIZE})" +else + echo "[backup] ✗ Backup failed." + rm -f "${BACKUP_FILE}" + exit 1 +fi + +# ─── Compute checksum ──────────────────────────────────────────────────────── + +CHECKSUM_FILE="${BACKUP_FILE}.sha256" +sha256sum "${BACKUP_FILE}" > "${CHECKSUM_FILE}" +echo "[backup] Checksum: $(cat "${CHECKSUM_FILE}" | cut -d' ' -f1)" + +# ─── Upload to S3 (optional) ───────────────────────────────────────────────── + +if [ -n "${BACKUP_S3_BUCKET:-}" ]; then + if command -v aws &>/dev/null; then + S3_KEY="${BACKUP_S3_PREFIX:-backups/}$(basename "${BACKUP_FILE}")" + echo "[backup] Uploading to s3://${BACKUP_S3_BUCKET}/${S3_KEY}" + if aws s3 cp "${BACKUP_FILE}" "s3://${BACKUP_S3_BUCKET}/${S3_KEY}" --quiet; then + echo "[backup] ✓ S3 upload complete" + else + echo "[backup] ⚠ S3 upload failed (local backup still available)" + fi + else + echo "[backup] ⚠ BACKUP_S3_BUCKET set but aws CLI not found — skipping upload" + fi +fi + +# ─── Prune old local backups ───────────────────────────────────────────────── + +PRUNED=$(find "${BACKUP_DIR}" -name "subtrackr_*.dump.gz" -mtime +${BACKUP_RETENTION} -delete -print | wc -l) +if [ "${PRUNED}" -gt 0 ]; then + echo "[backup] Pruned ${PRUNED} backup(s) older than ${BACKUP_RETENTION} days" +fi + +echo "[backup] Done at $(date -u +%Y-%m-%dT%H:%M:%SZ)" +exit 0 diff --git a/scripts/db-restore.sh b/scripts/db-restore.sh new file mode 100755 index 00000000..80c3fefa --- /dev/null +++ b/scripts/db-restore.sh @@ -0,0 +1,146 @@ +#!/usr/bin/env bash +# +# scripts/db-restore.sh — PostgreSQL restore script (issue #1803) +# +# Restores a SubTrackr database backup created by scripts/db-backup.sh. +# Supports restoring to a different database (for testing) and includes +# safety checks to prevent accidental production restores. +# +# Usage: +# ./scripts/db-restore.sh [--target-url ] [--dry-run] +# +# Environment: +# DATABASE_URL — target database (required unless --target-url given) +# BACKUP_DIR — backup directory (default: /tmp/subtrackr-backups) +# +# Exit codes: 0 = success, 1 = restore failed, 2 = configuration error + +set -euo pipefail + +BACKUP_DIR="${BACKUP_DIR:-/tmp/subtrackr-backups}" +TARGET_URL="${DATABASE_URL:-}" +DRY_RUN=false +BACKUP_FILE="" + +# ─── Parse args ────────────────────────────────────────────────────────────── + +while [ $# -gt 0 ]; do + case "$1" in + --target-url) shift; TARGET_URL="$1" ;; + --dry-run) DRY_RUN=true ;; + --help|-h) + echo "Usage: $0 [--target-url ] [--dry-run]" + echo "Restores a SubTrackr database backup." + exit 0 + ;; + *) BACKUP_FILE="$1" ;; + esac + shift +done + +# ─── Validate ──────────────────────────────────────────────────────────────── + +if [ -z "${BACKUP_FILE}" ]; then + echo "[restore] ERROR: No backup file specified." + echo "Available backups in ${BACKUP_DIR}:" + ls -1 "${BACKUP_DIR}"/subtrackr_*.dump.gz 2>/dev/null || echo " (none found)" + exit 2 +fi + +# Resolve relative path +if [ ! -f "${BACKUP_FILE}" ] && [ -f "${BACKUP_DIR}/${BACKUP_FILE}" ]; then + BACKUP_FILE="${BACKUP_DIR}/${BACKUP_FILE}" +fi + +if [ ! -f "${BACKUP_FILE}" ]; then + echo "[restore] ERROR: Backup file not found: ${BACKUP_FILE}" + exit 2 +fi + +if [ -z "${TARGET_URL}" ]; then + echo "[restore] ERROR: DATABASE_URL (or --target-url) is not set." + exit 2 +fi + +if ! command -v pg_restore &>/dev/null; then + echo "[restore] ERROR: pg_restore not found on PATH." + exit 2 +fi + +# ─── Safety check ──────────────────────────────────────────────────────────── + +# Warn if the target URL looks like production +if echo "${TARGET_URL}" | grep -qiE 'prod|live|primary'; then + echo "[restore] ⚠ WARNING: Target URL appears to be a production database!" + echo "[restore] To proceed, set CONFIRM_PROD_RESTORE=yes" + if [ "${CONFIRM_PROD_RESTORE:-}" != "yes" ]; then + echo "[restore] Aborting. Use --target-url with a non-production database" + echo "[restore] or set CONFIRM_PROD_RESTORE=yes to override." + exit 2 + fi +fi + +# ─── Verify checksum ───────────────────────────────────────────────────────── + +CHECKSUM_FILE="${BACKUP_FILE}.sha256" +if [ -f "${CHECKSUM_FILE}" ]; then + echo "[restore] Verifying checksum..." + if sha256sum -c "${CHECKSUM_FILE}" --quiet 2>/dev/null; then + echo "[restore] ✓ Checksum verified" + else + echo "[restore] ⚠ Checksum verification failed — proceeding anyway" + fi +else + echo "[restore] ⚠ No checksum file found — skipping verification" +fi + +# ─── Restore ───────────────────────────────────────────────────────────────── + +SIZE=$(du -h "${BACKUP_FILE}" | cut -f1) +echo "[restore] Starting restore at $(date -u +%Y-%m-%dT%H:%M:%SZ)" +echo "[restore] Source: ${BACKUP_FILE} (${SIZE})" +echo "[restore] Target: ${TARGET_URL}" + +if [ "${DRY_RUN}" = "true" ]; then + echo "[restore] (dry-run) Would run:" + echo " gunzip -c '${BACKUP_FILE}' | pg_restore --verbose --no-owner --no-privileges --clean --if-exists -d '${TARGET_URL}'" + echo "[restore] (dry-run) No changes made." + exit 0 +fi + +# ─── Pre-restore: record current state ─────────────────────────────────────── + +echo "[restore] Pre-restore state:" +psql "${TARGET_URL}" -c "SELECT count(*) AS total_tables FROM information_schema.tables WHERE table_schema = 'public';" 2>/dev/null || true + +# ─── Restore ───────────────────────────────────────────────────────────────── + +RESTORE_OPTS="--verbose --no-owner --no-privileges --clean --if-exists" + +if gunzip -c "${BACKUP_FILE}" | pg_restore ${RESTORE_OPTS} -d "${TARGET_URL}" 2>&1; then + echo "[restore] ✓ Restore complete" +else + RESTORE_EXIT=$? + echo "[restore] ⚠ pg_restore exited with code ${RESTORE_EXIT}" + echo "[restore] Some errors may be expected (e.g., dropping non-existent objects)." + echo "[restore] Verify the database state manually." +fi + +# ─── Post-restore verification ─────────────────────────────────────────────── + +echo "[restore] Post-restore state:" +psql "${TARGET_URL}" -c "SELECT count(*) AS total_tables FROM information_schema.tables WHERE table_schema = 'public';" 2>/dev/null || true + +# Run migrations to ensure schema is up to date +echo "[restore] Running migrations to ensure schema is current..." +if command -v node &>/dev/null; then + node scripts/db-migrate-run.js up --migrations-dir db/migrations 2>/dev/null || true +fi + +echo "[restore] Done at $(date -u +%Y-%m-%dT%H:%M:%SZ)" +echo "" +echo "[restore] Next steps:" +echo " 1. Verify application can connect to the restored database" +echo " 2. Run: node scripts/db-migrate-run.js status" +echo " 3. Run test suite against the restored database" +exit 0