feat(backend): telemetry, retry, distributed locking and stress tests for Log Retention & Archival Service

[Olorunfemi20]

Summary

#1473 Prometheus alert metrics and health telemetry

• New src/lib/log-retention-telemetry.js, with its own registry merged into GET /metrics. Series:
  • log_retention_purge_runs_total{outcome} (success, failure, skipped)
  • log_retention_purge_duration_seconds{outcome}
  • log_retention_purge_deleted_rows_total{table} and log_retention_purge_batches_total{table}
  • log_retention_purge_duration_exceeded_total
  • log_retention_purge_retries_total{result}
  • log_retention_purge_lock_total{result}
  • gauges: health state, in progress, consecutive failures, last success time, last run time
• Health is derived from run history:
  • idle: no run in this process yet
  • healthy
  • degraded: last run failed, overran its time budget, or a run is overdue
  • unhealthy: failure streak at the limit, or no success within the staleness limit
• Endpoints: new GET /health/log-retention (503 only when unhealthy). GET /health lists services.log_retention for information only; it does not change ok.
• The cron script pushes its metrics to a Pushgateway when LOG_RETENTION_PUSHGATEWAY_URL is set, because the process exits before any scrape could reach it.
• Alert rules: docs/alerts/log-retention.rules.yml (unhealthy, failing, stale, duration exceeded, retries exhausted, lock errors).
• Tests: src/lib/log-retention-telemetry.test.js covers every health transition, label sanitising, gauges refreshed at scrape time, and the Pushgateway success, refusal and failure paths.

#1474 Automated retry with exponential backoff

• New src/lib/log-retention-retry.js. Every purge query runs through withLogRetentionRetry(): index check, FK check, each delete batch, lock connect and lock acquire.
• Retried: SQLSTATE 40001, 40P01, 55P03, 57014, 57P01-57P03, 53300, class 08, and socket errors.
• Never retried: safety-check failures (new LogRetentionSafetyError, retryable: false) and any other SQLSTATE.
• Delay: full jitter, random(0, min(max, base * 2^n)).
• Settings: LOG_PURGE_RETRY_MAX_ATTEMPTS / _BASE_DELAY_MS / _MAX_DELAY_MS, clamped to hard ceilings (8 attempts, 60 s). Retrying stops once the run passes LOG_PURGE_MAX_DURATION_MS.
• Tests: src/lib/log-retention-retry.test.js (classification, option clamping, backoff math, recovery, exhaustion, abort, log fields) and new cases in src/lib/log-retention.test.js.

#1475 Distributed concurrency control and locking

• New src/lib/log-retention-lock.js:
  • an in-process single flight per pool;
  • a Postgres session advisory lock, pg_try_advisory_lock(hashtext(<fixed key>)), on a dedicated pooled connection. It never waits; contended means skipped, not failed.
  • Postgres drops the lock if the holder dies. If unlocking fails, the connection is destroyed instead of returned to the pool, which also drops the lock.
• Delete batches now use FOR UPDATE SKIP LOCKED, so even unlocked callers never block on, or double-delete, each other's rows.
• New runLogRetentionPurge() wraps purgeWebhookLogs() in the lock and returns { skipped: true, reason: "locked" } when another run holds it.
• scripts/purge-webhook-logs.js now uses it.
• Optional in-API schedule LOG_RETENTION_PURGE_CRON (off by default), safe on every instance.
• Tests: src/lib/log-retention-lock.test.js (acquire, contend, in-process skip, per-pool instances, acquire errors, retry, release failures, single client) and runLogRetentionPurge cases in src/lib/log-retention.test.js.

#1476 Integration and stress test suite

• tests/helpers/fake-log-retention-db.js: an in-memory Postgres double implementing the safety checks, DELETE ... FOR UPDATE SKIP LOCKED (rejects a delete without it), and session advisory locks. Several pools sharing one database simulate several instances. It records deleted ids, double deletes, peak lock holders, open connections and held locks.
• tests/integration/log-retention.test.js covers the full lock, retry, purge and telemetry pipeline:
  • batching, and only expired rows deleted;
  • skip while another host holds the lock;
  • one of two concurrent instances runs; a same-process double trigger is skipped;
  • recovery from transient lock, check and delete failures;
  • a clean failure after exhaustion; the missing-index refusal; dropping the lock when unlock fails; overlapping unlocked callers;
  • over HTTP through the real app: /health/log-retention (idle, healthy, 503), /health and /metrics.
• load-tests/log-retention-stress.test.js:
  • 50 racing instances: exactly one purges 5,000 rows;
  • 10 waves with 20% injected transient failures and random latency (seeded PRNG): converge with no leaks;
  • 16 unlocked callers: every row deleted exactly once;
  • a 100-trigger burst on one instance: only one reaches the database lock;
  • a 50,000-row backlog throughput check.

Test output

Security notes

• Table, column and lock key are server-side constants. Only retention days and batch size reach SQL, and only as bind parameters.
• Metric labels come from fixed sets: no row ids, log contents or error text. Unknown values map to other / failure / error.
• Health bodies contain counts and timestamps only.
• The Pushgateway URL must be http(s) and is never logged, because it may carry credentials.
• Retry ceilings prevent a failing purge from becoming load on the primary. Safety checks still fail closed and are never retried.
• The lock never waits and cannot get stuck past the holder's connection.

Notes

• Reading of the issues: "Log Retention & Archival Service" is src/lib/log-retention.js and its cron script, the webhook delivery log purge. Payment archival in maintenance.js is untouched.
• Backwards compatibility:
  • purgeWebhookLogs() keeps its signature and return value, and still takes no lock. It gains an optional retry option.
  • The existing console log and alert lines are unchanged.
  • Schedulers should use runLogRetentionPurge().
• Behaviour changes:
  • The delete batch query adds FOR UPDATE SKIP LOCKED.
  • Transient query failures are now retried instead of failing the run immediately.
  • The cron script exits successfully with a "Run skipped" line when another host holds the lock.
• Health is per process. API instances report idle unless LOG_RETENTION_PURGE_CRON is set there; for the cron script, use the Pushgateway metrics.
• New metrics also register on prom-client's default registry, the same as the existing per-module registries.
• No existing tests or fixtures were changed. The original two tests in log-retention.test.js are untouched; new cases were appended.

closes #1473
closes #1474
closes #1475
closes #1476

[drips-wave]

@Olorunfemi20 Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

[vercel]

@Olorunfemi20 is attempting to deploy a commit to the Emmanuel's projects Team on Vercel.

A member of the Team first needs to authorize it.