Feature: Add Scheduled Background Job Runner #152

Description

Every deferred queue in the API is drained opportunistically from the request that enqueued the work, not on a schedule. EmailService, NotificationService, WebhookService, and StellarFundingService each call this.processQueue().catch(...) immediately after enqueueing, so a row whose nextAttemptAt is in the future is only retried if unrelated traffic happens to trigger another drain. When traffic stops, retries stop, and failed deliveries can sit indefinitely. Only the account-lifecycle sweep has a real timer (setInterval in src/server.ts), and its own comment notes that a dedicated worker is the intended owner.

Introduce one scheduled runner process that owns every recurring queue drain, so delivery is driven by time rather than by incidental request volume, and so the work stops competing with request handling inside the API process.

File Location

learnault-api/src/workers/, learnault-api/src/server.ts, learnault-api/docker/, learnault-api/docker-compose.yml, and learnault-api/package.json

Design Reference

API Roadmap Phase 1.1: Add Scheduled Background Job Runner. Builds on the leasing primitives delivered by #122.

Dependencies

• Status: Ready
• Blocked by: Feature: Add Transaction Outbox and Job Delivery Foundation #122 (Add Transaction Outbox and Job Delivery Foundation) — done
• Blocks: Feature: Add Outbox Event Relay and Handler Registry #153 (Add Outbox Event Relay and Handler Registry)

Tasks

• Add a scheduler entrypoint that owns every recurring drain on a configurable interval
• Register the email, notification, webhook, Stellar-funding, and data-export queues with it
• Lease each tick through job-lease.service.ts so multiple replicas cannot double-process
• Remove the fire-and-forget processQueue() calls from the request path, keeping enqueue synchronous
• Move the src/server.ts lifecycle sweep onto the runner and keep an opt-in flag for single-process deployments
• Emit per-queue depth, attempt, failure, and lag metrics, and honour graceful shutdown
• Add the runner to the Docker Compose stack and the production image

Acceptance Criteria

• Queued work is retried on schedule with no inbound HTTP traffic
• A due nextAttemptAt row is picked up within one configured interval
• Two concurrently running replicas never process the same row twice
• API request latency no longer includes queue-drain work
• Shutdown drains or releases in-flight leases without orphaning rows

Verification Evidence

• Attach an idle-instance run showing a failed delivery retried on schedule, and a two-replica run showing no duplicate processing

Difficulty

Advanced