feat(messaging): improve reliability of Telegram and WhatsApp message delivery (#493)

[OAKVISUALZ]

Summary

Closes #493

Strengthens message delivery, queue behavior, fallback handling, and manual recovery for Telegram bot notifications and WhatsApp communication.

Key Goals & Acceptance Criteria Addressed

• Delivery statuses are tracked: Messages are tracked with lifecycle states PENDING, SENDING, DELIVERED, FAILED, and DEAD_LETTER backed by the Prisma MessageDelivery model and indexed queries.
• Retry/fallback policy exists:
  • Exponential backoff with jitter (min(maxDelay, baseDelay * 2^(attempts-1)) * (1 + jitter)) preventing thundering herds.
  • Upstream provider rate-limit compliance: automatically parses HTTP 429 retry_after parameters for Telegram and Twilio/WhatsApp transient error codes.
  • Automated cross-channel fallback (Telegram ↔ WhatsApp) when primary delivery permanently fails and fallback credentials exist.
• Failures are visible in alerting and logs:
  • Prometheus metrics exposed for delivery counters, retries, fallbacks, dead-letters, queue depth, and delivery latency histogram.
  • alertingService trigger integrated when dead-letter threshold is exceeded during periodic background sweep.
• Safe manual recovery:
  • Admin endpoints with audit logging for inspecting message deliveries, manual retry of individual messages, bulk dead-letter queue re-enqueueing, and safe message cancellation.

---

Architectural Changes

1. Database Schema & Migrations (prisma/schema.prisma):

   • Added enums MessageChannel, MessageDeliveryStatus, and MessagePriority.
   • Added MessageDelivery model with compound indexes for queue sweeping ([status, nextAttemptAt], [channel, status]) and filtering (userId, recipient, createdAt).
   • Added migration SQL and rollback SQL scripts under prisma/migrations/20260927000000_add_message_deliveries/.

2. Core Delivery Engine (src/messaging/):

   • MessageDeliveryService: Unified singleton managing message dispatch, lifecycle tracking, backoff calculation, fallback orchestration, and administrative recovery.
   • Dual-mode persistence: Uses Prisma database in production with smooth memory fallback for unit testing environments.
   • Concurrency-bounded queue sweeping and worker processing.

3. Client Adapters:

   • src/utils/telegram-client.ts: Dedicated Telegram Bot API client with HTTP client adapter injection for deterministic unit testing, transient error classification, HTML mode, and retry-after header/response parsing.
   • src/utils/twilio-client.ts: Enhanced with transient error classifier (isWhatsAppTransientError) and retry-after extractor (extractWhatsAppRetryAfter).

4. Background Resilient Sweep Job (src/jobs/messageDeliverySweep.ts):

   • Periodic worker sweeping overdue PENDING messages ordered by priority (CRITICAL, HIGH, NORMAL, LOW) and timestamp.
   • Registered in server startup and graceful shutdown in src/index.ts.

5. Admin Tooling & Operations (src/routes/admin.ts):

   • GET /api/v1/admin/messages: Paginated list of tracked message deliveries with status/channel filters.
   • GET /api/v1/admin/messages/stats: Overview counts of messages across statuses and per-channel breakdown.
   • GET /api/v1/admin/messages/:id: Detailed message inspection.
   • POST /api/v1/admin/messages/:id/retry: Safe manual retry of failed or dead-lettered message.
   • POST /api/v1/admin/messages/retry-dead-letters: Bulk re-enqueueing of dead-lettered messages.
   • DELETE /api/v1/admin/messages/:id: Safe cancellation of pending or dead messages.
   • Scopes registered in src/middleware/adminAuth.ts: messages:read, messages:write.

6. Service & Notification Integrations:

   • src/routes/telegram.ts: Telegram webhook bot replies routed through messageDeliveryService.
   • src/jobs/alertRules.ts: Alert notification deliveries routed through messageDeliveryService.
   • src/strategy/service.ts: Strategy update notifications to followers routed through messageDeliveryService.

7. Documentation:

   • Added docs/MESSAGING_DELIVERY.md.
   • Updated docs/NOTIFICATIONS.md, docs/RUNBOOK.md, and docs/DOCUMENTATION_INDEX.md.

---

Verification & Testing

• TypeScript: npm run typecheck passed with 0 errors across the entire codebase.
• Unit Tests:
  • tests/unit/messaging/telegramClient.test.ts: PASSED
  • tests/unit/messaging/messageDeliveryService.test.ts: PASSED
  • tests/unit/jobs/messageDeliverySweep.test.ts: PASSED
  • tests/unit/routes/admin-messages.test.ts: PASSED
  • tests/unit/telegram/telegramHandler.test.ts: PASSED
  • tests/unit/telegram/telegramConfirmations.test.ts: PASSED
  • tests/unit/strategy/service.test.ts: PASSED
  • tests/unit/jobs/alertRules.test.ts: PASSED

[drips-wave]

@OAKVISUALZ Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

[Abidoyesimze]

fix conflicts