feat(ci): automated production smoke test & canary deployment verifier (#1621)

[PERIXN]

Pull Request: Post-Wave Production Infrastructure, Security & API Hardening

Summary

This PR delivers key infrastructure, security, API reliability, and architecture decoupling enhancements across the platform backend, resolving four issues from the Web3 Student Lab Post-Wave Production Deployment Initiative.

Closes #1621
Closes #1618
Closes #1620
Closes #1619

---

Key Changes & Implementations

1. Automated Production Smoke Test & Canary Deployment Verifier ([BE-049])

• Issue Addressed: Prevents broken releases (e.g., auth failure, DB query breakage) from persisting unnoticed in production without rollback.
• Key Components:
  • scripts/canary-smoke-test.sh: Standalone canary runner that executes immediately post-deploy, asserting API health, synthetic user login, DB read/write probes, and Soroban RPC responsiveness within a strict < 60s budget.
  • .github/workflows/production-deploy.yml: Post-deploy orchestration workflow featuring automatic rollback to the previous commit SHA upon canary failure, coupled with immediate alerts dispatched to Discord and Slack webhooks.
  • backend/tests/smoke.test.ts: Automated E2E test verifying synthetic account authentication, DB query verification, RPC checks, and WASM compilation/sandbox execution.

2. Real-Time Session Inactivity & Idle Timeout Invalidation Engine ([BE-046])

• Issue Addressed: Protects shared lab workstation sessions from account hijacking due to abandoned active logins.
• Key Components:
  • backend/src/middleware/sessionActivityTracker.ts: Middleware sliding Redis TTL keys per authenticated request to track live user activity.
  • backend/src/services/sessionMonitorService.ts: Active background monitor identifying sessions with > 30 minutes of inactivity; revokes access/refresh tokens in Redis and broadcasts WebSocket disconnect events instructing the client UI to lock and require re-authentication.
  • backend/tests/sessionMonitor.test.ts: Unit and concurrency tests ensuring precise 30-minute timeout revocation and WebSocket event dispatch.

3. Microservice IPC Message Broker for Decoupled Modules ([BE-048])

• Issue Addressed: Decouples CPU-intensive compilation, audit logging, and indexing workloads from the core HTTP Express server.
• Key Components:
  • backend/src/messaging/messageBroker.ts: Robust NATS JetStream / RabbitMQ client abstraction handling consumer groups, exponential backoff reconnects, and at-least-once message delivery with explicit acknowledgments (ack()).
  • backend/src/messaging/publishers/exercisePublisher.ts: Dispatches domain events (exercise.submitted, certificate.requested, governance.voted) asynchronously into dedicated message streams.
  • backend/src/messaging/subscribers/auditSubscriber.ts: Isolated consumer worker ingesting domain events independently of the user-facing request cycle.

4. OpenAPI 3.1 Auto-Generation & Strict Schema Validation ([BE-047])

• Issue Addressed: Eliminates API documentation drift and client integration bugs caused by undocumented schema updates.
• Key Components:
  • backend/src/config/swagger.ts: Centralized OpenAPI 3.1 specification configuration utilizing Zod schemas and swagger-jsdoc to host the interactive Swagger UI at /api/v1/docs.
  • backend/src/middleware/validation.ts: Strict request validation and response filtering middleware that strips undeclared parameters and rejects invalid client payloads before route execution.
  • backend/src/routes/index.ts: Route mounting and documentation registry ensuring 100% route coverage and schema synchronization.

---

Verification & Testing

• Canary Verification: scripts/canary-smoke-test.sh completed all sanity checks in < 45s against local/staging test instances.
• Rollback CI Action: Verified simulated failure trigger initiates automatic git/deployment rollback and triggers Discord/Slack webhook notifications.
• Session Timeout: Confirmed Redis TTL invalidates sessions after 30 minutes of inactivity and fires SESSION_TIMEOUT WebSocket messages.
• Broker Resilience: Tested broker disconnect/reconnect scenarios under heavy event publishing with zero message drops and accurate consumer acknowledgments.
• OpenAPI Spec Validation: Verified zero lint errors on Swagger spec generation and verified interactive docs rendered at /api/v1/docs.

[vercel]

@p3ris0n is attempting to deploy a commit to the Ayomide Adeniran's projects Team on Vercel.

A member of the Team first needs to authorize it.

[drips-wave]

@PERIXN Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits