feat(backend): enhance distributed concurrency control and locking for Idempotency Key Manager

[Abidoyesimze]

Summary

Closes #1465

src/lib/idempotency.js previously used a naive GET-then-SET check with no locking: concurrent requests with the same key could both miss the cache and both execute the handler (duplicate writes), and a crashed in-flight request left a key claimed forever with no recovery path.

Rewrites the middleware around a Redis atomic lock:

• SET NX PX acquires a per-key lock before the handler runs, with a short TTL and a lock token so only the owner can release it.
• Result storage + lock release happen in a single Lua compare-and-set (COMPLETE_LOCK_SCRIPT) — a slow or killed request cannot overwrite a completed result, and a completed result cannot be clobbered by a re-acquired lock.
• RELEASE_LOCK_SCRIPT releases the lock with a token check (no unlock-by-accident).
• Concurrent requests for an in-flight key get 409 Conflict with Retry-After instead of duplicate execution.
• Exact replays return the stored status code, stored body and an Idempotency-Replayed: true header; request-body hashes must match or the request is rejected.
• Legacy records (rows written before this change, without the lock token) are still replayed read-only, so the deploy is backward compatible.
• Redis errors fail open (the original request still executes; a duplicate risk is preferable to an outage) and are logged and counted.

New metrics (in the main registry, backend/src/lib/metrics.js):

• idempotency_requests_total{outcome=completed|replayed|conflict|error}
• idempotency_lock_duration_seconds histogram
• idempotency_orphaned_completions_total (Lua CAS mismatches — a signal of crashed holders or clock skew)

Test plan

npx vitest run src/lib/idempotency.test.js src/lib/metrics.test.js
 Test Files  2 passed (2)
      Tests  25 passed (25)

15 of those are new idempotency unit tests: happy path, replay (same body → replayed; different body → mismatch), 409 on in-flight conflict, lock TTL expiry and re-acquisition, token-checked release, Lua CAS mismatch counting, legacy-record replay, Redis-error fail-open, and metric emission.

Full backend suite: no new failures vs. origin/main (177 pre-existing failures on main, identical count on this branch — mostly network-dependent oracle/horizon tests).

Security notes

• Lock tokens are random per-acquisition UUIDs; the release/complete Lua scripts compare the token, so one request can never release or complete another request's lock.
• Key TTLs bound the blast radius of a leaked lock: a crashed holder's key expires, and the orphaned-completion counter records CAS misses for later investigation.
• No merchant data, API keys or request bodies are written to Redis or emitted as metric labels — label values are a fixed server-side set.
• Fail-open was chosen deliberately over fail-closed: idempotency is a safety optimization, not an authorization control, so an Redis outage must not take down the payments API. The failure is logged and counted for alerting.

[vercel]

@Abidoyesimze is attempting to deploy a commit to the Emmanuel's projects Team on Vercel.

A member of the Team first needs to authorize it.

[drips-wave]

@Abidoyesimze Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits