Skip to content

test(backend): add comprehensive integration and stress test suite for Idempotency Key Manager - #1608

Open
Abidoyesimze wants to merge 2 commits into
emdevelopa:mainfrom
Abidoyesimze:feature/be-add-comprehensive-integration-and-stress-test-suite-for-idempotency-key-manager
Open

Abidoyesimze wants to merge 2 commits into
emdevelopa:mainfrom
Abidoyesimze:feature/be-add-comprehensive-integration-and-stress-test-suite-for-idempotency-key-manager

Conversation

@Abidoyesimze

Copy link
Copy Markdown
Contributor

Summary

Closes #1466

Depends on #1607 (the Idempotency Key Manager locking rewrite). This branch is stacked on #1607's branch, so until #1607 merges the diff below also shows that commit — after #1607 merges this PR auto-retargets to main and shrinks to just the test suite. These tests exercise the locking behaviour added there, so they cannot pass against main as-is.

Adds the integration and stress suite for the Idempotency Key Manager, which had none: the middleware's correctness properties (exactly-once execution, replay identity, conflict semantics, lock expiry, TTL behaviour) were previously only argued about, never executed.

New shared test double src/lib/__testing__/fake-redis.js — an in-memory Redis with the exact command surface the middleware uses (SET with NX/PX, GET, DEL, EVALSHA/Lua script semantics, TTL expiry with an injectable clock), so tests are deterministic and run without a Redis container. The existing unit tests were refactored to import this same double instead of defining their own.

New suite src/lib/idempotency.integration.test.js:

  • Concurrency: N concurrent requests with the same key under a slow handler → handler executes exactly once; the others get 409. Different keys run in parallel.
  • Replay: identical method/path/body → stored status + body + Idempotency-Replayed header, handler not re-executed. Mutated body → mismatch rejection. Missing/changed key material → not treated as a replay.
  • Crash recovery: lock TTL expiry lets a new request re-acquire; a holder that dies before completion doesn't wedge the key forever; Lua CAS mismatch is counted as an orphaned completion.
  • Stress: a few hundred mixed operations (creates, replays, conflicts) against the fake — total executions equal the number of distinct keys, zero orphaned locks, counters end in the expected state.
  • Real Redis suite (opt-in): same properties run against a live Redis when REDIS_URL is set; skipped by default (describe.skipIf) so CI without Redis stays green.

Test plan

npx vitest run src/lib/idempotency.test.js src/lib/idempotency.integration.test.js
 Test Files  2 passed (2)
      Tests  22 passed | 1 skipped (23)

(the 1 skip is the real-Redis suite, gated on REDIS_URL)

Full backend suite: no new failures vs. origin/main (identical pre-existing failure count).

Security notes

  • The fake Redis lives under src/lib/__testing__/ and is only imported by tests; nothing in the runtime import graph pulls it in.
  • No real credentials are used or required; the real-Redis suite reads REDIS_URL from the environment like the application does.
  • Stress tests assert bounded key counts and no leaked locks, so a regression that leaks Redis keys (a slow memory-growth vector under key-guessing traffic) fails the suite.

…r Idempotency Key Manager

Replaces the check-then-set idempotency flow (GET followed by a
fire-and-forget SET) with a distributed lock built on an atomic
SET NX PX acquisition keyed per merchant+key:

- Concurrent duplicates now receive 409 while the original request is
  still in flight instead of both executing (the previous race let every
  duplicate through).
- Completion is written with two Lua compare-and-swap scripts so a
  worker that lost its lock (TTL expiry or crash) can never clobber the
  state of the request that re-acquired the key; orphaned completions
  are counted and logged instead.
- Non-2xx responses release the lock immediately so clients can retry,
  and connections that close before a response release it too.
- Replays now return the original status code (legacy 201 fallback) and
  set Idempotency-Replayed; legacy {"hash","response"} records are still
  honoured so the change is backwards compatible.
- Redis failures still fail open but are now logged and counted.

Adds Prometheus instrumentation (idempotency_requests_total,
idempotency_lock_duration_seconds, idempotency_orphaned_completions_total)
and unit coverage for the race, lock TTL self-healing, payload mismatch,
legacy replay, fail-open and lock cleanup paths.
…r Idempotency Key Manager

Adds integration/bench-style tests for the distributed idempotency lock under
contention: multi-way duplication across many keys, 100-parallel hammering of
the same key, multi-instance sharing of a single Redis store, time-budget
checks, racey payload mismatch against an in-flight request, lock self-healing
after simulated crash (TTL expiry), TTL persistence of completed records, and a
skippable real-Redis Lua CAS smoke test. Also adds a reusable in-memory FakeRedis
double that mirrors the script semantics for isolated/fast runs.
@vercel

vercel Bot commented Oct 17, 2026

Copy link
Copy Markdown

@Abidoyesimze is attempting to deploy a commit to the Emmanuel's projects Team on Vercel.

A member of the Team first needs to authorize it.

@drips-wave

drips-wave Bot commented Oct 17, 2026

Copy link
Copy Markdown

@Abidoyesimze Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Backend] Add comprehensive integration and stress test suite for Idempotency Key Manager

1 participant