fix(backend): make transfer lifecycle transitions concurrency-safe

README.md

@@ -260,6 +260,9 @@ rounded away.
 - `POST /api/transfers/:id/archive` — archive a transfer, hiding it from default list results.
 - `POST /api/transfers/:id/unarchive` — unarchive a transfer, restoring it to default list results.
 
+Claim and cancel require a quoted transfer version in `If-Match` and a stable
+`Idempotency-Key`; see [claim and cancel preconditions](#claim-and-cancel-preconditions).
+
 #### Idempotency
 
 `POST /api/transfers` requires an `Idempotency-Key` header. The endpoint moves
@@ -288,6 +291,26 @@ that backs this demo a restart clears both together. That keeps them
 consistent: a surviving reservation would replay a transfer that no longer
 exists.
 
+#### Claim and cancel preconditions
+
+Read `GET /api/transfers/:id` and copy its `ETag` into `If-Match`, including
+the double quotes (for example, `"1"`). Choose a new `Idempotency-Key` for each
+new logical claim or cancel operation.
+
+Retry an uncertain operation with the **same key and original If-Match value**.
+A successful replay returns the first terminal result, even though the
+transfer's version has advanced. Do not turn a retry into a new operation by
+changing its key or version.
+
+| Status | Client action |
+|--------|---------------|
+| `400` | Supply a nonblank `Idempotency-Key` of at most 255 characters and a correctly quoted positive-integer `If-Match`. |
+| `428` | Supply the missing `If-Match` precondition. |
+| `409` for a stale version or invalid transition | Fetch the transfer again and inspect its current state before deciding on a new operation. |
+
+See [Transfer lifecycle concurrency](docs/TRANSFER_LIFECYCLE_CONCURRENCY.md)
+for version increments, replay behavior and the process-local storage boundary.
+
 ### Users
 
 - `GET /api/users` — list users.
@@ -321,24 +344,40 @@ curl "http://localhost:3000/api/health"
 # Set your token once (use a demo token for local dev, or your own via API_TOKENS)
 TOKEN="test-token-admin"
 
-# Create a transfer  (requires transfers:write)
-curl -X POST http://localhost:3000/api/transfers \
+# Choose a new key for each new logical operation; retain it for retries.
+CREATE_KEY="example-create-1"
+
+# Create a transfer  (requires transfers:write); -i also displays its ETag.
+curl -i -X POST http://localhost:3000/api/transfers \
   -H "Authorization: Bearer $TOKEN" \
+  -H "Idempotency-Key: $CREATE_KEY" \
   -H "Content-Type: application/json" \
   -d '{"senderName":"Alice","recipientName":"Bob","amount":100,"from":"USD","to":"INR"}'
 
-# Claim a transfer  (requires transfers:write)
-curl -X POST http://localhost:3000/api/transfers/<id>/claim \
-  -H "Authorization: Bearer $TOKEN"
+# Copy the created transfer's id, then read its current ETag (requires transfers:read).
+TRANSFER_ID="replace-with-created-transfer-id"
+curl -i -H "Authorization: Bearer $TOKEN" \
+  "http://localhost:3000/api/transfers/$TRANSFER_ID"
+
+# Copy that ETag verbatim, including its double quotes; "1" is only an example.
+TRANSFER_ETAG='"1"'
+CLAIM_KEY="example-claim-1"
+
+# Claim the transfer  (requires transfers:write).
+# An uncertain retry reuses this same CLAIM_KEY and original TRANSFER_ETAG.
+curl -i -X POST "http://localhost:3000/api/transfers/$TRANSFER_ID/claim" \
+  -H "Authorization: Bearer $TOKEN" \
+  -H "If-Match: $TRANSFER_ETAG" \
+  -H "Idempotency-Key: $CLAIM_KEY"
 
 # Search transfers by name and view aggregate stats  (requires transfers:read)
 curl -H "Authorization: Bearer $TOKEN" "http://localhost:3000/api/transfers?q=alice"
 curl -H "Authorization: Bearer $TOKEN" "http://localhost:3000/api/transfers/stats"
 
 # Archive and unarchive transfers  (requires transfers:write)
-curl -X POST http://localhost:3000/api/transfers/<id>/archive \
+curl -X POST "http://localhost:3000/api/transfers/$TRANSFER_ID/archive" \
   -H "Authorization: Bearer $TOKEN"
-curl -X POST http://localhost:3000/api/transfers/<id>/unarchive \
+curl -X POST "http://localhost:3000/api/transfers/$TRANSFER_ID/unarchive" \
   -H "Authorization: Bearer $TOKEN"
 
 # List only archived transfers  (requires transfers:read)

docs/TRANSFER_LIFECYCLE_CONCURRENCY.md

@@ -0,0 +1,57 @@
+# Transfer lifecycle concurrency
+
+Transfer creation already uses actor-scoped idempotency. Terminal lifecycle
+mutations (claim / cancel) add optimistic resource versions, a per-transfer
+lease, and an idempotent settlement worker.
+
+## Client contract
+
+Every single-transfer response carries an ETag containing the integer transfer
+version, for example:
+
+    ETag: "1"
+
+To claim or cancel a transfer, clients send both:
+
+    If-Match: "1"
+    Idempotency-Key: <stable operation key>
+
+The server reserves the actor/key pair before provider work. A repeated request
+with the same key replays the first terminal result. A request based on an old
+version returns HTTP 409 with expected version, actual version, and current
+status. A missing If-Match returns HTTP 428. Invalid state transitions also
+return 409.
+
+Creation starts at version 1. Every lifecycle, archive, or unarchive mutation
+increments the version.
+
+## Commit order
+
+Terminal mutations follow this order:
+
+1. reserve the actor-scoped operation key;
+2. acquire an exclusive per-transfer lifecycle lease;
+3. compare expected version and allowed transition;
+4. prepare the provider-side artifact via the settlement worker (stable
+   operation key);
+5. compare-and-set status + version (single commit path);
+6. complete the replay receipt and append the audit event;
+7. release the lease.
+
+A provider failure occurs before the local terminal commit, so the transfer
+remains pending and both the lease and reservation are released for a safe
+retry. The lease is what closes the double-settlement window when two different
+operation keys race: only one caller may prepare provider work for a given
+transfer at a time.
+
+## Storage boundary
+
+The current demo store is process-local. Version check plus mutation is a
+compare-and-set within this process. When the store moves to a database,
+preserve the contract atomically with an update constrained by transfer id and
+version, and move lifecycle idempotency, leases, and settlement receipts to the
+same durable/shared boundary so multiple workers share reservation and replay
+state.
+
+Regression coverage lives in `test/transferLifecycleConcurrency.test.js` and
+`test/transferLifecycleHttp.test.js`.

docs/validation/archive-settlement-lease-20261004.md

@@ -0,0 +1,17 @@
+# Archive and settlement lease regression
+
+Archive and unarchive now check the existing transfer-scoped lifecycle lease before changing state. Without that check, a reentrant archive can increment the resource version after provider preparation; the claim then rejects locally despite the retained provider receipt. The check is inside the actual archive mutation, so a retry of an already archived transfer remains a no-op. Other transfers and operations after lease release remain available. No provider, CAS, resource-version or idempotency policy is removed.
+
+## Executed evidence
+
+Base: `b67b7e7639bada58642a12abffa6da6ac5293c5b`. Production blob before: `da558d83b3852188adfc61f12b94ca8ccf8f837d`; after: `75ec847a3610607b171b5f716a016f57ad9dcac5`. Test blob: `c629e2049e3c86a93faee27c4ed8c1bc8c30be5c`. Runtime: `v22.23.3`.
+
+The five new cases on the original production source: 3 pass / 2 fail, exit 1. The archive and unarchive settlement interleavings both fail before the repair.
+
+After the repair, the four focused files below: 42 pass / 0 fail / 0 cancelled / 0 skipped; exit 0. Existing state-machine, worker-reload, duplicate-callback, rollback and HTTP tests were not altered.
+
+```sh
+NODE_ENV=test node --test --test-reporter=tap test/transferLifecycleConcurrency.test.js test/transferLifecycleHttp.test.js test/transferArchive.test.js test/transferArchiveSettlement.test.js
+```
+
+Actual synchronous in-memory transfer service, existing mock Stellar adapter and maintained HTTP/lifecycle tests; no live provider, durable database, deployment or full-suite claim.

src/controllers/transferController.js

@@ -20,11 +20,11 @@ const MAX_IDEMPOTENCY_KEY_LENGTH = 255;
  * @returns {string}
  * @throws {ApiError} 400 when the header is missing or unusable.
  */
-function requireIdempotencyKey(req) {
+function requireIdempotencyKey(req, purpose = 'create a transfer') {
   const raw = req.get('Idempotency-Key');
   if (typeof raw !== 'string' || raw.trim() === '') {
     throw ApiError.badRequest(
-      'Idempotency-Key header is required to create a transfer'
+      `Idempotency-Key header is required to ${purpose}`
     );
   }
   const key = raw.trim();
@@ -36,6 +36,41 @@ function requireIdempotencyKey(req) {
   return key;
 }
 
+/**
+ * Parse the strong ETag version used as the optimistic precondition for
+ * terminal transfer mutations.
+ * @param {import('express').Request} req
+ * @returns {number}
+ */
+function requireTransferVersion(req) {
+  const raw = req.get('If-Match');
+  if (typeof raw !== 'string' || raw.trim() === '') {
+    throw new ApiError(
+      428,
+      'If-Match header is required for transfer lifecycle mutations'
+    );
+  }
+
+  const match = /^"([1-9][0-9]*)"$/.exec(raw.trim());
+  if (!match) {
+    throw ApiError.badRequest(
+      'If-Match must contain the quoted transfer version, for example "1"'
+    );
+  }
+  return Number(match[1]);
+}
+
+/**
+ * Return a transfer with its current version as a strong ETag.
+ * @param {import('express').Response} res
+ * @param {object} transfer
+ * @param {number} [status]
+ */
+function sendTransfer(res, transfer, status = 200) {
+  res.set('ETag', `"${transfer.version}"`);
+  res.status(status).json(transfer);
+}
+
 /**
  * Transfer controllers.
  */
@@ -70,7 +105,7 @@ function createTransfer(req, res) {
   // did. Replaying the stored result means replaying all of it; downgrading the
   // status would make a successful retry look different from the response it is
   // standing in for.
-  res.status(201).json(transfer);
+  sendTransfer(res, transfer, 201);
 }
 
 /**
@@ -127,25 +162,37 @@ function getStats(req, res) {
  */
 function getTransfer(req, res) {
   const transfer = transferService.getTransferOrThrow(req.params.id);
-  res.json(transfer);
+  sendTransfer(res, transfer);
 }
 
 /**
  * POST /api/transfers/:id/claim
  * Mark a transfer as claimed by the recipient.
  */
 function claimTransfer(req, res) {
-  const transfer = transferService.claimTransfer(req.params.id, req.id);
-  res.json(transfer);
+  const expectedVersion = requireTransferVersion(req);
+  const key = requireIdempotencyKey(req, 'claim a transfer');
+  const transfer = transferService.claimTransfer(req.params.id, req.id, {
+    actor: req.token,
+    key,
+    expectedVersion,
+  });
+  sendTransfer(res, transfer);
 }
 
 /**
  * POST /api/transfers/:id/cancel
  * Cancel a pending transfer.
  */
 function cancelTransfer(req, res) {
-  const transfer = transferService.cancelTransfer(req.params.id, req.id);
-  res.json(transfer);
+  const expectedVersion = requireTransferVersion(req);
+  const key = requireIdempotencyKey(req, 'cancel a transfer');
+  const transfer = transferService.cancelTransfer(req.params.id, req.id, {
+    actor: req.token,
+    key,
+    expectedVersion,
+  });
+  sendTransfer(res, transfer);
 }
 
 /**
@@ -154,7 +201,7 @@ function cancelTransfer(req, res) {
  */
 function archiveTransfer(req, res) {
   const transfer = transferService.archiveTransfer(req.params.id);
-  res.json(transfer);
+  sendTransfer(res, transfer);
 }
 
 /**
@@ -163,7 +210,7 @@ function archiveTransfer(req, res) {
  */
 function unarchiveTransfer(req, res) {
   const transfer = transferService.unarchiveTransfer(req.params.id);
-  res.json(transfer);
+  sendTransfer(res, transfer);
 }
 
 module.exports = {

src/services/settlementWorker.js

@@ -0,0 +1,66 @@
+'use strict';
+
+const { store } = require('../store');
+const stellarService = require('./stellarService');
+const ApiError = require('../utils/ApiError');
+
+/**
+ * Idempotent settlement worker for terminal claim operations.
+ *
+ * Provider work is keyed by a stable operation id. Retries — including after a
+ * worker-module reload that shares the same process store — return the first
+ * settlement receipt instead of creating a second claimable balance.
+ *
+ * Receipts live in the shared store so their lifetime matches transfers: a
+ * restart that clears transfers also clears receipts, keeping the two
+ * consistent. When the store becomes durable, keep receipts on the same
+ * boundary as the transfer row that commits the claim.
+ */
+
+/**
+ * Settle a claim against the payment provider exactly once for `operationId`.
+ * @param {string} operationId - stable id spanning retries of one claim attempt
+ * @returns {{ operationId: string, claimableBalanceId: string, settledAt: string }}
+ */
+function settleClaim(operationId) {
+  if (typeof operationId !== 'string' || operationId.trim() === '') {
+    throw new Error('settlementWorker.settleClaim requires a non-empty operationId');
+  }
+
+  const existing = store.settlementReceipts.get(operationId);
+  if (existing) {
+    return existing;
+  }
+
+  const claimableBalanceId = stellarService.createClaimableBalanceId(operationId);
+  // A fulfilled adapter call is not a completed settlement without its ID.
+  // Reject before caching so the lifecycle stays pending and the same stable
+  // provider operation can be retried after recovery. Do not coerce or expose
+  // an unexpected provider payload.
+  if (typeof claimableBalanceId !== 'string' || claimableBalanceId.trim() === '') {
+    throw ApiError.serviceUnavailable('Payment provider returned an invalid settlement receipt', {
+      code: 'SETTLEMENT_RECEIPT_INVALID',
+    });
+  }
+  const receipt = {
+    operationId,
+    claimableBalanceId,
+    settledAt: new Date().toISOString(),
+  };
+  store.settlementReceipts.set(operationId, receipt);
+  return receipt;
+}
+
+/**
+ * Look up a prior settlement receipt without contacting the provider.
+ * @param {string} operationId
+ * @returns {object|null}
+ */
+function getReceipt(operationId) {
+  return store.settlementReceipts.get(operationId) || null;
+}
+
+module.exports = {
+  settleClaim,
+  getReceipt,
+};