diff --git a/.env.example b/.env.example index dcba576..a217342 100644 --- a/.env.example +++ b/.env.example @@ -20,6 +20,21 @@ CORS_ORIGIN=* # Rate limiting (per client IP, applied to /api) RATE_LIMIT_WINDOW_MS=60000 RATE_LIMIT_MAX=100 +RATE_LIMIT_MAX_KEYS=10000 + +# Honour X-Forwarded-For only behind a trusted reverse proxy +TRUST_PROXY=false + +# Per-actor mutation budgets (see docs/ABUSE_CONTROLS.md) +MUTATION_RATE_LIMIT_MAX_KEYS=10000 +MUTATION_RATE_LIMIT_TRANSFERS_WINDOW_MS=60000 +MUTATION_RATE_LIMIT_TRANSFERS_MAX=30 +MUTATION_RATE_LIMIT_USERS_WINDOW_MS=60000 +MUTATION_RATE_LIMIT_USERS_MAX=20 +MUTATION_RATE_LIMIT_QUOTE_WINDOW_MS=60000 +MUTATION_RATE_LIMIT_QUOTE_MAX=60 +MUTATION_RATE_LIMIT_ADMIN_WINDOW_MS=60000 +MUTATION_RATE_LIMIT_ADMIN_MAX=30 # Error tracking ERROR_TRACKING_ENABLED=true diff --git a/CHANGELOG.md b/CHANGELOG.md index a8e4e0a..7e7cd77 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,13 @@ When preparing a new release: ### Added +- Route-family mutation rate limits and sanitized correlation IDs for abuse + control on transfer writes, user writes, quote, and admin diagnostics. + Actor keys are truncated token fingerprints (never raw secrets); limiter + tables are bounded by `RATE_LIMIT_MAX_KEYS` / + `MUTATION_RATE_LIMIT_MAX_KEYS`. `TRUST_PROXY` gates `X-Forwarded-For`. + See `docs/ABUSE_CONTROLS.md`. + - Cursor pagination for `GET /api/transfers` and `GET /api/audit`. Pass `?cursor=` (with optional `?order=asc|desc`) to page by an indexed position instead of a row offset; responses carry a `pageInfo` block with diff --git a/README.md b/README.md index 4791414..8de6f7b 100644 --- a/README.md +++ b/README.md @@ -37,6 +37,9 @@ The application is configured using environment variables (typically defined in | `CORS_ORIGIN` | Allowed CORS origin | `*` | | `RATE_LIMIT_WINDOW_MS` | Time window for rate limiting (ms) | `60000` | | `RATE_LIMIT_MAX` | Max requests per window | `100` | +| `RATE_LIMIT_MAX_KEYS` | Max distinct client identities retained by the global limiter | `10000` | +| `TRUST_PROXY` | Honour `X-Forwarded-For` when resolving client IP (`true`/`1`) | `false` | +| `MUTATION_RATE_LIMIT_*` | Per-actor budgets for transfer/user/quote/admin mutations (see `docs/ABUSE_CONTROLS.md`) | see docs | | `BODY_LIMIT` | Max JSON request body size | `100kb` | | `REQUEST_TIMEOUT_MS` | Request timeout before returning 503 (ms) | `15000` | | `DB_POOL_MIN` | Minimum database connections in pool | `2` | diff --git a/docs/ABUSE_CONTROLS.md b/docs/ABUSE_CONTROLS.md new file mode 100644 index 0000000..2b803a2 --- /dev/null +++ b/docs/ABUSE_CONTROLS.md @@ -0,0 +1,237 @@ +# Abuse controls and correlation IDs + +RemitFlow Backend bounds high-volume retries and automated abuse on mutation +and provider-adjacent routes, and propagates a safe correlation id on every +request so incidents can be traced without leaking account secrets. + +## Global limit + +Every `/api/*` request is subject to an IP-keyed fixed-window limit: + +| Setting | Env | Default | +|---|---|---| +| Window | `RATE_LIMIT_WINDOW_MS` | `60000` | +| Max requests | `RATE_LIMIT_MAX` | `100` | +| Max tracked keys | `RATE_LIMIT_MAX_KEYS` | `10000` | + +Responses carry `X-RateLimit-Limit`, `X-RateLimit-Remaining`, +`X-RateLimit-Reset`, and `X-RateLimit-Policy`. Exhausted budgets answer +`429` with `Retry-After` and a JSON error that includes `requestId`. + +## Mutation / route-family limits + +Authenticated write paths use a **stricter actor-keyed** budget on top of the +global limit. The actor key is a truncated SHA-256 fingerprint of the API +token (or admin key) — never the raw secret. Public quote uses the client IP. + +| Family | Routes | Env (window / max) | Default max / min | +|---|---|---|---| +| `transfers` | `POST /api/transfers`, claim, cancel, archive, unarchive | `MUTATION_RATE_LIMIT_TRANSFERS_*` | 30 / 60s | +| `users` | `POST /api/users` | `MUTATION_RATE_LIMIT_USERS_*` | 20 / 60s | +| `quote` | `GET /api/quote` | `MUTATION_RATE_LIMIT_QUOTE_*` | 60 / 60s | +| `admin` | `GET /api/admin/diagnostics` | `MUTATION_RATE_LIMIT_ADMIN_*` | 30 / 60s | + +Shared cap: `MUTATION_RATE_LIMIT_MAX_KEYS` (default `10000`). + +Actors are isolated: one token burning its transfer budget does not exhaust +another token's budget. Limiter tables are bounded. Expired windows are pruned +first; when every key is still live, a new identity receives 429 with +`Retry-After` until a slot expires. This preserves active budgets under an +identity flood, at the cost of delaying new identities when the table is full. + +## Proxy trust + +`TRUST_PROXY=true` uses Express `trust proxy` for one proxy hop. Rate +limiting reads Express's resolved `req.ip`, so an attacker-controlled left-most +`X-Forwarded-For` value cannot rotate the identity when the trusted proxy +appends the connecting address. Enable this only when the deployment has +exactly one trusted reverse proxy; otherwise leave it off until the app's +trust proxy setting matches the deployment topology. + +## Correlation IDs + +On every request, the [request-ID middleware](../src/middleware/requestId.js): + +- Trims leading and trailing whitespace from each candidate header value. +- Accepts a nonempty value of at most 128 characters after trimming, matching + `^[A-Za-z0-9._:-]+$`. +- Rejects a candidate containing the Bearer token or `X-Admin-Token` supplied + on the same request, even when the candidate otherwise matches the format. + This runs before authentication and does not change authorization decisions. +- Uses a valid, credential-independent `X-Request-Id` first, otherwise a valid, + credential-independent `X-Correlation-Id`. If neither is acceptable, + generates a fresh UUID. +- Echoes the selected value on both `X-Request-Id` and `X-Correlation-Id`. +- Exposes it as `req.id` / `req.correlationId` and on error envelopes as + `error.requestId`. + +Values outside this format or containing a credential presented with the same +request are ignored; accepted identifiers are not hashed or redacted. The +credential exclusion only recognizes those presented Bearer/admin tokens. It +does not identify unrelated credentials, account identifiers or arbitrary +personal data, so a syntactically valid ID is not proof that its content is safe. + +Supply a fresh opaque ID, such as a UUID, and keep credentials, account details +and other personal data out of both headers. Omit both headers when a +server-generated ID is sufficient. Accepted IDs are echoed and written by the +[request logger](../src/middleware/requestLogger.js), so treat them as values +that can appear in response headers, error envelopes and logs. + +## Test behaviour + +When `NODE_ENV=test`, rate limiters are no-ops unless +`ENABLE_RATE_LIMIT_IN_TEST=1` (or a limiter is constructed with +`forceInTest: true`). This keeps the functional suite independent of the +abuse budget while still allowing focused regression coverage. + +## Reproduce saturated-table HTTP load + +With this checkout's normal dependencies installed, run: + +```sh +node scripts/benchmark-rate-limit.cjs rate-limit-result.json > /dev/null +``` + +Use a new result filename. The script starts the real application on an ephemeral +loopback port, enables the limiter explicitly in test mode, and uses one trusted +proxy hop to generate distinct local identities. It admits 10,000 identities, +then measures three batches of 2,000 rejected newcomers at 32 concurrent +connections. Every measured response must be 429 with the global policy and +matching correlation ID; an exhausted original identity must remain blocked. +The script closes its server and connections when finished. + +JSON output retains each elapsed/CPU sample, Node and Express versions, and +SHA-256 hashes of the application and limiter source. CPU includes both the +HTTP client and server in the same process. Standard application logs go to +stdout, so keep the same redirection for both versions being compared. + +To compare another checkout with its dependencies already installed, pass its +path as the second argument. Alternate original/repaired/original/repaired runs +using the same script and distinct output filenames; report medians and ranges +because load on the host can vary. The workload uses generated local traffic and +does not measure deployed throughput or external provider behavior. Repeated +rejection does not traverse the table. Expiry cleanup uses the deadline index +described below. + +## Staggered-window expiry + +A binary min-heap now orders deadlines independently of insertion order. It has +one node per tracked identity, and both the map and heap remain bounded by +`maxKeys`. Repeated hits reuse the same node; renewal first removes the expired +generation. Reset clears both structures. This ordering also preserves the +existing behavior when the wall clock moves backward. + +Admitting a new identity takes O(log n) heap work. Removing k expired identities +takes O(k log n), without scanning the n surviving identities. The additional +index costs O(n) memory. A maximum deadline also tracks when the entire table +has expired; that case clears both collections without popping individual heap +nodes. If any budget remains live, the existing heap pruning still applies and +expiring many older identities can do more work than a linear sweep. Both paths +preserve active budgets, capacity rejection and rounded-up `Retry-After`. + +### Measured component result, October 4, 2026 + +The exact production middleware was loaded with its real `ApiError` and +`clientIdentity` modules on Node 24.19.0. A controlled clock admitted 10,000 +identities with staggered windows, then 1,000 rolling newcomers while preserving +10,000 tracked identities. All 11,000 admissions and the final exhausted +survivor's 429 response matched the original implementation. + +| Measurement | Original | Updated | +|---|---:|---:| +| Full-map entry visits during the rolling workload | 10,000,000 | 0 | +| Median rolling time, three alternating pairs | 224.325 ms | 2.818 ms | +| Rolling time range | 183.074–474.427 ms | 2.466–4.710 ms | + +The timing ratio was 79.6× in this component workload. Map iteration counts do +not count heap operations or ordinary map lookups. Clock, request and response +objects were controlled; this does not measure HTTP service or fleet throughput. +A separate deterministic comparison covered 20,000 requests and 629 resets at +capacities 1, 2, 7, 31 and 256. Complete headers, errors, decisions and tracked +sizes matched, including backward-clock admissions and renewals. + +Original source: commit `f501ce5f4fd6f6f96c82fb9fffed4dc58f3def8e`, limiter +blob `6748ce8105628e781cccea40c368d2bb3a830b17`. Updated limiter blob: +`84c97f7b183c1c8e96be0695e5f332f9c2e3acca`. +[Raw samples and response digests](validation/rate-limit-expiry-20261004.json) +retain every measured pair. The existing abuse-control suite retains all prior +cases and adds one regression for staggered expiry. + +To reproduce the component comparison from a checkout containing this change: + +```sh +benchmark_dir=$(mktemp -d) +mkdir "$benchmark_dir/original" "$benchmark_dir/final" +git archive f501ce5f4fd6f6f96c82fb9fffed4dc58f3def8e | tar -x -C "$benchmark_dir/original" +git archive HEAD | tar -x -C "$benchmark_dir/final" +cp scripts/benchmark-rate-limit-expiry.cjs "$benchmark_dir/replay.cjs" +node "$benchmark_dir/replay.cjs" +``` + +The replay writes `results.json` inside the fresh benchmark directory. It uses +only Node built-ins and the archived production modules; it installs no packages +and makes no network requests. Host load can change the timing samples. + +### Maintained application checks + +On the updated limiter/test blobs above, `node --test test/abuseControls.test.js` +reported 17 passed and 0 failed; `npm test` reported 273 passed and 0 failed, +with no skipped or cancelled tests. These runs used Node 24.19.0 and the unchanged +lockfile `be3ad84b812121d2ec4f47541df3a55816bb210c`. The focused suite includes real +Express loopback requests for proxy identity, correlation, parser failures and +429 responses. The complete 178-file parent source was checked against its Git +blob identities before composing the two changed source/test files. + +Dependencies came from the matching-lock artifact of the already-completed +[archive acceptance job](https://github.com/woahwhattheheck/RemitFlow-Backend/actions/runs/37191775463). +That job's 46 archive passes concern PR #144; the 17/273 results above are separate +local executions of this PR #138 composition. Sharing dependencies adds no claim +that the hosted archive job executed the limiter patch. + +### Whole-table expiry continuation, October 4, 2026 + +The maximum deadline is independent of insertion order. It changes only when +an identity is admitted, and resets when the limiter or whole table is cleared. +A clock rollback can create an earlier new deadline without causing a later +live budget to be removed. Clearing waits until the latest deadline, inclusive. +JavaScript collection clearing and later garbage collection remain runtime +costs; this is not a constant-time or process-throughput guarantee. + +The following Node 24.19.0 measurements compare the exact preceding middleware +at `ebda2cb811d1c3ad35c1fec8af46cf83d45df8b3` with this continuation. Each +workload fills 10,000 identities, then measures the first admission once every +budget has expired. Five alternating before/after pairs contain 20 rounds each. + +| First-admission latency | Previous median | Updated median | +|---|---:|---:| +| All identities share one deadline | 0.6200 ms | 0.01655 ms | +| Staggered deadlines, all expired | 1.8185 ms | 0.01114 ms | + +Both versions return identical headers, errors and retained sizes, including +the next request's 429. Instrumented runs remove 200,000 expired-entry `Map` +deletions over 20 rounds; the new identity's normal insertion remains. These +are actual production middleware modules with a controlled clock and request +objects, not an HTTP or external-provider benchmark. + +Admission-fill timings are also retained, not folded into the first-admission +claim. Across these pairs their medians changed from 174.5 to 227.1 ms for the +same-deadline workload and 164.3 to 181.8 ms for staggered deadlines. Allocation, +garbage collection and host load affect those measurements; the result +establishes less work on the expiry-triggering request, not higher total +throughput. Raw samples and source hashes are in +[the batch-expiry result](validation/rate-limit-batch-expiry-20261004.json). + +Four maintained expiry/renewal regressions pass, including one new boundary, +clock-rollback and reset case. The exact command is: + +```sh +node --test --test-name-pattern='expiry|renewal|clock rollback' test/abuseControls.test.js +``` + +The focused run uses the existing matching-lock dependency artifact referenced +above, without dependency changes. It does not rerun the full application suite. +Reproduce the component comparison with two checkouts and a new output filename: + +```sh +node scripts/benchmark-rate-limit-batch-expiry.cjs BEFORE_CHECKOUT AFTER_CHECKOUT NEW_RESULT.json +``` diff --git a/docs/validation/capacity-reset-20261004.md b/docs/validation/capacity-reset-20261004.md new file mode 100644 index 0000000..31ccbbc --- /dev/null +++ b/docs/validation/capacity-reset-20261004.md @@ -0,0 +1,56 @@ +# Capacity rejection reset header + +A full limiter table already returned the correct policy, limit, remaining +budget and Retry-After. It omitted X-RateLimit-Reset. When global and mutation +limiters ran in sequence, that omission retained the global policy's deadline +on a mutation-policy 429 response. A standalone limiter omitted the header. + +The capacity response now sets X-RateLimit-Reset to the earliest retained +expiry, rounded up to epoch seconds, using the existing expiry index. +This is two added production lines. Budget admission, capacity, pruning, +Retry-After, credential handling and existing timing results are unchanged. + +## Observed HTTP behavior + +The same three maintained cases ran once against the preceding production +source and once against the repair on Node 24.19.0. Each phase sent 14 actual +loopback HTTP requests. The preceding source failed all three header assertions; +the repaired source passed all three cases with no skipped or cancelled cases. + +| Case | Previous reset header | Repaired reset header | +| --- | --- | --- | +| Global window starts at 0 s; mutation window at 30 s; capacity rejection at 45 s | 1700000060, 30 seconds early | 1700000090 | +| Standalone limiter; earliest expiry at epoch 1700000061.250 | Absent | 1700000062 | +| First slot expires and is reused; another live slot next expires at 70 s | 1700000122, 52 seconds late | 1700000070 | + +The cases also check status, policy, limit, remaining budget, Retry-After and +error details. Requests immediately before the relevant deadline remain +rejected, while a request at the deadline can take the expired slot. Partial +expiry advances the advertised deadline to the next surviving entry. + +## Reproduce + +From the repository root: + + node --test test/rateLimitCapacity.test.js + +The regression uses only Node built-ins and the actual rateLimit, ApiError +and clientIdentity production modules. The HTTP server supplies the response +adapter that the middleware normally receives from Express, a controlled clock +and a synthetic authenticated-token assignment. It does not substitute limiter, +expiry, actor-key or error-class logic. No dependencies were installed for this +execution, and no provider requests or complete application suite were run. +The result establishes the response-header repair, not authentication, +full Express application integration or a throughput benchmark. + +## Executed source + +- Preceding commit: 752ec30bbc36825e75c78e5fba8e921f19d15a70. +- Preceding rateLimit.js Git blob: 91979defd7337634903af5219926112b635cbb1a. +- Repaired rateLimit.js Git blob: a51b607ff3cd526748da98e26c3c1f9db337d22e. +- Unchanged ApiError.js Git blob: 6b586f60d991ab4e5174e0af4e0d94cf1e3ceb58. +- Unchanged clientIdentity.js Git blob: 68c51d25194ef031b6b5a329b982a24c25662271. +- Identical regression Git blob in both phases: efa7ac2553db731f743fbbdd601344483e8d3542. + +Existing expiry-index and batch-expiry repairs and their historical evidence +remain intact. The original PR and contribution are retained. diff --git a/docs/validation/rate-limit-batch-expiry-20261004.json b/docs/validation/rate-limit-batch-expiry-20261004.json new file mode 100644 index 0000000..65ec89f --- /dev/null +++ b/docs/validation/rate-limit-batch-expiry-20261004.json @@ -0,0 +1,700 @@ +{ + "node": "v24.19.0", + "before_blob": "84c97f7b183c1c8e96be0695e5f332f9c2e3acca", + "after_blob": "91979defd7337634903af5219926112b635cbb1a", + "workloads": { + "same_deadline": { + "summary": { + "before": { + "expiry_median_ms": 0.6200084999999262, + "expiry_range_ms": [ + 0.40938699999969685, + 19.459208999999873 + ], + "admission_median_ms": 174.46474700000022, + "expiry_deletes": 200020, + "digest": "a375ccf4fb6ea46b9cb0570f8e503ba1b98a089992472e5c315fd6309ebe3278" + }, + "after": { + "expiry_median_ms": 0.016549499999996442, + "expiry_range_ms": [ + 0.006919999999809079, + 1.0525450000000092 + ], + "admission_median_ms": 227.07911400000046, + "expiry_deletes": 20, + "digest": "a375ccf4fb6ea46b9cb0570f8e503ba1b98a089992472e5c315fd6309ebe3278" + } + }, + "pairs": [ + { + "order": [ + "before", + "after" + ], + "before": { + "capacity": 10000, + "rounds": 20, + "expiry_ms": [ + 2.177929000000063, + 0.7893679999999677, + 0.5879150000000664, + 0.6544250000000602, + 0.8117430000000923, + 2.5163690000000543, + 0.8245220000000018, + 0.9922740000000658, + 0.7048909999999751, + 1.4297330000000557, + 0.589747999999986, + 0.6003740000001017, + 0.5958069999999225, + 0.5678249999999707, + 0.5013149999999769, + 0.6684860000000299, + 0.6071739999999863, + 0.7119320000000471, + 0.8518329999999423, + 0.9040209999999433 + ], + "admission_ms": 182.96299899999974, + "expiry_deletes": null, + "digest": "a375ccf4fb6ea46b9cb0570f8e503ba1b98a089992472e5c315fd6309ebe3278" + }, + "after": { + "capacity": 10000, + "rounds": 20, + "expiry_ms": [ + 0.06164300000000367, + 0.010476000000039676, + 0.013830999999981941, + 0.016483999999991283, + 0.036324000000035994, + 0.01960000000008222, + 0.016443999999978587, + 0.018538000000035026, + 0.016393999999991138, + 0.015071999999918262, + 0.012830000000008113, + 0.016143999999940206, + 0.03269000000000233, + 0.013970999999969536, + 0.013720000000034815, + 0.01332999999999629, + 0.019709000000034393, + 0.022484000000076776, + 0.021152000000029147, + 0.015563000000042848 + ], + "admission_ms": 227.07911400000046, + "expiry_deletes": null, + "digest": "a375ccf4fb6ea46b9cb0570f8e503ba1b98a089992472e5c315fd6309ebe3278" + } + }, + { + "order": [ + "after", + "before" + ], + "after": { + "capacity": 10000, + "rounds": 20, + "expiry_ms": [ + 0.016955000000052678, + 0.016715000000090185, + 0.01833699999997407, + 0.020721999999977925, + 0.020881999999915024, + 0.01874800000018695, + 0.017595999999912237, + 0.01845699999989847, + 0.018488000000161264, + 0.019248999999945227, + 0.01706600000011349, + 0.017005000000153814, + 0.016985999999860724, + 0.020410999999967316, + 0.02152299999988827, + 1.0525450000000092, + 0.4182700000001205, + 0.01798699999994824, + 0.2414840000001277, + 0.6991330000000744 + ], + "admission_ms": 366.2796639999998, + "expiry_deletes": null, + "digest": "a375ccf4fb6ea46b9cb0570f8e503ba1b98a089992472e5c315fd6309ebe3278" + }, + "before": { + "capacity": 10000, + "rounds": 20, + "expiry_ms": [ + 1.326908000000003, + 1.9826649999999972, + 0.5698879999999917, + 0.4939740000002075, + 0.49842100000000755, + 9.843151000000034, + 0.5863520000000335, + 0.5090069999998832, + 0.43922199999997247, + 0.654835999999932, + 0.6222369999998136, + 0.5774189999999635, + 0.7262229999998908, + 0.5899480000000494, + 1.321358999999802, + 1.0809179999998832, + 0.4903580000000147, + 0.9685989999998128, + 0.6371090000000095, + 0.741767000000209 + ], + "admission_ms": 225.1981990000004, + "expiry_deletes": null, + "digest": "a375ccf4fb6ea46b9cb0570f8e503ba1b98a089992472e5c315fd6309ebe3278" + } + }, + { + "order": [ + "before", + "after" + ], + "before": { + "capacity": 10000, + "rounds": 20, + "expiry_ms": [ + 0.6981719999998859, + 0.4215849999998227, + 19.459208999999873, + 0.7548670000001039, + 0.5875040000000809, + 0.5315610000000106, + 0.8101490000001377, + 0.5238289999999779, + 0.7386119999998755, + 0.6426480000000083, + 0.8059940000000552, + 0.6935439999999744, + 0.6734840000001441, + 0.807365000000118, + 0.7136440000001585, + 0.7770399999999427, + 0.6345049999999901, + 0.7698590000002241, + 0.7212960000001658, + 0.7062140000000454 + ], + "admission_ms": 174.46474700000022, + "expiry_deletes": null, + "digest": "a375ccf4fb6ea46b9cb0570f8e503ba1b98a089992472e5c315fd6309ebe3278" + }, + "after": { + "capacity": 10000, + "rounds": 20, + "expiry_ms": [ + 0.07973999999990156, + 0.016875000000027285, + 0.01559399999996458, + 0.01318000000014763, + 0.01350999999999658, + 0.020440000000007785, + 0.014482000000043627, + 0.01577299999985371, + 0.017046000000163986, + 0.012950000000046202, + 0.01558300000010604, + 0.015754000000015367, + 0.0166150000000016, + 0.01400100000000748, + 0.02066200000012941, + 0.015383000000156244, + 0.01928900000007161, + 0.013641000000006898, + 0.0200009999998656, + 0.019400000000132422 + ], + "admission_ms": 229.4123109999996, + "expiry_deletes": null, + "digest": "a375ccf4fb6ea46b9cb0570f8e503ba1b98a089992472e5c315fd6309ebe3278" + } + }, + { + "order": [ + "after", + "before" + ], + "after": { + "capacity": 10000, + "rounds": 20, + "expiry_ms": [ + 0.015503000000080647, + 0.012108000000353059, + 0.01542299999982788, + 0.009665000000040891, + 0.011577000000215776, + 0.011286999999811087, + 0.013198999999985972, + 0.01200799999969604, + 0.012509000000136439, + 0.011907999999948515, + 0.011066999999911786, + 0.016864999999597785, + 0.010926000000381464, + 0.010275000000092405, + 0.008762999999817112, + 0.007771999999931722, + 0.01392100000020946, + 0.009152999999969325, + 0.007481000000097993, + 0.008333000000220636 + ], + "admission_ms": 180.58825699999943, + "expiry_deletes": null, + "digest": "a375ccf4fb6ea46b9cb0570f8e503ba1b98a089992472e5c315fd6309ebe3278" + }, + "before": { + "capacity": 10000, + "rounds": 20, + "expiry_ms": [ + 0.537579000000278, + 0.5717810000001009, + 0.6157670000002327, + 10.680802000000313, + 6.511794999999893, + 0.7787520000001678, + 0.7594140000001062, + 0.6177800000000389, + 0.4764380000001438, + 0.5450009999999565, + 0.4558560000000398, + 0.4637980000002244, + 0.46911699999964185, + 0.4686259999998583, + 0.4598820000001069, + 0.4685660000000098, + 0.48051399999985733, + 0.4899669999999787, + 0.4678439999997863, + 0.46624199999996563 + ], + "admission_ms": 141.44744599999785, + "expiry_deletes": null, + "digest": "a375ccf4fb6ea46b9cb0570f8e503ba1b98a089992472e5c315fd6309ebe3278" + } + }, + { + "order": [ + "before", + "after" + ], + "before": { + "capacity": 10000, + "rounds": 20, + "expiry_ms": [ + 0.4238080000000082, + 0.4628170000000864, + 0.48319699999956356, + 0.4435880000000907, + 0.53330300000016, + 0.40938699999969685, + 0.4681449999998222, + 0.4100579999999354, + 0.469677000000047, + 0.4280940000003284, + 0.42032300000028044, + 0.5207839999998214, + 0.4107790000002751, + 0.8945269999999255, + 0.4905389999998988, + 0.6588320000000749, + 0.6481149999999616, + 0.6947359999999208, + 0.768177000000378, + 0.6731829999998808 + ], + "admission_ms": 139.35454299999992, + "expiry_deletes": null, + "digest": "a375ccf4fb6ea46b9cb0570f8e503ba1b98a089992472e5c315fd6309ebe3278" + }, + "after": { + "capacity": 10000, + "rounds": 20, + "expiry_ms": [ + 0.05711599999995087, + 0.017245999999886408, + 0.014080999999805499, + 0.019499999999879947, + 0.018707999999605818, + 0.017986999999720865, + 0.0188479999997071, + 0.019059000000197557, + 0.020010000000183936, + 0.01767599999993763, + 0.013781000000108179, + 0.012078000000201428, + 0.015002999999978783, + 0.009434000000055676, + 0.022492999999940366, + 0.017256000000088534, + 0.01731599999993705, + 0.012779000000136875, + 0.01192799999989802, + 0.006919999999809079 + ], + "admission_ms": 192.45708099999956, + "expiry_deletes": null, + "digest": "a375ccf4fb6ea46b9cb0570f8e503ba1b98a089992472e5c315fd6309ebe3278" + } + } + ] + }, + "staggered_all_expired": { + "summary": { + "before": { + "expiry_median_ms": 1.8185034999999061, + "expiry_range_ms": [ + 1.5505249999996522, + 4.854990000000271 + ], + "admission_median_ms": 164.27379099999962, + "expiry_deletes": 200020, + "digest": "a4b2b27b94a97c7d2007f178a38f5698c2470ac858681a3e65e01f36be9e32fb" + }, + "after": { + "expiry_median_ms": 0.011136500000247906, + "expiry_range_ms": [ + 0.003885999999511114, + 0.5745550000001458 + ], + "admission_median_ms": 181.82916499999965, + "expiry_deletes": 20, + "digest": "a4b2b27b94a97c7d2007f178a38f5698c2470ac858681a3e65e01f36be9e32fb" + } + }, + "pairs": [ + { + "order": [ + "before", + "after" + ], + "before": { + "capacity": 10000, + "rounds": 20, + "expiry_ms": [ + 2.236978000000363, + 2.1456610000000182, + 1.9874219999996967, + 2.02628999999979, + 2.0268620000001647, + 2.063716999999997, + 2.1914990000000216, + 1.9623449999999139, + 2.228925999999774, + 1.6204000000002452, + 2.2560570000000553, + 2.417609999999968, + 2.0855599999999868, + 2.2322110000000066, + 2.126922000000377, + 1.9760150000001886, + 2.41633899999988, + 1.9431559999998171, + 2.001973999999791, + 1.5506649999997535 + ], + "admission_ms": 189.25910799999883, + "expiry_deletes": null, + "digest": "a4b2b27b94a97c7d2007f178a38f5698c2470ac858681a3e65e01f36be9e32fb" + }, + "after": { + "capacity": 10000, + "rounds": 20, + "expiry_ms": [ + 0.009794999999940046, + 0.012378999999782536, + 0.010815999999977066, + 0.009855000000243308, + 0.5745550000001458, + 0.014631999999892287, + 0.01566400000001522, + 0.01534300000002986, + 0.01141700000016499, + 0.012939000000187661, + 0.011096000000179629, + 0.01680499999974927, + 0.01581400000031863, + 0.015452999999979511, + 0.015852999999879103, + 0.014280999999755295, + 0.011367000000063854, + 0.010014999999839347, + 0.010354999999890424, + 0.011316999999962718 + ], + "admission_ms": 215.816385000001, + "expiry_deletes": null, + "digest": "a4b2b27b94a97c7d2007f178a38f5698c2470ac858681a3e65e01f36be9e32fb" + } + }, + { + "order": [ + "after", + "before" + ], + "after": { + "capacity": 10000, + "rounds": 20, + "expiry_ms": [ + 0.016384999999900174, + 0.01960000000008222, + 0.019037999999909516, + 0.01887899999974252, + 0.01981999999998152, + 0.01756699999987177, + 0.015443000000232132, + 0.01636400000006688, + 0.015653999999813095, + 0.014582000000245898, + 0.014512000000195258, + 0.01672499999995125, + 0.018317999999908352, + 0.015292999999928725, + 0.03795700000000579, + 0.013259999999718275, + 0.015533000000232278, + 0.014101000000209751, + 0.014572000000043772, + 0.01737600000024031 + ], + "admission_ms": 181.82916499999965, + "expiry_deletes": null, + "digest": "a4b2b27b94a97c7d2007f178a38f5698c2470ac858681a3e65e01f36be9e32fb" + }, + "before": { + "capacity": 10000, + "rounds": 20, + "expiry_ms": [ + 2.396447999999964, + 4.854990000000271, + 1.7521579999997812, + 2.0000609999997323, + 3.3307949999998527, + 2.4195819999999912, + 2.569488999999976, + 2.2450410000001284, + 2.4612660000002506, + 2.6900799999998526, + 2.5022219999996196, + 1.6974759999998241, + 1.8155639999999948, + 1.7037850000001526, + 1.5505249999996522, + 1.931988999999703, + 1.624286000000211, + 1.8793000000000575, + 2.205891000000065, + 1.719249000000218 + ], + "admission_ms": 189.27236699999958, + "expiry_deletes": null, + "digest": "a4b2b27b94a97c7d2007f178a38f5698c2470ac858681a3e65e01f36be9e32fb" + } + }, + { + "order": [ + "before", + "after" + ], + "before": { + "capacity": 10000, + "rounds": 20, + "expiry_ms": [ + 1.8508460000002742, + 1.7536199999999553, + 1.6319770000000062, + 1.7375059999999394, + 1.788202000000183, + 1.6707150000002002, + 3.0887410000000273, + 1.664466000000175, + 1.8307569999997213, + 1.6160839999993186, + 1.8214429999998174, + 1.8729700000003504, + 1.6337389999998777, + 1.8311770000000251, + 1.9417240000002494, + 1.696774000000005, + 1.8022439999995186, + 1.6916570000003048, + 1.658726999999999, + 1.6060189999998329 + ], + "admission_ms": 130.313494, + "expiry_deletes": null, + "digest": "a4b2b27b94a97c7d2007f178a38f5698c2470ac858681a3e65e01f36be9e32fb" + }, + "after": { + "capacity": 10000, + "rounds": 20, + "expiry_ms": [ + 0.006048999999620719, + 0.005108000000291213, + 0.007330999999794585, + 0.006509999999252614, + 0.006929999999556458, + 0.004487000000153785, + 0.00490799999988667, + 0.004707000000053085, + 0.005537999999432941, + 0.0053079999997862615, + 0.004966999999851396, + 0.00539799999933166, + 0.008513000000675675, + 0.009074000000509841, + 0.006278999999267398, + 0.007000000000516593, + 0.006459999999606225, + 0.006288999999924272, + 0.006971000000703498, + 0.004947999999785679 + ], + "admission_ms": 121.8151799999996, + "expiry_deletes": null, + "digest": "a4b2b27b94a97c7d2007f178a38f5698c2470ac858681a3e65e01f36be9e32fb" + } + }, + { + "order": [ + "after", + "before" + ], + "after": { + "capacity": 10000, + "rounds": 20, + "expiry_ms": [ + 0.007531000000199128, + 0.007250999999996566, + 0.007402000000183762, + 0.005868999999620428, + 0.008912999999665772, + 0.005548000000089814, + 0.004958000000442553, + 0.006709999999657157, + 0.0059689999998226995, + 0.005008000000088941, + 0.0048569999999017455, + 0.003885999999511114, + 0.004496999999901163, + 0.004036000000269269, + 0.007241000000249187, + 0.005287999999382009, + 0.006129000000328233, + 0.004866999999649124, + 0.00489700000071025, + 0.005227000000559201 + ], + "admission_ms": 114.7533340000009, + "expiry_deletes": null, + "digest": "a4b2b27b94a97c7d2007f178a38f5698c2470ac858681a3e65e01f36be9e32fb" + }, + "before": { + "capacity": 10000, + "rounds": 20, + "expiry_ms": [ + 1.589904000000388, + 1.5738289999999324, + 1.619198000000324, + 1.590063999999984, + 2.2713300000004892, + 1.6508160000003045, + 1.6013810000004014, + 1.5640949999997247, + 1.6493640000007872, + 1.8619930000004388, + 1.6790379999993092, + 1.5859980000004725, + 1.6929689999997208, + 1.8467110000001412, + 1.6122470000000249, + 1.6491530000002967, + 1.5940600000003542, + 1.7723180000002685, + 1.6198790000007648, + 1.8334209999993618 + ], + "admission_ms": 133.40308399999867, + "expiry_deletes": null, + "digest": "a4b2b27b94a97c7d2007f178a38f5698c2470ac858681a3e65e01f36be9e32fb" + } + }, + { + "order": [ + "before", + "after" + ], + "before": { + "capacity": 10000, + "rounds": 20, + "expiry_ms": [ + 2.9932070000004387, + 1.6759940000001734, + 1.608100999999806, + 1.6383770000002187, + 1.5702640000008614, + 1.635352000000239, + 1.8996999999999389, + 1.5880409999999756, + 1.6307449999994788, + 1.8662100000001374, + 1.6749620000000505, + 1.7161940000005416, + 1.905749000000469, + 1.673899999999776, + 1.6166439999997237, + 1.9608829999997397, + 1.9607020000003104, + 2.400383999999576, + 1.763104999999996, + 2.2779490000002625 + ], + "admission_ms": 164.27379099999962, + "expiry_deletes": null, + "digest": "a4b2b27b94a97c7d2007f178a38f5698c2470ac858681a3e65e01f36be9e32fb" + }, + "after": { + "capacity": 10000, + "rounds": 20, + "expiry_ms": [ + 0.04839300000003277, + 0.011556999999811524, + 0.014312000000245462, + 0.015813999999409134, + 0.015583999999762455, + 0.01769700000022567, + 0.017495999999482592, + 0.015713000000687316, + 0.010715999999774795, + 0.018066999999973632, + 0.016845000000103028, + 0.013570000000072469, + 0.015743999999358493, + 0.012338999999883526, + 0.011177000000316184, + 0.009664999999586144, + 0.007811999999830732, + 0.010446000000229105, + 0.015072000000145636, + 0.013039999999818974 + ], + "admission_ms": 194.66671599999881, + "expiry_deletes": null, + "digest": "a4b2b27b94a97c7d2007f178a38f5698c2470ac858681a3e65e01f36be9e32fb" + } + } + ] + } + }, + "boundary": "Production middleware and real ApiError/clientIdentity modules; controlled clock/request/response, local component timing. Not HTTP, deployed service, provider or fleet throughput." +} diff --git a/docs/validation/rate-limit-expiry-20261004.json b/docs/validation/rate-limit-expiry-20261004.json new file mode 100644 index 0000000..51b2f7c --- /dev/null +++ b/docs/validation/rate-limit-expiry-20261004.json @@ -0,0 +1,148 @@ +{ + "node": "v24.19.0", + "parent": "f501ce5f4fd6f6f96c82fb9fffed4dc58f3def8e", + "before_blob": "6748ce8105628e781cccea40c368d2bb3a830b17", + "after_blob": "84c97f7b183c1c8e96be0695e5f332f9c2e3acca", + "semantic": { + "calls": 20000, + "resets": 629, + "capacities": [ + 1, + 2, + 7, + 31, + 256 + ], + "outputs_identical": true, + "digest": "e33b264b79ca6ee6608a4bea3d2cd2c88fc3f8f1890205c16793019add83c124" + }, + "counted": { + "before": { + "warm_ms": 40.84700099999998, + "rolling_ms": 441.346272, + "accepted": 11000, + "tracked": 10000, + "map_entry_visits": 10000000, + "output_digest": "4d3275de14312bc8d9ad15fa5c01bd8a8a76d8df612b5b37c31c3f86e5ce84a3" + }, + "after": { + "warm_ms": 23.80022500000007, + "rolling_ms": 3.1638240000000906, + "accepted": 11000, + "tracked": 10000, + "map_entry_visits": 0, + "output_digest": "4d3275de14312bc8d9ad15fa5c01bd8a8a76d8df612b5b37c31c3f86e5ce84a3" + } + }, + "pairs": [ + { + "order": [ + "before", + "after" + ], + "before": { + "warm_ms": 21.79057899999998, + "rolling_ms": 474.4274180000001, + "accepted": 11000, + "tracked": 10000, + "map_entry_visits": null, + "output_digest": "4d3275de14312bc8d9ad15fa5c01bd8a8a76d8df612b5b37c31c3f86e5ce84a3" + }, + "after": { + "warm_ms": 20.588298000000123, + "rolling_ms": 2.465562000000091, + "accepted": 11000, + "tracked": 10000, + "map_entry_visits": null, + "output_digest": "4d3275de14312bc8d9ad15fa5c01bd8a8a76d8df612b5b37c31c3f86e5ce84a3" + } + }, + { + "order": [ + "after", + "before" + ], + "after": { + "warm_ms": 32.072959999999966, + "rolling_ms": 2.818273999999974, + "accepted": 11000, + "tracked": 10000, + "map_entry_visits": null, + "output_digest": "4d3275de14312bc8d9ad15fa5c01bd8a8a76d8df612b5b37c31c3f86e5ce84a3" + }, + "before": { + "warm_ms": 19.74772299999995, + "rolling_ms": 224.3245569999999, + "accepted": 11000, + "tracked": 10000, + "map_entry_visits": null, + "output_digest": "4d3275de14312bc8d9ad15fa5c01bd8a8a76d8df612b5b37c31c3f86e5ce84a3" + } + }, + { + "order": [ + "before", + "after" + ], + "before": { + "warm_ms": 18.826625999999806, + "rolling_ms": 183.07371799999987, + "accepted": 11000, + "tracked": 10000, + "map_entry_visits": null, + "output_digest": "4d3275de14312bc8d9ad15fa5c01bd8a8a76d8df612b5b37c31c3f86e5ce84a3" + }, + "after": { + "warm_ms": 107.33902499999999, + "rolling_ms": 4.7101410000000214, + "accepted": 11000, + "tracked": 10000, + "map_entry_visits": null, + "output_digest": "4d3275de14312bc8d9ad15fa5c01bd8a8a76d8df612b5b37c31c3f86e5ce84a3" + } + } + ], + "rolling_median_ms": { + "before": 224.3245569999999, + "after": 2.818273999999974, + "ratio": 79.59643278119941 + }, + "boundary": "Unmodified loaded middleware modules and retained real ApiError/clientIdentity dependencies; simulated clock/requests/responses. No Express, HTTP stack, dependency install, external service or fleet throughput measurement.", + "integration_validation": { + "node": "v24.19.0", + "lock_blob": "be3ad84b812121d2ec4f47541df3a55816bb210c", + "dependencies_reused_from_run": 37191775463, + "dependencies_artifact": 11299062132, + "checks": [ + { + "command": "node --test test/abuseControls.test.js", + "reported_summary": { + "tests": 17, + "suites": 0, + "pass": 17, + "fail": 0, + "cancelled": 0, + "skipped": 0, + "todo": 0, + "duration_ms": 403.177995 + }, + "log_sha256": "e768d229d4136ad76c5762da86e92514644baf0c5c89b727ba091ada88c3a669" + }, + { + "command": "npm test", + "reported_summary": { + "tests": 273, + "suites": 1, + "pass": 273, + "fail": 0, + "cancelled": 0, + "skipped": 0, + "todo": 0, + "duration_ms": 2582.066949 + }, + "log_sha256": "dfc0d8041520deeff839ad360f72804b7fb8ba0382cfa5af03aba0d6d23a441f" + } + ], + "scope": "Maintained tests on all 178 exact parent files, composed with the two candidate source/test blobs; real Express loopback cases included. Original npm test command reported 273 passes. No deployed or provider acceptance claimed." + } +} diff --git a/scripts/benchmark-rate-limit-batch-expiry.cjs b/scripts/benchmark-rate-limit-batch-expiry.cjs new file mode 100644 index 0000000..bd7a3fe --- /dev/null +++ b/scripts/benchmark-rate-limit-batch-expiry.cjs @@ -0,0 +1,116 @@ +'use strict'; + +// Compare the actual production modules in two checkouts; no installed packages +// or external services are needed. Only request objects and the clock are fixed. +const assert = require('node:assert/strict'); +const crypto = require('node:crypto'); +const fs = require('node:fs'); +const path = require('node:path'); +const { performance } = require('node:perf_hooks'); + +const [beforeDir, afterDir, output] = process.argv.slice(2); +if (!beforeDir || !afterDir || !output) { + console.error('Usage: node benchmark-rate-limit-batch-expiry.cjs BEFORE AFTER NEW_RESULT.json'); + process.exit(1); +} +const filename = (root) => path.resolve(root, 'src/middleware/rateLimit.js'); +const modules = { before: require(filename(beforeDir)), after: require(filename(afterDir)) }; +const nativeNow = Date.now; +let now = 0; +Date.now = () => now; + +function invoke(limiter, key) { + const headers = {}; + let error = null; + limiter({ key }, { set(name, value) { headers[name] = value; } }, (err) => { + if (err) error = { statusCode: err.statusCode, message: err.message, details: err.details }; + }); + return { headers, error, size: limiter.size() }; +} + +function sample(build, staggered, countDeletes = false) { + const capacity = 10000; + const rounds = 20; + const NativeMap = global.Map; + let deletes = 0; + class CountedMap extends NativeMap { + delete(key) { deletes += 1; return super.delete(key); } + } + let limiter; + try { + if (countDeletes) global.Map = CountedMap; + limiter = build({ maxKeys: capacity, windowMs: capacity, max: 1, + forceInTest: true, keyGenerator: (req) => req.key }); + } finally { global.Map = NativeMap; } + const request = { key: '' }; + const response = { set() {} }; + const next = (error) => { assert.equal(error, undefined); }; + const digest = crypto.createHash('sha256'); + const expirySamples = []; + let admissionMs = 0; + let expiryDeletes = 0; + for (let round = 0; round < rounds; round += 1) { + limiter.reset(); + const start = round * capacity * 3; + const admitStart = performance.now(); + for (let index = 0; index < capacity; index += 1) { + now = start + (staggered ? index : 0); + request.key = `identity-${index}`; + limiter(request, response, next); + } + admissionMs += performance.now() - admitStart; + assert.equal(limiter.size(), capacity); + now = start + capacity + (staggered ? capacity - 1 : 0); + const previousDeletes = deletes; + const expiryStart = performance.now(); + const admitted = invoke(limiter, 'new'); + expirySamples.push(performance.now() - expiryStart); + expiryDeletes += deletes - previousDeletes; + assert.equal(admitted.error, null); + assert.equal(admitted.size, 1); + const blocked = invoke(limiter, 'new'); + assert.equal(blocked.error.statusCode, 429); + digest.update(JSON.stringify({ admitted, blocked })); + } + return { capacity, rounds, expiry_ms: expirySamples, + admission_ms: admissionMs, expiry_deletes: countDeletes ? expiryDeletes : null, + digest: digest.digest('hex') }; +} + +const median = (values) => { + const sorted = values.slice().sort((a, b) => a - b); + const middle = Math.floor(sorted.length / 2); + return sorted.length % 2 ? sorted[middle] : (sorted[middle - 1] + sorted[middle]) / 2; +}; +function blob(root) { + const bytes = fs.readFileSync(filename(root)); + return crypto.createHash('sha1').update(`blob ${bytes.length}\0`).update(bytes).digest('hex'); +} + +try { + const result = { node: process.version, before_blob: blob(beforeDir), after_blob: blob(afterDir), workloads: {} }; + for (const [label, staggered] of [['same_deadline', false], ['staggered_all_expired', true]]) { + const counted = {}; + for (const phase of ['before', 'after']) counted[phase] = sample(modules[phase], staggered, true); + assert.equal(counted.before.digest, counted.after.digest); + const pairs = []; + for (let index = 0; index < 5; index += 1) { + const pair = { order: index % 2 ? ['after', 'before'] : ['before', 'after'] }; + for (const phase of pair.order) pair[phase] = sample(modules[phase], staggered); + assert.equal(pair.before.digest, pair.after.digest); + pairs.push(pair); + } + const summary = {}; + for (const phase of ['before', 'after']) { + const times = pairs.flatMap((pair) => pair[phase].expiry_ms); + summary[phase] = { expiry_median_ms: median(times), expiry_range_ms: [Math.min(...times), Math.max(...times)], + admission_median_ms: median(pairs.map((pair) => pair[phase].admission_ms)), + expiry_deletes: counted[phase].expiry_deletes, digest: counted[phase].digest }; + } + result.workloads[label] = { summary, pairs }; + } + result.boundary = 'Production middleware and real ApiError/clientIdentity modules; controlled clock/request/response, local component timing. Not HTTP, deployed service, provider or fleet throughput.'; + fs.writeFileSync(output, JSON.stringify(result, null, 2) + '\n', { flag: 'wx' }); + console.log(JSON.stringify({ node: result.node, before_blob: result.before_blob, after_blob: result.after_blob, + workloads: Object.fromEntries(Object.entries(result.workloads).map(([key, value]) => [key, value.summary])) }, null, 2)); +} finally { Date.now = nativeNow; } diff --git a/scripts/benchmark-rate-limit-expiry.cjs b/scripts/benchmark-rate-limit-expiry.cjs new file mode 100644 index 0000000..2df8f93 --- /dev/null +++ b/scripts/benchmark-rate-limit-expiry.cjs @@ -0,0 +1,129 @@ +'use strict'; + +const assert = require('node:assert/strict'); +const fs = require('node:fs'); +const path = require('node:path'); +const crypto = require('node:crypto'); +const { performance } = require('node:perf_hooks'); + +const root = __dirname; +const before = require('./original/src/middleware/rateLimit'); +const after = require('./final/src/middleware/rateLimit'); +const nativeNow = Date.now; +let now = 0; +Date.now = () => now; +process.env.NODE_ENV = 'test'; + +function invoke(limiter, key) { + const headers = {}; + let error = null; + limiter({ key }, { set(name, value) { headers[name] = String(value); } }, (err) => { + if (err) error = { name: err.name, statusCode: err.statusCode, message: err.message, details: err.details }; + }); + return { headers, error, size: limiter.size() }; +} + +function differential() { + let seed = 0x138; + const random = () => { seed = (Math.imul(seed, 1664525) + 1013904223) >>> 0; return seed; }; + const hash = crypto.createHash('sha256'); + let calls = 0; + let resets = 0; + const capacities = [1, 2, 7, 31, 256]; + for (const maxKeys of capacities) { + const options = { maxKeys, windowMs: 101, max: 3, forceInTest: true, keyGenerator: (req) => req.key }; + const old = before(options); + const next = after(options); + now = 1000; + for (let index = 0; index < 4000; index += 1) { + const action = random() % 30; + if (action === 0) { old.reset(); next.reset(); resets += 1; } + if (action < 4) now -= random() % 150; + else now += random() % 24; + const key = `actor-${random() % (maxKeys * 3 + 1)}`; + const a = invoke(old, key); + const b = invoke(next, key); + assert.deepEqual(b, a, `capacity=${maxKeys}, index=${index}, now=${now}, key=${key}`); + assert.ok(next.size() <= maxKeys); + hash.update(JSON.stringify(a)); + calls += 1; + } + } + return { calls, resets, capacities, outputs_identical: true, digest: hash.digest('hex') }; +} + +function rolling(build, instrument = false) { + const NativeMap = global.Map; + let visits = 0; + class CountingMap extends NativeMap { + *[Symbol.iterator]() { + for (const entry of super[Symbol.iterator]()) { visits += 1; yield entry; } + } + } + const maxKeys = 10000; + let limiter; + try { + if (instrument) global.Map = CountingMap; + limiter = build({ maxKeys, windowMs: maxKeys, max: 1, forceInTest: true, keyGenerator: (req) => req.key }); + } finally { global.Map = NativeMap; } + let accepted = 0; + const hash = crypto.createHash('sha256'); + const warmStart = performance.now(); + for (let index = 0; index < maxKeys; index += 1) { + now = index; + const result = invoke(limiter, `known-${index}`); + assert.equal(result.error, null); + accepted += 1; + hash.update(JSON.stringify(result)); + } + const warmMs = performance.now() - warmStart; + const rollingStart = performance.now(); + for (let index = 0; index < 1000; index += 1) { + now = maxKeys + index; + const result = invoke(limiter, `new-${index}`); + assert.equal(result.error, null); + accepted += 1; + hash.update(JSON.stringify(result)); + } + const rollingMs = performance.now() - rollingStart; + const survivor = invoke(limiter, `known-${maxKeys - 1}`); + assert.equal(survivor.error.statusCode, 429); + hash.update(JSON.stringify(survivor)); + return { warm_ms: warmMs, rolling_ms: rollingMs, accepted, tracked: limiter.size(), map_entry_visits: instrument ? visits : null, output_digest: hash.digest('hex') }; +} + +function gitBlob(file) { + const data = fs.readFileSync(file); + return crypto.createHash('sha1').update(`blob ${data.length}\0`).update(data).digest('hex'); +} + +try { + const semantic = differential(); + const counted = { before: rolling(before, true), after: rolling(after, true) }; + assert.equal(counted.before.map_entry_visits, 10000000); + assert.equal(counted.after.map_entry_visits, 0); + assert.equal(counted.before.output_digest, counted.after.output_digest); + const pairs = []; + for (let index = 0; index < 3; index += 1) { + const pair = { order: index % 2 ? ['after', 'before'] : ['before', 'after'] }; + for (const phase of pair.order) pair[phase] = rolling(phase === 'before' ? before : after); + assert.equal(pair.before.output_digest, pair.after.output_digest); + pairs.push(pair); + } + const median = (values) => values.slice().sort((a, b) => a - b)[Math.floor(values.length / 2)]; + const oldMedian = median(pairs.map((p) => p.before.rolling_ms)); + const newMedian = median(pairs.map((p) => p.after.rolling_ms)); + const result = { + node: process.version, + parent: 'f501ce5f4fd6f6f96c82fb9fffed4dc58f3def8e', + before_blob: gitBlob(path.join(root, 'original/src/middleware/rateLimit.js')), + after_blob: gitBlob(path.join(root, 'final/src/middleware/rateLimit.js')), + semantic, + counted, + pairs, + rolling_median_ms: { before: oldMedian, after: newMedian, ratio: oldMedian / newMedian }, + boundary: 'Unmodified loaded middleware modules and retained real ApiError/clientIdentity dependencies; simulated clock/requests/responses. No Express, HTTP stack, dependency install, external service or fleet throughput measurement.' + }; + fs.writeFileSync(path.join(root, 'results.json'), JSON.stringify(result, null, 2) + '\n'); + console.log(JSON.stringify(result, null, 2)); +} finally { Date.now = nativeNow; } diff --git a/scripts/benchmark-rate-limit.cjs b/scripts/benchmark-rate-limit.cjs new file mode 100644 index 0000000..9f3b948 --- /dev/null +++ b/scripts/benchmark-rate-limit.cjs @@ -0,0 +1,119 @@ +'use strict'; + +const assert = require('node:assert/strict'); +const { createHash } = require('node:crypto'); +const fs = require('node:fs'); +const http = require('node:http'); +const path = require('node:path'); +const { performance } = require('node:perf_hooks'); + +// Usage: node scripts/benchmark-rate-limit.cjs NEW-RESULT.json [APP-CHECKOUT] +const outputPath = process.argv[2]; +if (!outputPath || process.argv.length > 4) { + console.error('Usage: node scripts/benchmark-rate-limit.cjs NEW-RESULT.json [APP-CHECKOUT]'); + process.exitCode = 2; +} else { + run().catch((error) => { + console.error(error); + process.exitCode = 1; + }); +} + +async function run() { + if (fs.existsSync(outputPath)) throw new Error(`Result already exists: ${outputPath}`); + const appRoot = path.resolve(process.argv[3] || path.join(__dirname, '..')); + Object.assign(process.env, { + NODE_ENV: 'test', + ENABLE_RATE_LIMIT_IN_TEST: '1', + RATE_LIMIT_MAX_KEYS: '10000', + RATE_LIMIT_MAX: '1', + RATE_LIMIT_WINDOW_MS: '180000', + TRUST_PROXY: 'true', + ERROR_TRACKING_ENABLED: 'false', + }); + + const createApp = require(path.join(appRoot, 'src/app')); + const server = http.createServer(createApp()); + const agent = new http.Agent({ keepAlive: true, maxSockets: 32 }); + let port; + + function request(identity) { + const ip = `10.${(identity >>> 16) & 255}.${(identity >>> 8) & 255}.${identity & 255}`; + return new Promise((resolve, reject) => { + const req = http.get({ + host: '127.0.0.1', port, path: '/api/version', agent, + headers: { 'X-Forwarded-For': ip, 'X-Request-Id': `load-${identity}` }, + }, (res) => { + let body = ''; + res.on('data', (chunk) => { body += chunk; }); + res.on('error', reject); + res.on('end', () => { + try { + resolve({ status: res.statusCode, headers: res.headers, body: JSON.parse(body) }); + } catch (error) { + reject(error); + } + }); + }); + req.setTimeout(30_000, () => req.destroy(new Error('Local benchmark request timed out'))); + req.on('error', reject); + }); + } + + async function batch(start, count, expectedStatus) { + let next = 0; + let completed = 0; + await Promise.all(Array.from({ length: 32 }, async () => { + while (next < count) { + const identity = start + next++; + const result = await request(identity); + assert.equal(result.status, expectedStatus); + if (expectedStatus === 429) { + assert.equal(result.body.error.details.policy, 'global'); + assert.equal(result.headers['x-request-id'], `load-${identity}`); + } + completed += 1; + } + })); + return completed; + } + + try { + await new Promise((resolve, reject) => { + server.once('error', reject); + server.listen(0, '127.0.0.1', resolve); + }); + port = server.address().port; + const fill = await batch(1, 10_000, 200); + const samples = []; + for (let round = 0; round < 3; round += 1) { + const started = performance.now(); + const cpuStarted = process.cpuUsage(); + const count = await batch(20_000 + round * 2_000, 2_000, 429); + const cpu = process.cpuUsage(cpuStarted); + samples.push({ count, elapsedMs: performance.now() - started, cpuMs: (cpu.user + cpu.system) / 1_000 }); + } + + const original = await request(1); + assert.equal(original.status, 429); + assert.equal(original.body.error.message, 'Too many requests, please try again later'); + + const sourceSha256 = {}; + for (const file of ['src/app.js', 'src/middleware/rateLimit.js']) { + sourceSha256[file] = createHash('sha256').update(fs.readFileSync(path.join(appRoot, file))).digest('hex'); + } + const result = { + node: process.version, + express: require(path.join(appRoot, 'node_modules/express/package.json')).version, + sourceSha256, + path: '/api/version', concurrency: 32, maxKeys: 10_000, fill, samples, + activeBudgetPreserved: true, + scope: 'Real createApp HTTP on loopback; test mode with rate limiting enabled; one trusted proxy hop; error tracking disabled; CPU includes client and server in one process; generated local load, not deployed performance.', + }; + fs.writeFileSync(outputPath, `${JSON.stringify(result, null, 2)}\n`, { flag: 'wx' }); + } finally { + agent.destroy(); + server.closeAllConnections(); + if (server.listening) await new Promise((resolve) => server.close(resolve)); + } +} diff --git a/src/app.js b/src/app.js index bd5b065..82909b2 100644 --- a/src/app.js +++ b/src/app.js @@ -16,6 +16,7 @@ const maintenanceMode = require('./middleware/maintenanceMode'); const jsonError = require('./middleware/jsonError'); const notFound = require('./middleware/notFound'); const errorHandler = require('./middleware/errorHandler'); +const { resolveClientIp } = require('./utils/clientIdentity'); /** * Build and configure the Express application. @@ -26,16 +27,16 @@ const errorHandler = require('./middleware/errorHandler'); function createApp() { const app = express(); + // Only honour X-Forwarded-* when explicitly configured. Blind trust lets + // clients rotate forged IPs and evade the global abuse budget. + if (config.trustProxy) { + app.set('trust proxy', 1); + } + // Core middleware. app.use(securityHeaders); app.use(cacheControl({ policy: config.cache.defaultPolicy })); app.use(cors({ origin: config.corsOrigin })); - app.use(express.json({ limit: config.bodyLimit })); - app.use(express.urlencoded({ extended: false, limit: config.bodyLimit })); - app.use(jsonError); - - // Fail slow requests instead of hanging the connection. - app.use(requestTimeout({ ms: config.requestTimeoutMs })); // Assign/propagate a correlation id before logging. app.use(requestId); @@ -46,8 +47,27 @@ function createApp() { } app.use(requestLogger); - // Basic abuse protection on the API surface. - app.use('/api', rateLimit(config.rateLimit)); + // Apply the API budget before body parsing so invalid bodies cannot bypass it. + app.use( + '/api', + rateLimit({ + name: 'global', + windowMs: config.rateLimit.windowMs, + max: config.rateLimit.max, + maxKeys: config.rateLimit.maxKeys, + trustProxy: config.trustProxy, + keyGenerator(req) { + return resolveClientIp(req, { trustProxy: config.trustProxy }); + }, + }) + ); + + app.use(express.json({ limit: config.bodyLimit })); + app.use(express.urlencoded({ extended: false, limit: config.bodyLimit })); + app.use(jsonError); + + // Fail slow handlers instead of hanging the connection. + app.use(requestTimeout({ ms: config.requestTimeoutMs })); // Block all non-health API traffic while maintenance mode is active. app.use(maintenanceMode); diff --git a/src/config/index.js b/src/config/index.js index 41f9a71..320e04f 100644 --- a/src/config/index.js +++ b/src/config/index.js @@ -2,13 +2,27 @@ require('dotenv').config(); +/** + * Parse a positive integer env var with a fallback. + * @param {string|undefined} value + * @param {number} fallback + * @returns {number} + */ +function intEnv(value, fallback) { + const parsed = parseInt(value, 10); + return Number.isFinite(parsed) && parsed > 0 ? parsed : fallback; +} + /** * Centralized application configuration. * Values are read from environment variables with sensible defaults * so the app can boot even without a .env file present. */ +const env = process.env.NODE_ENV || 'development'; +const isTest = env === 'test'; + const config = { - env: process.env.NODE_ENV || 'development', + env, port: parseInt(process.env.PORT, 10) || 3000, baseCurrency: process.env.DEFAULT_BASE_CURRENCY || 'USD', @@ -34,9 +48,44 @@ const config = { // Per-request time budget before a 503 is returned. requestTimeoutMs: parseInt(process.env.REQUEST_TIMEOUT_MS, 10) || 15 * 1000, + /** + * Whether to trust `X-Forwarded-For` when resolving the client IP. + * Off by default so untrusted clients cannot rotate IPs to bypass limits. + * Enable only behind a reverse proxy that strips/forges the header safely. + */ + trustProxy: process.env.TRUST_PROXY === 'true' || process.env.TRUST_PROXY === '1', + rateLimit: { - windowMs: parseInt(process.env.RATE_LIMIT_WINDOW_MS, 10) || 60 * 1000, - max: parseInt(process.env.RATE_LIMIT_MAX, 10) || 100, + windowMs: intEnv(process.env.RATE_LIMIT_WINDOW_MS, 60 * 1000), + max: intEnv(process.env.RATE_LIMIT_MAX, isTest ? 10_000 : 100), + maxKeys: intEnv(process.env.RATE_LIMIT_MAX_KEYS, 10_000), + }, + + /** + * Stricter per-actor budgets for mutation / expensive routes. + * Defaults are deliberately higher than a single integration test suite + * needs, while still bounding automated abuse of provider-backed paths. + */ + mutationRateLimit: { + maxKeys: intEnv(process.env.MUTATION_RATE_LIMIT_MAX_KEYS, 10_000), + transfers: { + windowMs: intEnv(process.env.MUTATION_RATE_LIMIT_TRANSFERS_WINDOW_MS, 60 * 1000), + // Generous under test so the suite does not trip the abuse budget; production + // defaults stay tight enough to bound provider-quota exhaustion. + max: intEnv(process.env.MUTATION_RATE_LIMIT_TRANSFERS_MAX, isTest ? 10_000 : 30), + }, + users: { + windowMs: intEnv(process.env.MUTATION_RATE_LIMIT_USERS_WINDOW_MS, 60 * 1000), + max: intEnv(process.env.MUTATION_RATE_LIMIT_USERS_MAX, isTest ? 10_000 : 20), + }, + quote: { + windowMs: intEnv(process.env.MUTATION_RATE_LIMIT_QUOTE_WINDOW_MS, 60 * 1000), + max: intEnv(process.env.MUTATION_RATE_LIMIT_QUOTE_MAX, isTest ? 10_000 : 60), + }, + admin: { + windowMs: intEnv(process.env.MUTATION_RATE_LIMIT_ADMIN_WINDOW_MS, 60 * 1000), + max: intEnv(process.env.MUTATION_RATE_LIMIT_ADMIN_MAX, isTest ? 10_000 : 30), + }, }, errorTracking: { diff --git a/src/controllers/adminController.js b/src/controllers/adminController.js index 870e10e..ea41def 100644 --- a/src/controllers/adminController.js +++ b/src/controllers/adminController.js @@ -30,7 +30,9 @@ function getDiagnostics(req, res) { fee: config.fee, maxTransferAmount: config.maxTransferAmount, stellar: config.stellar, + trustProxy: config.trustProxy, rateLimit: config.rateLimit, + mutationRateLimit: config.mutationRateLimit, errorTrackingEnabled: config.errorTracking.enabled, }, stats: { diff --git a/src/middleware/adminAuth.js b/src/middleware/adminAuth.js index 7828ad7..5217b65 100644 --- a/src/middleware/adminAuth.js +++ b/src/middleware/adminAuth.js @@ -23,6 +23,8 @@ function adminAuth(req, res, next) { return next(new ApiError(401, 'Unauthorized')); } + // Expose for downstream actor-keyed rate limiting (never log the raw value). + req.adminToken = token; next(); } diff --git a/src/middleware/mutationRateLimit.js b/src/middleware/mutationRateLimit.js new file mode 100644 index 0000000..c4f6ccb --- /dev/null +++ b/src/middleware/mutationRateLimit.js @@ -0,0 +1,55 @@ +'use strict'; + +const config = require('../config'); +const rateLimit = require('./rateLimit'); +const { resolveActorKey } = require('../utils/clientIdentity'); + +/** + * Route-family mutation rate limiters. + * + * Applied after authentication so the actor key is a token fingerprint + * (never the raw secret). Quote stays IP-keyed because it is public. + * Each limiter is independently bounded so one hot route cannot starve + * another family's budget, and the shared maxKeys cap keeps memory finite. + */ + +function buildLimiter(family, overrides = {}) { + const settings = config.mutationRateLimit[family] || {}; + const trustProxy = config.trustProxy; + + return rateLimit({ + name: `mutation:${family}`, + windowMs: overrides.windowMs || settings.windowMs, + max: overrides.max || settings.max, + maxKeys: overrides.maxKeys || config.mutationRateLimit.maxKeys, + trustProxy, + forceInTest: Boolean(overrides.forceInTest), + keyGenerator(req) { + return `${family}:${resolveActorKey(req, { trustProxy })}`; + }, + }); +} + +const transfers = buildLimiter('transfers'); +const users = buildLimiter('users'); +const quote = buildLimiter('quote'); +const admin = buildLimiter('admin'); + +/** + * Reset every mutation limiter. Intended for tests only. + */ +function resetAll() { + transfers.reset(); + users.reset(); + quote.reset(); + admin.reset(); +} + +module.exports = { + transfers, + users, + quote, + admin, + resetAll, + buildLimiter, +}; diff --git a/src/middleware/rateLimit.js b/src/middleware/rateLimit.js index b7fa7bd..a51b607 100644 --- a/src/middleware/rateLimit.js +++ b/src/middleware/rateLimit.js @@ -1,52 +1,181 @@ 'use strict'; const ApiError = require('../utils/ApiError'); +const { resolveClientIp } = require('../utils/clientIdentity'); /** - * Minimal in-memory rate limiter. - * Tracks request counts per client (by IP) within a fixed time window. - * Counters live only in process memory, which is fine for a single-node - * demo; a real deployment would use a shared store such as Redis. + * In-memory fixed-window rate limiter with a bounded key table. + * + * Counters live only in process memory (fine for a single-node demo). A + * production deployment should back this with a shared store such as Redis. + * The map is capped by `maxKeys`: expired entries are pruned first. When + * all slots are live, new identities receive 429 until a slot expires rather + * than evicting an active budget and allowing repeat attempts. + * + * Under `NODE_ENV=test` the limiter is a no-op unless + * `ENABLE_RATE_LIMIT_IN_TEST=1`, so the rest of the suite is not coupled to + * the abuse budget. Dedicated limiter and abuse-control tests opt back in. * * @param {object} [options] * @param {number} [options.windowMs] - length of the window in milliseconds. - * @param {number} [options.max] - max requests allowed per window per client. - * @returns {import('express').RequestHandler} + * @param {number} [options.max] - max requests allowed per window per key. + * @param {number} [options.maxKeys] - hard cap on tracked identities. + * @param {(req: import('express').Request) => string} [options.keyGenerator] + * @param {string} [options.name] - label included in 429 details (no secrets). + * @param {boolean} [options.trustProxy] - whether to honour X-Forwarded-For. + * @param {boolean} [options.forceInTest] - enforce even when NODE_ENV=test. + * @returns {import('express').RequestHandler & { reset: Function, size: Function }} */ function rateLimit(options = {}) { const windowMs = options.windowMs || 60 * 1000; const max = options.max || 100; + const maxKeys = options.maxKeys || 10_000; + const name = options.name || 'default'; + const trustProxy = Boolean(options.trustProxy); + const forceInTest = Boolean(options.forceInTest); + const keyGenerator = + options.keyGenerator || + ((req) => resolveClientIp(req, { trustProxy })); + + /** @type {Map} */ const hits = new Map(); + // One heap node per tracked identity. A rolled-back clock can give a newly + // admitted key an earlier deadline, so insertion order is not expiry order. + const expiries = []; + // Maximum deadline, independent of admission order after clock rollback. + let latestResetAt = -Infinity; + + function scheduleExpiry(key, entry) { + latestResetAt = Math.max(latestResetAt, entry.resetAt); + const node = { key, entry }; + let index = expiries.length; + expiries.push(node); + while (index > 0) { + const parent = Math.floor((index - 1) / 2); + if (expiries[parent].entry.resetAt <= entry.resetAt) break; + expiries[index] = expiries[parent]; + index = parent; + } + expiries[index] = node; + } + + function pruneExpired(now) { + // A burst can expire the whole table together. No budget survives this + // deadline, even after clock rollback, so avoid removing each heap node. + if (expiries.length && now >= latestResetAt) { + hits.clear(); + expiries.length = 0; + latestResetAt = -Infinity; + return; + } + // Remove expired entries without scanning the surviving budgets. Repeated + // hits never enqueue again; pruning before renewal keeps both tables bounded. + while (expiries.length && now >= expiries[0].entry.resetAt) { + const expired = expiries[0]; + const tail = expiries.pop(); + if (expiries.length) { + let index = 0; + while (index * 2 + 1 < expiries.length) { + let child = index * 2 + 1; + if (child + 1 < expiries.length && + expiries[child + 1].entry.resetAt < expiries[child].entry.resetAt) { + child += 1; + } + if (tail.entry.resetAt <= expiries[child].entry.resetAt) break; + expiries[index] = expiries[child]; + index = child; + } + expiries[index] = tail; + } + if (hits.get(expired.key) === expired.entry) hits.delete(expired.key); + } + } + + function hasCapacity(now) { + // Also prune before renewal below capacity, removing the old generation + // from the expiry index before the same identity receives a new budget. + pruneExpired(now); + return hits.size < maxKeys; + } + + function capacityRetryAfter(now) { + const nextResetAt = expiries.length ? expiries[0].entry.resetAt : Infinity; + return Math.max(1, Math.ceil((nextResetAt - now) / 1000)); + } + + function rateLimitMiddleware(req, res, next) { + const skipForTest = + process.env.NODE_ENV === 'test' && + process.env.ENABLE_RATE_LIMIT_IN_TEST !== '1' && + !forceInTest; + if (skipForTest) { + return next(); + } - return function rateLimitMiddleware(req, res, next) { const now = Date.now(); - const key = req.ip || 'unknown'; + const key = keyGenerator(req) || 'unknown'; let entry = hits.get(key); if (!entry || now >= entry.resetAt) { - entry = { count: 0, resetAt: now + windowMs }; + if (!hasCapacity(now)) { + const retryAfter = capacityRetryAfter(now); + const resetAt = expiries.length ? expiries[0].entry.resetAt : Infinity; + res.set('X-RateLimit-Limit', String(max)); + res.set('X-RateLimit-Remaining', '0'); + res.set('X-RateLimit-Reset', String(Math.ceil(resetAt / 1000))); + res.set('X-RateLimit-Policy', name); + res.set('Retry-After', String(retryAfter)); + return next( + ApiError.tooManyRequests('Rate limit capacity reached, please try again later', { + retryAfter, + limit: max, + windowMs, + policy: name, + }) + ); + } + entry = { count: 0, resetAt: now + windowMs, touchedAt: now }; + hits.delete(key); hits.set(key, entry); + scheduleExpiry(key, entry); } entry.count += 1; + entry.touchedAt = now; const remaining = Math.max(0, max - entry.count); res.set('X-RateLimit-Limit', String(max)); res.set('X-RateLimit-Remaining', String(remaining)); res.set('X-RateLimit-Reset', String(Math.ceil(entry.resetAt / 1000))); + res.set('X-RateLimit-Policy', name); if (entry.count > max) { - const retryAfter = Math.ceil((entry.resetAt - now) / 1000); + const retryAfter = Math.max(1, Math.ceil((entry.resetAt - now) / 1000)); res.set('Retry-After', String(retryAfter)); return next( ApiError.tooManyRequests('Too many requests, please try again later', { retryAfter, + limit: max, + windowMs, + policy: name, }) ); } return next(); + } + + rateLimitMiddleware.reset = function reset() { + hits.clear(); + expiries.length = 0; + latestResetAt = -Infinity; + }; + + rateLimitMiddleware.size = function size() { + return hits.size; }; + + return rateLimitMiddleware; } module.exports = rateLimit; diff --git a/src/middleware/requestId.js b/src/middleware/requestId.js index 91cecea..ac16827 100644 --- a/src/middleware/requestId.js +++ b/src/middleware/requestId.js @@ -2,18 +2,67 @@ const { newId } = require('../utils/ids'); +/** Max length accepted for an inbound correlation / request id. */ +const MAX_CORRELATION_ID_LENGTH = 128; + +/** + * Validate correlation-id syntax and size. This alone cannot identify + * credentials or arbitrary personal data; requestId separately excludes + * credentials presented with the current request. + * @param {*} value + * @returns {string|null} + */ +function sanitizeCorrelationId(value) { + if (typeof value !== 'string') { + return null; + } + const trimmed = value.trim(); + if (!trimmed || trimmed.length > MAX_CORRELATION_ID_LENGTH) { + return null; + } + if (!/^[A-Za-z0-9._:-]+$/.test(trimmed)) { + return null; + } + return trimmed; +} + /** - * Attach a unique identifier to every request. - * Honours an inbound `X-Request-Id` header when present so a caller can - * correlate logs across services; otherwise a fresh id is generated. - * The id is exposed on `req.id` and echoed back in the response header. + * Attach a unique correlation identifier to every request. + * + * Honour inbound `X-Request-Id` or `X-Correlation-Id` when they pass + * sanitization and contain neither presented Bearer nor admin token. A + * syntactically valid ID is not proof that arbitrary data is non-secret. + * Callers must use independent correlation IDs. Otherwise a + * fresh id is generated. The same value is exposed as `req.id` and + * `req.correlationId`, and echoed on both response headers so either + * convention works for clients. */ function requestId(req, res, next) { - const incoming = req.get('X-Request-Id'); - const id = incoming && incoming.trim() ? incoming.trim() : newId(); + // This middleware runs before authentication, so inspect the presented + // credentials without changing whether the request is authorized. + const authorization = req.get('Authorization'); + const bearer = typeof authorization === 'string' + ? /^Bearer\s+(\S+)/i.exec(authorization) + : null; + const credentials = [bearer && bearer[1], req.get('X-Admin-Token')] + .filter((value) => typeof value === 'string' && value.length > 0); + const safeIncoming = (value) => { + const candidate = sanitizeCorrelationId(value); + return candidate && !credentials.some((token) => candidate.includes(token)) + ? candidate + : null; + }; + const incoming = + safeIncoming(req.get('X-Request-Id')) || + safeIncoming(req.get('X-Correlation-Id')); + const id = incoming || newId(); req.id = id; + req.correlationId = id; res.set('X-Request-Id', id); + res.set('X-Correlation-Id', id); next(); } module.exports = requestId; +module.exports.sanitizeCorrelationId = sanitizeCorrelationId; +module.exports.MAX_CORRELATION_ID_LENGTH = MAX_CORRELATION_ID_LENGTH; diff --git a/src/routes/adminRoutes.js b/src/routes/adminRoutes.js index 8fe8f4f..472a6b4 100644 --- a/src/routes/adminRoutes.js +++ b/src/routes/adminRoutes.js @@ -3,6 +3,7 @@ const express = require('express'); const asyncHandler = require('../utils/asyncHandler'); const adminAuth = require('../middleware/adminAuth'); +const mutationRateLimit = require('../middleware/mutationRateLimit'); const adminController = require('../controllers/adminController'); const router = express.Router(); @@ -11,6 +12,7 @@ const router = express.Router(); router.get( '/diagnostics', adminAuth, + mutationRateLimit.admin, asyncHandler(adminController.getDiagnostics) ); diff --git a/src/routes/rateRoutes.js b/src/routes/rateRoutes.js index 6f0fc53..1b18982 100644 --- a/src/routes/rateRoutes.js +++ b/src/routes/rateRoutes.js @@ -5,6 +5,7 @@ const config = require('../config'); const cacheControl = require('../middleware/cacheControl'); const asyncHandler = require('../utils/asyncHandler'); const validate = require('../middleware/validate'); +const mutationRateLimit = require('../middleware/mutationRateLimit'); const rateController = require('../controllers/rateController'); const { validateQuoteQuery } = require('../validators/quoteValidator'); @@ -25,8 +26,11 @@ router.get( ); // GET /api/quote?amount=&from=&to= +// Quote is public but provider-adjacent; apply a dedicated IP budget so bursts +// cannot exhaust FX / settlement quotas shared with write paths. router.get( '/quote', + mutationRateLimit.quote, validate(validateQuoteQuery), asyncHandler(rateController.getQuote) ); diff --git a/src/routes/transferRoutes.js b/src/routes/transferRoutes.js index 58eac20..2181954 100644 --- a/src/routes/transferRoutes.js +++ b/src/routes/transferRoutes.js @@ -4,6 +4,7 @@ const express = require('express'); const asyncHandler = require('../utils/asyncHandler'); const validate = require('../middleware/validate'); const requireScope = require('../middleware/requireScope'); +const mutationRateLimit = require('../middleware/mutationRateLimit'); const transferController = require('../controllers/transferController'); const { validateCreateTransfer } = require('../validators/transferValidator'); @@ -13,6 +14,7 @@ const router = express.Router(); router.post( '/', requireScope(['transfers:write']), + mutationRateLimit.transfers, validate(validateCreateTransfer), asyncHandler(transferController.createTransfer) ); @@ -27,15 +29,35 @@ router.get('/stats', requireScope(['transfers:read']), asyncHandler(transferCont router.get('/:id', requireScope(['transfers:read']), asyncHandler(transferController.getTransfer)); // POST /api/transfers/:id/claim -router.post('/:id/claim', requireScope(['transfers:write']), asyncHandler(transferController.claimTransfer)); +router.post( + '/:id/claim', + requireScope(['transfers:write']), + mutationRateLimit.transfers, + asyncHandler(transferController.claimTransfer) +); // POST /api/transfers/:id/cancel -router.post('/:id/cancel', requireScope(['transfers:write']), asyncHandler(transferController.cancelTransfer)); +router.post( + '/:id/cancel', + requireScope(['transfers:write']), + mutationRateLimit.transfers, + asyncHandler(transferController.cancelTransfer) +); // POST /api/transfers/:id/archive -router.post('/:id/archive', requireScope(['transfers:write']), asyncHandler(transferController.archiveTransfer)); +router.post( + '/:id/archive', + requireScope(['transfers:write']), + mutationRateLimit.transfers, + asyncHandler(transferController.archiveTransfer) +); // POST /api/transfers/:id/unarchive -router.post('/:id/unarchive', requireScope(['transfers:write']), asyncHandler(transferController.unarchiveTransfer)); +router.post( + '/:id/unarchive', + requireScope(['transfers:write']), + mutationRateLimit.transfers, + asyncHandler(transferController.unarchiveTransfer) +); module.exports = router; diff --git a/src/routes/userRoutes.js b/src/routes/userRoutes.js index 1e82221..00b85ac 100644 --- a/src/routes/userRoutes.js +++ b/src/routes/userRoutes.js @@ -4,6 +4,7 @@ const express = require('express'); const asyncHandler = require('../utils/asyncHandler'); const validate = require('../middleware/validate'); const requireScope = require('../middleware/requireScope'); +const mutationRateLimit = require('../middleware/mutationRateLimit'); const userController = require('../controllers/userController'); const { validateCreateUser } = require('../validators/userValidator'); @@ -19,6 +20,7 @@ router.get('/:id', requireScope(['users:read']), asyncHandler(userController.get router.post( '/', requireScope(['users:write']), + mutationRateLimit.users, validate(validateCreateUser), asyncHandler(userController.createUser) ); diff --git a/src/utils/clientIdentity.js b/src/utils/clientIdentity.js new file mode 100644 index 0000000..68c51d2 --- /dev/null +++ b/src/utils/clientIdentity.js @@ -0,0 +1,69 @@ +'use strict'; + +const crypto = require('crypto'); + +/** + * Client identity helpers for abuse controls. + * + * Keys and correlation values must never embed raw API tokens, admin keys, + * account numbers, or other secrets. Fingerprints are one-way and truncated + * so they are useful for isolation without being reversible. + */ + +const FINGERPRINT_CHARS = 16; + +/** + * One-way, truncated fingerprint of a secret string. + * @param {string} value + * @returns {string} + */ +function fingerprint(value) { + if (typeof value !== 'string' || value.length === 0) { + return 'anon'; + } + return crypto.createHash('sha256').update(value).digest('hex').slice(0, FINGERPRINT_CHARS); +} + +/** + * Resolve the client IP for rate limiting. + * + * When `trustProxy` is false (default), only the direct socket address is + * used. When true, Express resolves `req.ip` from its configured trusted + * proxy hops. The left-most forwarded value is not necessarily trustworthy. + * + * @param {import('express').Request} req + * @param {{ trustProxy?: boolean }} [options] + * @returns {string} + */ +function resolveClientIp(req, options = {}) { + const trustProxy = Boolean(options.trustProxy); + if (trustProxy && typeof req.ip === 'string' && req.ip) { + return req.ip; + } + return (req.socket && req.socket.remoteAddress) || req.ip || 'unknown'; +} + +/** + * Build a stable actor key for mutation rate limiting. + * Prefers an authenticated token fingerprint; falls back to client IP. + * Never returns the raw token or admin key. + * + * @param {import('express').Request} req + * @param {{ trustProxy?: boolean }} [options] + * @returns {string} + */ +function resolveActorKey(req, options = {}) { + if (typeof req.token === 'string' && req.token) { + return `actor:${fingerprint(req.token)}`; + } + if (typeof req.adminToken === 'string' && req.adminToken) { + return `admin:${fingerprint(req.adminToken)}`; + } + return `ip:${resolveClientIp(req, options)}`; +} + +module.exports = { + fingerprint, + resolveClientIp, + resolveActorKey, +}; diff --git a/test/abuseControls.test.js b/test/abuseControls.test.js new file mode 100644 index 0000000..c0f1bbb --- /dev/null +++ b/test/abuseControls.test.js @@ -0,0 +1,577 @@ +'use strict'; + +const { test, beforeEach } = require('node:test'); +const assert = require('node:assert/strict'); +const http = require('node:http'); +const express = require('express'); + +process.env.NODE_ENV = 'test'; + +const rateLimit = require('../src/middleware/rateLimit'); +const requestId = require('../src/middleware/requestId'); +const { + fingerprint, + resolveClientIp, + resolveActorKey, +} = require('../src/utils/clientIdentity'); +const { sanitizeCorrelationId } = require('../src/middleware/requestId'); +const errorHandler = require('../src/middleware/errorHandler'); +const ApiError = require('../src/utils/ApiError'); + +function mockReq(overrides = {}) { + const headers = { ...(overrides.headers || {}) }; + return { + ip: overrides.ip || '127.0.0.1', + socket: { remoteAddress: overrides.remoteAddress || '127.0.0.1' }, + token: overrides.token, + adminToken: overrides.adminToken, + get(name) { + const key = Object.keys(headers).find( + (k) => k.toLowerCase() === String(name).toLowerCase() + ); + return key ? headers[key] : undefined; + }, + ...overrides, + }; +} + +function mockRes() { + const headers = {}; + return { + headers, + set(name, value) { + headers[name] = String(value); + }, + statusCode: 200, + status(code) { + this.statusCode = code; + return this; + }, + json(body) { + this.body = body; + return this; + }, + }; +} + +function run(middleware, req, res) { + return new Promise((resolve, reject) => { + middleware(req, res, (err) => { + if (err) { + resolve({ err, res }); + } else { + resolve({ err: null, res }); + } + }); + }); +} + +test('fingerprint is stable, truncated, and never echoes the secret', () => { + const a = fingerprint('test-token-admin'); + const b = fingerprint('test-token-admin'); + assert.equal(a, b); + assert.equal(a.length, 16); + assert.ok(!a.includes('test-token')); + assert.equal(fingerprint(''), 'anon'); +}); + +test('resolveClientIp ignores X-Forwarded-For unless trustProxy is on', () => { + const req = mockReq({ + remoteAddress: '10.0.0.5', + // Express trust proxy 1 resolves the nearest untrusted hop. + ip: '10.0.0.1', + headers: { 'X-Forwarded-For': '203.0.113.9, 10.0.0.1' }, + }); + assert.equal(resolveClientIp(req, { trustProxy: false }), '10.0.0.5'); + assert.equal(resolveClientIp(req, { trustProxy: true }), '10.0.0.1'); +}); + +test('resolveActorKey prefers token fingerprint over IP', () => { + const req = mockReq({ + token: 'test-token-admin', + remoteAddress: '10.0.0.5', + }); + const key = resolveActorKey(req); + assert.match(key, /^actor:/); + assert.ok(!key.includes('test-token-admin')); + assert.ok(!key.includes('10.0.0.5')); +}); + +test('sanitizeCorrelationId rejects oversized or unsafe values', () => { + assert.equal(sanitizeCorrelationId('abc-123'), 'abc-123'); + assert.equal(sanitizeCorrelationId(' ok_id '), 'ok_id'); + assert.equal(sanitizeCorrelationId('has space'), null); + assert.equal(sanitizeCorrelationId('bad"quote'), null); + assert.equal(sanitizeCorrelationId('x'.repeat(200)), null); + assert.equal(sanitizeCorrelationId(''), null); +}); + +test('burst over max returns 429 with Retry-After and policy details', async () => { + const limiter = rateLimit({ + name: 'mutation:transfers', + windowMs: 60_000, + max: 3, + forceInTest: true, + keyGenerator: () => 'actor:one', + }); + + for (let i = 0; i < 3; i += 1) { + const { err } = await run(limiter, mockReq(), mockRes()); + assert.equal(err, null); + } + + const { err, res } = await run(limiter, mockReq(), mockRes()); + assert.ok(err instanceof ApiError); + assert.equal(err.statusCode, 429); + assert.equal(err.details.policy, 'mutation:transfers'); + assert.equal(err.details.limit, 3); + assert.ok(err.details.retryAfter >= 1); + assert.equal(res.headers['Retry-After'], String(err.details.retryAfter)); + assert.equal(res.headers['X-RateLimit-Remaining'], '0'); +}); + +test('identity isolation: one actor bursting does not block another', async () => { + const limiter = rateLimit({ + name: 'mutation:transfers', + windowMs: 60_000, + max: 2, + forceInTest: true, + keyGenerator: (req) => resolveActorKey(req), + }); + + const actorA = mockReq({ token: 'token-a' }); + const actorB = mockReq({ token: 'token-b' }); + + assert.equal((await run(limiter, actorA, mockRes())).err, null); + assert.equal((await run(limiter, actorA, mockRes())).err, null); + const blocked = await run(limiter, actorA, mockRes()); + assert.equal(blocked.err.statusCode, 429); + + const other = await run(limiter, actorB, mockRes()); + assert.equal(other.err, null); +}); + +test('proxy trust: forged X-Forwarded-For cannot rotate identity when trustProxy is false', async () => { + const limiter = rateLimit({ + name: 'global', + windowMs: 60_000, + max: 2, + forceInTest: true, + trustProxy: false, + }); + + const first = mockReq({ + remoteAddress: '10.0.0.5', + headers: { 'X-Forwarded-For': '198.51.100.1' }, + }); + const second = mockReq({ + remoteAddress: '10.0.0.5', + headers: { 'X-Forwarded-For': '198.51.100.2' }, + }); + + assert.equal((await run(limiter, first, mockRes())).err, null); + assert.equal((await run(limiter, first, mockRes())).err, null); + const blocked = await run(limiter, second, mockRes()); + assert.equal(blocked.err.statusCode, 429); +}); + +test('proxy trust: forged left-most X-Forwarded-For cannot rotate a one-hop identity', async () => { + const app = express(); + app.set('trust proxy', 1); + app.use(rateLimit({ + name: 'global', + windowMs: 60_000, + max: 1, + forceInTest: true, + trustProxy: true, + })); + app.get('/ping', (_req, res) => res.json({ ok: true })); + app.use(errorHandler); + + const server = http.createServer(app); + await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve)); + const { port } = server.address(); + + try { + async function ping(forwarded) { + return fetch(`http://127.0.0.1:${port}/ping`, { + headers: { 'X-Forwarded-For': forwarded }, + }); + } + const first = await ping('198.51.100.1, 203.0.113.9'); + assert.equal(first.status, 200); + const forged = await ping('198.51.100.2, 203.0.113.9'); + assert.equal(forged.status, 429); + const other = await ping('198.51.100.2, 203.0.113.10'); + assert.equal(other.status, 200); + } finally { + await new Promise((resolve) => server.close(resolve)); + } +}); + +test('maxKeys capacity preserves active budgets under identity flood', async () => { + const limiter = rateLimit({ + name: 'global', + windowMs: 60_000, + max: 1, + maxKeys: 5, + forceInTest: true, + keyGenerator: (req) => req.actor, + }); + + for (let i = 0; i < 5; i += 1) { + const req = mockReq(); + req.actor = `actor-${i}`; + assert.equal((await run(limiter, req, mockRes())).err, null); + } + + const newcomer = mockReq(); + newcomer.actor = 'actor-new'; + const full = await run(limiter, newcomer, mockRes()); + assert.equal(full.err.statusCode, 429); + assert.ok(Number(full.res.headers['Retry-After']) >= 1); + assert.equal(limiter.size(), 5); + + const original = mockReq(); + original.actor = 'actor-0'; + assert.equal((await run(limiter, original, mockRes())).err.statusCode, 429); +}); + +test('an unchanged full table has bounded traversal work during a rejection burst', async (t) => { + t.mock.method(Date, 'now', () => 1_000); + const NativeMap = global.Map; + let visited = 0; + function* countEntries(iterator) { + for (const entry of iterator) { + visited += 1; + yield entry; + } + } + class CountingMap extends NativeMap { + [Symbol.iterator]() { return countEntries(super[Symbol.iterator]()); } + entries() { return countEntries(super.entries()); } + values() { return countEntries(super.values()); } + keys() { return countEntries(super.keys()); } + forEach(callback, thisArg) { + return super.forEach((value, key) => { + visited += 1; + callback.call(thisArg, value, key, this); + }); + } + } + + const maxKeys = 256; + let limiter; + // Observe collection work only in this limiter. Restore the constructor + // before any asynchronous request so unrelated code keeps the native Map. + try { + global.Map = CountingMap; + limiter = rateLimit({ + windowMs: 60_000, + max: 1, + maxKeys, + forceInTest: true, + keyGenerator: (req) => req.actor, + }); + } finally { + global.Map = NativeMap; + } + + for (let i = 0; i < maxKeys; i += 1) { + assert.equal((await run(limiter, { actor: `known-${i}` }, mockRes())).err, null); + } + visited = 0; + for (let i = 0; i < 128; i += 1) { + const { err, res } = await run(limiter, { actor: `new-${i}` }, mockRes()); + assert.equal(err.statusCode, 429); + assert.equal(res.headers['Retry-After'], '60'); + } + assert.equal(limiter.size(), maxKeys); + assert.equal((await run(limiter, { actor: 'known-0' }, mockRes())).err.statusCode, 429); + // Permit one lazy scan, but reject repeated work proportional to the table + // size for a burst that cannot release or consume any slot. + assert.ok(visited <= maxKeys, `${visited} entries visited during unchanged-table rejection`); +}); + +test('staggered expiry avoids rescanning live budgets at steady-state capacity', async (t) => { + let now = 0; + t.mock.method(Date, 'now', () => now); + const NativeMap = global.Map; + let visited = 0; + class CountingMap extends NativeMap { + *[Symbol.iterator]() { + for (const entry of super[Symbol.iterator]()) { + visited += 1; + yield entry; + } + } + } + + const maxKeys = 256; + let limiter; + try { + global.Map = CountingMap; + limiter = rateLimit({ + windowMs: maxKeys * 1000, max: 1, maxKeys, forceInTest: true, + keyGenerator: (req) => req.actor, + }); + } finally { + global.Map = NativeMap; + } + const request = (actor) => run(limiter, { actor }, mockRes()); + for (let i = 0; i < maxKeys; i += 1) { + now = i * 1000; + assert.equal((await request(`known-${i}`)).err, null); + } + + visited = 0; + for (let i = 0; i < 128; i += 1) { + now = (maxKeys + i) * 1000; + assert.equal((await request(`new-${i}`)).err, null); + assert.equal(limiter.size(), maxKeys); + } + const survivor = await request(`known-${maxKeys - 1}`); + assert.equal(survivor.err.statusCode, 429); + assert.equal(survivor.res.headers['Retry-After'], '128'); + // Allow work proportional to the released identities, not a complete + // live-table traversal for each individual expiry. + assert.ok(visited <= maxKeys, `${visited} entries visited for 128 staggered expiries`); +}); + +test('capacity retry deadlines round up and release only expired budgets', async (t) => { + let now = 1_000; + t.mock.method(Date, 'now', () => now); + const limiter = rateLimit({ + windowMs: 10_000, max: 1, maxKeys: 2, forceInTest: true, + keyGenerator: (req) => req.actor, + }); + const request = (actor) => run(limiter, { actor }, mockRes()); + assert.equal((await request('a')).err, null); + now = 5_000; + assert.equal((await request('b')).err, null); + + now = 9_500; + let blocked = await request('c'); + assert.equal(blocked.err.statusCode, 429); + assert.equal(blocked.res.headers['Retry-After'], '2'); + now = 10_001; + blocked = await request('c'); + assert.equal(blocked.res.headers['Retry-After'], '1'); + + now = 11_000; + assert.equal((await request('a')).err, null); + assert.equal(limiter.size(), 2); + blocked = await request('b'); + assert.equal(blocked.err.statusCode, 429); + assert.equal(blocked.res.headers['Retry-After'], '4'); + now = 15_000; + assert.equal((await request('c')).err, null); + assert.equal((await request('a')).err.statusCode, 429); + assert.equal(limiter.size(), 2); +}); + +test('renewal below capacity and reset keep retry deadlines correct after clock rollback', async (t) => { + let now = 1_000; + t.mock.method(Date, 'now', () => now); + const limiter = rateLimit({ + windowMs: 10_000, max: 1, maxKeys: 2, forceInTest: true, + keyGenerator: (req) => req.actor, + }); + const request = (actor) => run(limiter, { actor }, mockRes()); + assert.equal((await request('a')).err, null); + now = 11_000; + assert.equal((await request('a')).err, null); + assert.equal((await request('b')).err, null); + now = 10_000; + let blocked = await request('c'); + assert.equal(blocked.err.statusCode, 429); + assert.equal(blocked.res.headers['Retry-After'], '11'); + + now = 20_000; + limiter.reset(); + assert.equal(limiter.size(), 0); + assert.equal((await request('a')).err, null); + assert.equal((await request('b')).err, null); + blocked = await request('c'); + assert.equal(blocked.err.statusCode, 429); + assert.equal(blocked.res.headers['Retry-After'], '10'); +}); + +test('a newly admitted earlier window becomes the capacity deadline after clock rollback', async (t) => { + let now = 11_000; + t.mock.method(Date, 'now', () => now); + const limiter = rateLimit({ + windowMs: 10_000, max: 1, maxKeys: 2, forceInTest: true, + keyGenerator: (req) => req.actor, + }); + const request = (actor) => run(limiter, { actor }, mockRes()); + assert.equal((await request('a')).err, null); + now = 1_000; + assert.equal((await request('b')).err, null); + const blocked = await request('c'); + assert.equal(blocked.err.statusCode, 429); + assert.equal(blocked.res.headers['Retry-After'], '10'); + now = 11_000; + assert.equal((await request('c')).err, null); + assert.equal((await request('a')).err.statusCode, 429); +}); + +test('whole-table expiry clears only after the latest deadline and resets cleanly', async (t) => { + let now = 1_000; + t.mock.method(Date, 'now', () => now); + const limiter = rateLimit({ + windowMs: 10_000, max: 1, maxKeys: 3, forceInTest: true, + keyGenerator: (req) => req.actor, + }); + const request = (actor) => run(limiter, { actor }, mockRes()); + for (const key of ['a', 'b', 'c']) assert.equal((await request(key)).err, null); + now = 10_999; + assert.equal((await request('d')).err.statusCode, 429); + now = 11_000; + assert.equal((await request('d')).err, null); + assert.equal(limiter.size(), 1); + + // A newer insertion after a clock rollback must not erase d's live budget. + now = 1_000; + for (const key of ['e', 'f']) assert.equal((await request(key)).err, null); + now = 11_000; + assert.equal((await request('g')).err, null); + assert.equal(limiter.size(), 2); + assert.equal((await request('d')).err.statusCode, 429); + now = 21_000; + assert.equal((await request('h')).err, null); + assert.equal(limiter.size(), 1); + + limiter.reset(); + now = 0; + assert.equal((await request('i')).err, null); + now = 10_000; + assert.equal((await request('j')).err, null); + assert.equal(limiter.size(), 1); +}); + +test('correlation id is echoed on success and on 429 without leaking tokens', async () => { + const app = express(); + app.use(requestId); + app.use( + rateLimit({ + name: 'mutation:quote', + windowMs: 60_000, + max: 1, + forceInTest: true, + keyGenerator: () => 'ip:test', + }) + ); + app.get('/quote', (req, res) => { + res.json({ ok: true, correlationId: req.correlationId }); + }); + app.use(errorHandler); + + const server = http.createServer(app); + await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve)); + const { port } = server.address(); + const base = `http://127.0.0.1:${port}`; + + try { + const first = await fetch(`${base}/quote`, { + headers: { 'X-Correlation-Id': 'client-corr-1' }, + }); + assert.equal(first.status, 200); + assert.equal(first.headers.get('x-correlation-id'), 'client-corr-1'); + assert.equal(first.headers.get('x-request-id'), 'client-corr-1'); + const firstBody = await first.json(); + assert.equal(firstBody.correlationId, 'client-corr-1'); + + const second = await fetch(`${base}/quote`, { + headers: { + 'X-Correlation-Id': 'client-corr-2', + Authorization: 'Bearer test-token-admin', + }, + }); + assert.equal(second.status, 429); + assert.equal(second.headers.get('x-correlation-id'), 'client-corr-2'); + assert.ok(second.headers.get('retry-after')); + const body = await second.json(); + assert.equal(body.error.status, 429); + assert.equal(body.error.requestId, 'client-corr-2'); + assert.equal(body.error.details.policy, 'mutation:quote'); + const dumped = JSON.stringify(body); + assert.ok(!dumped.includes('test-token-admin')); + } finally { + await new Promise((resolve) => server.close(resolve)); + } +}); + +test('unsafe inbound correlation id is replaced, not echoed', async () => { + const app = express(); + app.use(requestId); + app.get('/ping', (req, res) => res.json({ id: req.id })); + const server = http.createServer(app); + await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve)); + const { port } = server.address(); + + try { + const res = await fetch(`http://127.0.0.1:${port}/ping`, { + headers: { 'X-Request-Id': 'not a safe id' }, + }); + const body = await res.json(); + assert.notEqual(body.id, 'not a safe id'); + assert.equal(res.headers.get('x-request-id'), body.id); + assert.equal(res.headers.get('x-correlation-id'), body.id); + } finally { + await new Promise((resolve) => server.close(resolve)); + } +}); + +test('actual app counts parser failures in the global budget and correlates errors', async () => { + const config = require('../src/config'); + const createApp = require('../src/app'); + const originalMax = config.rateLimit.max; + const originalBodyLimit = config.bodyLimit; + const originalEnableInTest = process.env.ENABLE_RATE_LIMIT_IN_TEST; + let server; + + try { + config.rateLimit.max = 2; + config.bodyLimit = '32b'; + process.env.ENABLE_RATE_LIMIT_IN_TEST = '1'; + server = http.createServer(createApp()); + await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve)); + const base = `http://127.0.0.1:${server.address().port}`; + const malformed = '{"amount":'; + const oversized = JSON.stringify({ note: 'x'.repeat(64) }); + const responses = []; + + for (const [index, payload] of [malformed, oversized, malformed, oversized].entries()) { + const id = `parser-budget-${index}`; + const response = await fetch(`${base}/api/transfers`, { + method: 'POST', + headers: { 'Content-Type': 'application/json', 'X-Request-Id': id }, + body: payload, + }); + responses.push({ id, response, body: await response.json() }); + } + + assert.deepEqual(responses.map(({ response }) => response.status), [400, 413, 429, 429]); + for (const { id, response, body } of responses) { + assert.equal(response.headers.get('x-request-id'), id); + assert.equal(response.headers.get('x-correlation-id'), id); + assert.equal(body.error.requestId, id); + assert.equal(response.headers.get('x-ratelimit-policy'), 'global'); + if (response.status === 429) { + assert.ok(Number(response.headers.get('retry-after')) >= 1); + assert.equal(response.headers.get('x-ratelimit-remaining'), '0'); + assert.equal(body.error.details.policy, 'global'); + } + } + } finally { + config.rateLimit.max = originalMax; + config.bodyLimit = originalBodyLimit; + if (originalEnableInTest === undefined) delete process.env.ENABLE_RATE_LIMIT_IN_TEST; + else process.env.ENABLE_RATE_LIMIT_IN_TEST = originalEnableInTest; + if (server) { + server.closeAllConnections(); + await new Promise((resolve) => server.close(resolve)); + } + } +}); diff --git a/test/rateLimitCapacity.test.js b/test/rateLimitCapacity.test.js new file mode 100644 index 0000000..efa7ac2 --- /dev/null +++ b/test/rateLimitCapacity.test.js @@ -0,0 +1,118 @@ +'use strict'; + +const assert = require('node:assert/strict'); +const http = require('node:http'); +const { once } = require('node:events'); +const { test } = require('node:test'); +const rateLimit = require('../src/middleware/rateLimit'); +const { resolveActorKey } = require('../src/utils/clientIdentity'); + +// Exercise the complete production middleware and actor-key helper over actual +// loopback HTTP. Only the clock, authenticated token assignment and Express +// response/error adapter are controlled; no application/auth integration claim. +async function fixture(t, { maxKeys = 1, stacked = true } = {}) { + const epoch = 1_700_000_000_000; + let now = epoch; + t.mock.method(Date, 'now', () => now); + const outer = rateLimit({ name: 'global', max: 100, windowMs: 60_000, forceInTest: true }); + const inner = rateLimit({ + name: 'mutation:transfers', max: 10, maxKeys, windowMs: 60_000, + forceInTest: true, keyGenerator: resolveActorKey, + }); + const server = http.createServer((req, res) => { + req.token = req.headers['x-fixture-token']; + res.set = (name, value) => { res.setHeader(name, value); return res; }; + const finish = (error) => { + res.statusCode = error ? error.statusCode : 200; + res.setHeader('Content-Type', 'application/json'); + res.end(JSON.stringify(error ? { message: error.message, details: error.details } : { ok: true })); + }; + const route = (error) => { + if (error) finish(error); + else if (req.url === '/global') finish(); + else inner(req, res, finish); + }; + if (stacked) outer(req, res, route); + else route(); + }); + server.listen(0, '127.0.0.1'); + await once(server, 'listening'); + t.after(() => new Promise((resolve, reject) => server.close(error => error ? reject(error) : resolve()))); + + async function request(offset, token, path = '/transfer') { + now = epoch + offset; + return new Promise((resolve, reject) => { + const req = http.get({ + host: '127.0.0.1', port: server.address().port, path, agent: false, + headers: token ? { 'X-Fixture-Token': token } : {}, + }, res => { + let body = ''; + res.setEncoding('utf8'); + res.on('data', chunk => { body += chunk; }); + res.on('end', () => resolve({ status: res.statusCode, headers: res.headers, body: JSON.parse(body) })); + res.on('error', reject); + }); + req.on('error', reject); + }); + } + + function blocked(response, offset, retryAfter) { + assert.equal(response.status, 429); + assert.equal(response.headers['x-ratelimit-policy'], 'mutation:transfers'); + assert.equal(response.headers['x-ratelimit-limit'], '10'); + assert.equal(response.headers['x-ratelimit-remaining'], '0'); + assert.equal(response.headers['retry-after'], String(retryAfter)); + assert.equal(response.body.details.retryAfter, retryAfter); + assert.equal(response.body.details.policy, 'mutation:transfers'); + assert.equal(response.headers['x-ratelimit-reset'], String(Math.ceil((epoch + offset) / 1000))); + } + return { request, blocked }; +} + +test('capacity rejection replaces the outer policy reset and admits at its own deadline', async t => { + const { request, blocked } = await fixture(t); + const globalOnly = await request(0, undefined, '/global'); + const first = await request(30_000, 'actor-a'); + const rejected = await request(45_000, 'actor-b'); + const beforeDeadline = await request(89_999, 'actor-b'); + const atDeadline = await request(90_000, 'actor-b'); + + assert.equal(globalOnly.status, 200); + assert.equal(globalOnly.headers['x-ratelimit-reset'], '1700000060'); + assert.equal(first.status, 200); + assert.equal(first.headers['x-ratelimit-reset'], '1700000090'); + assert.equal(atDeadline.status, 200); + assert.equal(atDeadline.headers['x-ratelimit-remaining'], '9'); + blocked(rejected, 90_000, 45); + blocked(beforeDeadline, 90_000, 1); +}); + +test('standalone capacity rejection includes the rounded-up expiry header', async t => { + const { request, blocked } = await fixture(t, { stacked: false }); + const first = await request(1_250, 'actor-a'); + const rejected = await request(2_000, 'actor-b'); + const beforeDeadline = await request(61_249, 'actor-b'); + const atDeadline = await request(61_250, 'actor-b'); + + assert.equal(first.status, 200); + assert.equal(atDeadline.status, 200); + assert.equal(atDeadline.headers['x-ratelimit-remaining'], '9'); + blocked(rejected, 61_250, 60); + blocked(beforeDeadline, 61_250, 1); +}); + +test('capacity reset advances to the next surviving expiry after a slot is reused', async t => { + const { request, blocked } = await fixture(t, { maxKeys: 2 }); + const first = await request(2_000, 'actor-a'); + const second = await request(10_000, 'actor-b'); + const rejected = await request(15_000, 'actor-c'); + const reused = await request(62_000, 'actor-c'); + const nextRejected = await request(62_500, 'actor-a'); + + assert.equal(first.status, 200); + assert.equal(second.status, 200); + assert.equal(reused.status, 200); + assert.equal(reused.headers['x-ratelimit-remaining'], '9'); + blocked(rejected, 62_000, 47); + blocked(nextRejected, 70_000, 8); +}); diff --git a/test/requestIdCredentials.test.js b/test/requestIdCredentials.test.js new file mode 100644 index 0000000..796bc0a --- /dev/null +++ b/test/requestIdCredentials.test.js @@ -0,0 +1,79 @@ +'use strict'; + +const { test } = require('node:test'); +const assert = require('node:assert/strict'); +const requestId = require('../src/middleware/requestId'); + +function invoke(headers) { + const sent = {}; + const req = { get: (name) => headers[name.toLowerCase()] }; + let calls = 0; + requestId(req, { set: (name, value) => { sent[name] = value; } }, () => { calls += 1; }); + assert.equal(calls, 1); + assert.equal(req.id, req.correlationId); + assert.equal(sent['X-Request-Id'], req.id); + assert.equal(sent['X-Correlation-Id'], req.id); + return req.id; +} + +const uuid = /^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/; + +test('presented Bearer credentials cannot become correlation identifiers', () => { + const token = 'example-secret-token.42'; + for (const authorization of [`Bearer ${token}`, `bearer ${token}`]) { + for (const value of [token, `trace:${token}:retry`]) { + for (const header of ['x-request-id', 'x-correlation-id']) { + const id = invoke({ authorization, [header]: value }); + assert.ok(!id.includes(token)); + assert.match(id, uuid); + } + } + } +}); + +test('admin credentials are excluded even when another Bearer token is supplied', () => { + const token = 'example-admin-secret'; + for (const value of [token, `retry:${token}`]) { + const id = invoke({ + authorization: 'Bearer example-user-secret', + 'x-admin-token': token, + 'x-request-id': value, + 'x-correlation-id': 'example-user-secret', + }); + assert.match(id, uuid); + assert.ok(!id.includes(token)); + assert.ok(!id.includes('example-user-secret')); + } +}); + +test('an independent fallback is retained when the preferred id contains a credential', () => { + assert.equal(invoke({ + authorization: 'Bearer example-token', + 'x-request-id': 'example-token', + 'x-correlation-id': 'order-123:attempt-2', + }), 'order-123:attempt-2'); +}); + +test('ordinary identifier precedence and syntax handling are unchanged', () => { + assert.equal(invoke({ + authorization: 'Bearer example-token', + 'x-admin-token': 'example-admin', + 'x-request-id': ' upstream.123 ', + 'x-correlation-id': 'other-id', + }), 'upstream.123'); + assert.equal(invoke({ 'x-request-id': 'unsafe value', 'x-correlation-id': 'safe-id' }), 'safe-id'); + assert.equal(invoke({ 'x-request-id': 'x'.repeat(128) }), 'x'.repeat(128)); + for (const value of ['x'.repeat(129), 'has\ncontrol', 'has space', '"quoted"', ['array']]) { + assert.match(invoke({ 'x-request-id': value }), uuid); + } + assert.equal(requestId.sanitizeCorrelationId('trace-12'), 'trace-12'); +}); + +test('replacement identifiers are fresh for separate requests', () => { + const headers = { authorization: 'Bearer example-token', 'x-request-id': 'example-token' }; + const first = invoke(headers); + const second = invoke(headers); + assert.match(first, uuid); + assert.match(second, uuid); + assert.notEqual(first, second); +});