Repository navigation
fix(rate-limit): key all 21 Cloudflare-blind limiters on the client address, guarded by a behavioural probe (TASK-126) - #1887
Merged
Conversation
21 sites in 18 files fell back to `req.ip`: 20 via `ipKeyGenerator(req.ip)`,
and routes/agentHooks.ts:29 on a raw `` `ip:${req.ip || 'unknown'}` `` that no
spelling-based census could see. Behind nginx `req.ip` is a cloudflared pod
address for every external caller, so each of these was one limiter bucket
serving the whole internet. All 21 now call cloudflareIpRateLimitKeyGenerator,
which prefers `cf-connecting-ip` and falls back to `req.ip` only when that
header is absent — unchanged behaviour for a caller that sends none.
The guard for this is a behavioural probe over the loaded app's router stack
(ruled 2026-09-25): two requests differing only in `cf-connecting-ip` must not
share a bucket. A source scan is deliberately NOT the guard here — the census
that missed routes/agentHooks.ts:29 enumerated a spelling and called it the
class, and the same mistake is available to the next scan.
Verified: backend suite 429 suites / 4019 tests; lint:ts 0 errors. The site list
was derived twice, independently (kai from nearest-declaration + brace depth,
vera from nearest-preceding `const <name> =`), and the two agreed exactly.
(cherry picked from commit 661d3b0)
…cting-ip callers
TASK-126. Every limiter instance found in the loaded app is re-mounted alone on a
stub with a 204 terminal and exercised with A, A, B requests differing only in
`cf-connecting-ip`. Fails on main; passes on this branch.
Why a stub instead of requests through the app — each reason measured:
- limiters are identified by identity (`resetKey`/`getKey` own properties),
because EVERY router-stack entry has `handle.name === ''`;
- on a route carrying two limiters both write `RateLimit-Remaining` and the last
one wins, so earlier limiters are invisible: on
`POST /api/webhooks/groupme/:integrationId` the 3000-max IP limiter reported the
600-max limiter's values, and the walk entry for the 600-max limiter then read
596, because the first entry's three requests had decremented its bucket;
- a limiter mounted behind `auth` never runs for a header-free request;
- three instances never answer through the app at all (their handlers need Mongo).
On the stub no param and no auth header reaches a key function, so every limiter
takes its IP branch and every one of them answers.
A2 is the instrument's own control: two requests cannot separate "different
buckets" from "not counting at all". The fixtures prove the instrument
discriminates on the stub itself — a raw-`req.ip` limiter must COLLAPSE
(r3 === r2 - 1, a third hit in the same bucket) and a CF-aware one must SEPARATE
(r3 === r1). Without them a stub that separated everything would pass all
instances and establish nothing.
The app only mounts pg-messages/pg-status when PG_HOST is set AND the connect
resolves truthy (server.ts:67,380-400), so the probe presents that shape; an
earlier walk without it saw 82 instances and could not find pg-messages.
PG_HOST and NODE_ENV are restored per file/per request, since process.env is
per worker and a leak would change every later suite's DB path and skips.
Measured: 84 instances mounted, 84 separated, 0 collapsed, 0 unobserved. The 84
is 75 direct instances + 9 factory invocations (platformIpRateLimit x6,
taskWriteRateLimit x3) — 77 `rateLimit({` source sites, two of which are factory
bodies. On main the same file fails over 38 collapsing instances.
(cherry picked from commit 516d04f)
Two ruled disclosures, both about what the probe does NOT prove. The probe's own behaviour is unchanged. wren 73671 (ruling (b)): the assertion now says what the stub measures — a request with no credential, no param and no body, carrying only `cf-connecting-ip`, is bucketed per `cf-connecting-ip` on every instance. Stricter than a per-arm check, because a limiter with no IP arm at all keys that request on a CONSTANT: one bucket for every anonymous caller, the same defect as raw `req.ip` behind cloudflared, and it collapses A and B here identically. vera 73673, trap 2: `users.ts:65` `profileWriteUserLimit` keys on `req.userId || req.user?.id || req.user?._id` first, and `auth` runs before it (`users.ts:171`), so an unauthenticated request never reaches it. Its IP arm is reachable on this stub only. The arm must still be CF-derived, but the walk proves the stub's shape, not liveness — in production that limiter keys per account. Trap 1 needed no code change: the probe builds each request from nothing and sets only `cf-connecting-ip` (vera 73674), so no credential header — `authorization`, `x-auth-token`, `x-commonly-agent-token` — can select a token arm, and `req.agentTokenHash` is unset. (cherry picked from commit d7c27be)
… a witness The 84/84 result is unchanged; what changes is that the instrument can now fail. wren 73692, accepted: with 84 chains started in one tick, only the first reads the suite's NODE_ENV and the other 83 read the 'development' it just set, each restoring what it read — so the walk ended flipped and no per-request check could see it. The walk is now serial, and the assertion that NODE_ENV is where it started runs once after it. Measured: ends at 'test', same 84/84, 323ms against 177ms. The `for...of` and its `await` are eslint-disabled on purpose — the serialisation is the property under test. wren 73694, accepted, second half: the PG_HOST save/restore is gone. setup.js sets PG_HOST to the literal string 'undefined' in in-memory mode, which is truthy, so `ENTRY_PG_HOST || 'probe'` wrote back the value it had just read and changed nothing — the pg-messages mount comes from the connect mocks. Measured after removal: 84 instances, pg-messages still found. Two comments of mine were wrong and are corrected: process.env is per TEST FILE here, not per worker (measured — same pid, writer file first, reader file sees undefined), so the "later suite in this worker" rationale was false. The census dump is now written only when PROBE_CENSUS is set, so CI writes no /tmp file. (cherry picked from commit 005ff9c)
…ness accident The probe mounts pg-messages/pg-status only when PG_HOST is truthy (server.ts:67, 378-400), and until now it inherited that truthiness from setup.js storing the STRING 'undefined'. TASK-128 deletes the key, which would have dropped this probe to 82 instances and quietly falsified its own must-find assertion. Set it here. No restore machinery, and that is a measurement rather than a hope: process.env is per test FILE here, so the assignment cannot reach another suite. Verified in both worlds — 84 instances, 84 separated, pg-messages present with the key set here, including with TASK-128's `delete` applied to setup.js. (cherry picked from commit 6250ebc)
…ctually were Vera 73704, taken rather than refused: the control keyed `(req) => req.ip`, a paraphrase of the defect. The 21 sites were `req.ip ? ipKeyGenerator(req.ip) : 'anon'` (the spelling still at `middleware/ipRateLimit.ts:54`). Both collapse on the stub — no `trust proxy`, so every request carries the loopback socket address — but the bare form also hands express-rate-limit's IPv6 key validation a raw value, so it measured a second thing alongside the collapse. The fixture now reproduces what was there. Vera 73712: the comment cited `server.ts:67`; the gate is on **`:68`** on `origin/main` (`:67` is blank). Inherited from the comment this replaced, and it is in the file under her gate. Probe unchanged: 5/5, 84 instances / 84 separated / 0 collapsed / 0 not-counting / 0 unobserved, both pg-messages routes still found. (cherry picked from commit 1e6beae)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TASK-126 — the swap-all half, as ruled 2026-09-25 (wren: swap all 21 sites, no burn-down baseline, guard is a behavioural probe).
What it does
21 sites in 18 files fell back to
req.ip. Behind nginxreq.ipis a cloudflared pod address for every external caller, so each was one limiter bucket serving the whole internet:ipKeyGenerator(req.ip);routes/agentHooks.ts:29, a raw`ip:${req.ip || 'unknown'}`— the 21st, and the reason the guard is not a source scan. The census that missed it enumerated a spelling (ipKeyGenerator() and called it the class; the same mistake is available to the next scan.All 21 now call
cloudflareIpRateLimitKeyGenerator, which preferscf-connecting-ipand falls back toreq.iponly when that header is absent — unchanged for a caller that sends none.Measured on this branch:
grep -rn 'ipKeyGenerator(' backend --include=*.tsoutsidemiddleware/ipRateLimit.tsreturns 0.The guard is behavioural, deliberately
backend/__tests__/unit/middleware/rateLimitIpKeySeparation.test.jswalks the loaded app's router stack and drives two requests that differ only incf-connecting-ip:every walked instance counts and separates; none collapsesfinds the limiters that mount late or behind a routernegative control: the instrument catches a limiter keyed the way the swapped sites were— the probe is shown to fail on the old shape, so "all separate" is evidence and not a vacuous walkpositive control: the CF-aware key separates on the same stubHow this was cut (the stack, not the branch)
The old branch
1e6beaefwas a 3-deep stack whose merge-base (9a32fca5) predated #1880/#1881/#1883, and it overlapped #1883 on five files (as @vera flagged). This branch is cut fresh from mainb27b5aeawith only:661d3b07, 18 files), andDropped, because they are already in main under their own PRs:
2c37996d(TASK-123 Discord containment → #1867451c9db6) andcdfb0c64(TASK-125 → #1883a7dea33a). That removes the overlap rather than resolving it: nointegrationRateLimit.ts, no test-file collisions, and no baseline json touched (0 files).routes/users.tsmerged without a conflict and is verified in the result: all three sites on the CF generator,:59being the line #1883 merged and:38/:79the two this change adds. Per @vera,users.ts:79is user-first and mounted afterauth— it is inside "all 21" for uniformity, not a fallback any caller reaches.Victim measurement
@vera, live, across all 21 mounts: one live victim and twenty latent.
routes/podInvites.ts:161mountsinviteReadRateLimitwith no auth in the chain (its two sibling mounts,:90and:189, both carryauth), andGET /invites/:token/previewis called for logged-out visitors (V2InviteRedeem.tsx:59) — so all of them share one instance-wide bucket. The four that looked unauthenticated on a first pass (agentBinding.ts:179/:284/:352,machines.ts:99) sit behinddaemonAuth, which readsAuthorization— exactly what both key generators read — so daemons bucket per credential and share nothing.Verification
rateLimitDefaultKey,integrationRateLimit).eslint . --ext .ts→ 0 errors / 1210 warnings, identical to the same-instrument baseline atd2cfcb2f. Delta 0.Gate: @vera (
fix+ tests + the probe's shape).Clearance (vera, 2026-09-25)
13 green at
c621dd92, unconditional, and the piece the gate was holding was the guard's reach: she mutated a real site rather than trusting the synthetic control —routes/podInvites.ts:26to a constant key, in an isolated worktree — and the walk reddened, naming both limiters that key backs (POST /api/pods/:podId/invites [r1=19 r2=18 r3=17]and theGET) with exactly ther3 = r2 - 1collapse signature the probe documents. So the probe catches a regression at a real site, by path, not only in its own fixture.Corrected in passing: the probe runs in ~310ms. Two earlier attempts died in
globalSetupand were misattributed to the walk's length.Press: @connector-ops (asked by the gate). Backend-only, so no image-ordering constraint.