Rate limiting per credential, failed-key throttle, one-worker guard

README.md

@@ -11,6 +11,8 @@ reachable from the network; the PC and the agents reach it through an SSH tunnel
 restore are in [`deploy/mint/README.md`](deploy/mint/README.md). **This table describes the deployed build**, which is `main` as of 2026-10-11 (`5a8d756`, schema v9);
 anything merged after that day is not deployed until the next deploy.
 
+**Schema v10 is in the code, not yet on the box.** `main` is at schema **v10** (the per-tenant API-key control schema, merged 2026-10-16; see [`docs/tenant_keys.md`](docs/tenant_keys.md)). The deployed ledger is still v9: a deploy of `main` runs the v10 migration, which changes no ledger row, and is rehearsed on a throwaway stack first. Per-tenant keys are off by default (`JARVIS_KEY_TENANTS`). The way back is `pg_schema.v10_rollback_sql`.
+
 | Capability | Status |
 |---|---|
 | **Postgres row store**: one row per record, forced row-level security, two roles (the app cannot change history), hash-chained `record_history` | **live** (`docs/POSTGRES.md`) |

app/auth.py

@@ -582,11 +582,15 @@ async def identity_middleware(request: Request, call_next):
         if candidates:
             # A per-tenant API key (JARVIS_KEY_TENANTS): authenticated against the control schema, bound to the key's own tenant. Every way of failing is the one 401 a wrong ledger
             # key gets; an unreachable control schema is a 503 (never an admission); a valid key of a tenant that is not active is a 503 for that tenant only.
+            client_addr = request.client.host if request.client else "unknown"
+            if key_auth.attempts_blocked(candidates, client_addr):
+                return json_response(429, "Too many failed attempts", headers={"Retry-After": "60"}, path=path)
             try:
                 resolved = await run_in_threadpool(key_auth.authenticate, candidates)
             except key_auth.KeyAuthUnavailable:
                 return json_response(503, "Ledger unavailable", path=path)
             if resolved is None:
+                key_auth.record_failure(candidates, client_addr)
                 return json_response(401, "Invalid or missing API key", path=path)
             if resolved.tenant_status != "active":
                 return json_response(503, "tenant_unavailable", path=path)

app/key_auth.py

@@ -41,6 +41,39 @@ def token_candidates(headers) -> list[str]:
     return out
 
 
+def failure_limit() -> int:
+    """Failed key attempts per minute, per (client address, key id), before the key is not even checked (JARVIS_KEY_FAIL_LIMIT_PER_MINUTE)."""
+    try:
+        return max(1, int(os.getenv("JARVIS_KEY_FAIL_LIMIT_PER_MINUTE", "20")))
+    except ValueError:
+        return 20
+
+
+def _failure_subjects(candidates: Iterable[str], client: str) -> list[str]:
+    out = []
+    for token in candidates:
+        parsed = tenant_keys.parse_token(token)
+        # a malformed token has no key id to guess against: it is charged to the address alone
+        out.append(f"keyfail:{client}:{parsed[0] if parsed else '-'}")
+    return out or [f"keyfail:{client}:-"]
+
+
+def attempts_blocked(candidates: Iterable[str], client: str) -> bool:
+    """True when this address has already failed `failure_limit()` times against one of these key ids this minute: the request is refused with 429 BEFORE any HMAC is computed or the
+    database consulted, so guessing one key's secret is throttled. Keyed by address as well as key id, so a stranger who sends junk naming your key id cannot lock you out of your own key."""
+    from app.public_security import key_failure_limiter
+
+    limit = failure_limit()
+    return any(key_failure_limiter.blocked(subject, limit) for subject in _failure_subjects(candidates, client))
+
+
+def record_failure(candidates: Iterable[str], client: str) -> None:
+    from app.public_security import key_failure_limiter
+
+    for subject in _failure_subjects(candidates, client):
+        key_failure_limiter.allow(subject, limit=10**9)      # count it; the decision to refuse is made by attempts_blocked
+
+
 class KeyAuthUnavailable(RuntimeError):
     """The control schema could not be consulted (database down, misconfigured pepper): the request is refused with 503, never admitted."""
 

app/main.py

@@ -107,7 +107,7 @@
 from app import call_log, drift, key_auth, pg_store
 from app.call_witness import CallWitnessMiddleware
 from app.oauth import protected_resource_metadata
-from app.public_security import cors_origins, public_security_middleware
+from app.public_security import assert_config as assert_rate_limit_config, cors_origins, public_security_middleware
 from app.refusal import DENIED, LEDGER_UNAVAILABLE, VERSION_CONFLICT, json_response, retry_after_seconds
 from app.emr_latest import AUTHORITY_DENIED, LatestError, LatestParams, latest_memories, parse_limit
 from app.ledger_search import SearchParams, search_ledger
@@ -142,6 +142,7 @@ async def lifespan(app: FastAPI):
     """Application lifespan: track and cleanup background nx-search processes."""
     # Store for background processes
     assert_multi_tenant_config()  # JARVIS_MULTI_TENANT: refuse to start with a configuration that could fail open
+    assert_rate_limit_config()  # per-process limiters: more than one worker multiplies every quota, so refuse unless a shared limiter is declared
     if key_auth.key_tenants_enabled():
         key_auth.assert_config()  # JARVIS_KEY_TENANTS: needs the row store and a strong pepper
         pg_store.assert_key_auth_ready()  # ... and the control schema, reachable by the application role through the two lookup functions

app/public_security.py

@@ -3,6 +3,7 @@
 from __future__ import annotations
 
 import os
+import threading
 import time
 from collections import defaultdict, deque
 
@@ -27,24 +28,88 @@ def trusted_hosts() -> set[str]:
 
 
 class McpRateLimiter:
-    """Process-local backstop; the edge/WAF remains the primary limiter."""
+    """Process-local fixed-window limiter (the edge/WAF remains the primary limiter). Buckets idle for a whole window are evicted, so a stream of distinct subjects cannot grow the table forever."""
 
-    def __init__(self) -> None:
+    MAX_BUCKETS = 10_000
+
+    def __init__(self, window: float = 60.0) -> None:
         self._requests: dict[str, deque[float]] = defaultdict(deque)
+        self._window = window
+        self._lock = threading.Lock()
 
-    def allow(self, client: str) -> bool:
-        limit = int(os.getenv("JARVIS_MCP_RATE_LIMIT_PER_MINUTE", "60"))
+    def allow(self, subject: str, limit: int | None = None) -> bool:
+        limit = int(os.getenv("JARVIS_MCP_RATE_LIMIT_PER_MINUTE", "60")) if limit is None else limit
+        now = time.monotonic()
+        with self._lock:
+            if len(self._requests) >= self.MAX_BUCKETS and subject not in self._requests:
+                self._evict(now)
+            window = self._requests[subject]
+            while window and now - window[0] >= self._window:
+                window.popleft()
+            if len(window) >= limit:
+                return False
+            window.append(now)
+            return True
+
+    def blocked(self, subject: str, limit: int) -> bool:
+        """Is `subject` already at `limit` in the current window? (Does not count this call.)"""
         now = time.monotonic()
-        window = self._requests[client]
-        while window and now - window[0] >= 60:
-            window.popleft()
-        if len(window) >= limit:
-            return False
-        window.append(now)
-        return True
+        with self._lock:
+            window = self._requests.get(subject)
+            if window is None:
+                return False
+            while window and now - window[0] >= self._window:
+                window.popleft()
+            return len(window) >= limit
+
+    def _evict(self, now: float) -> None:
+        for key in [k for k, w in self._requests.items() if not w or now - w[-1] >= self._window]:
+            del self._requests[key]
+        if len(self._requests) >= self.MAX_BUCKETS:             # every bucket is live: drop the oldest-touched half rather than refuse to track anyone new
+            for key, _ in sorted(self._requests.items(), key=lambda kv: kv[1][-1])[: self.MAX_BUCKETS // 2]:
+                del self._requests[key]
 
 
 mcp_rate_limiter = McpRateLimiter()
+key_failure_limiter = McpRateLimiter()          # failed tenant-key attempts, keyed (client address, key id); see key_auth.attempts_blocked
+
+
+def rate_limit_subject(request: Request) -> str:
+    """Who a request is charged to: its authenticated credential when the identity layer has bound one (a tenant key or OAuth subject is a tenant of its own; the operator is one subject),
+    else the client address. Tenants behind one proxy address therefore no longer share a bucket. This runs inside the identity middleware, so a failed authentication never reaches it
+    (see key_auth.failure_limiter for those)."""
+    from app.identity import current_principal
+
+    principal = current_principal()
+    if principal is not None:
+        if principal.kind == "operator":
+            return "operator"
+        who = principal.credential_id or principal.tenant_key or principal.subject
+        return f"tenant:{principal.issuer}:{who}"
+    return "ip:" + (request.client.host if request.client else "unknown")
+
+
+def worker_count() -> int:
+    for name in ("WEB_CONCURRENCY", "UVICORN_WORKERS", "GUNICORN_WORKERS"):
+        raw = (os.getenv(name) or "").strip()
+        if raw.isdigit():
+            return int(raw)
+    return 1
+
+
+def config_problems() -> list[str]:
+    """The limiters are per process. With more than one worker each tenant's quota silently multiplies by the worker count, so that is a startup refusal unless a shared backend is declared
+    (JARVIS_RATE_LIMIT_SHARED_BACKEND names one; the limiters here do not implement it, so declaring it states that the edge enforces the real limit)."""
+    if worker_count() > 1 and not (os.getenv("JARVIS_RATE_LIMIT_SHARED_BACKEND") or "").strip():
+        return [f"{worker_count()} workers are configured but the rate limiters are per process (each tenant's quota would multiply by the worker count): run one worker, "
+                "or set JARVIS_RATE_LIMIT_SHARED_BACKEND to name the shared limiter that enforces the real limit"]
+    return []
+
+
+def assert_config() -> None:
+    problems = config_problems()
+    if problems:
+        raise RuntimeError("rate limiting refuses to start: " + "; ".join(problems))
 
 
 async def public_security_middleware(request: Request, call_next):
@@ -54,8 +119,7 @@ async def public_security_middleware(request: Request, call_next):
         if not allowed_hosts or request_host not in allowed_hosts:
             return JSONResponse(status_code=421, content={"detail": "Untrusted Host header"})
         if request.url.path.startswith("/mcp"):
-            client = request.client.host if request.client else "unknown"
-            if not mcp_rate_limiter.allow(client):
+            if not mcp_rate_limiter.allow(rate_limit_subject(request)):
                 return JSONResponse(status_code=429, content={"detail": "MCP rate limit exceeded"})
 
     response = await call_next(request)

deploy/mint/README.md

@@ -35,6 +35,7 @@ Windows PC ──ssh -L──►  127.0.0.1:8011  ─►  app (non-root, read-on
 * **Deployed on 2026-10-08 and 2026-10-10:** schema **v7** (the signature tables) on 2026-10-08; schema **v8** and **v9** on 2026-10-10 (`emr_latest`'s
   supersedes index and the ledger-search index; `main` at `792e99b`), with `emr_latest`, `emr_search_ledger`, the nx-search bridge and the removal of
   the digital twin line. The rollback for 2026-10-10 is the named backup restored under the `pre-v8v9` images; there is no down-migration for V8.
+* **Schema v10 (merged 2026-10-16, not deployed):** the control schema `<schema>_control` (tenants and per-tenant API keys; `docs/tenant_keys.md`). It touches no ledger row and no hash; a v9 application refuses a v10 database and the reverse. The box stays at v9 until a deploy of `main`, which needs the migration rehearsed on a throwaway restore first (the 2026-10-16 drill did this: `~/jarvis-ledger/drills/2026-10-16-keys/`). Down-migration: `app.pg_schema.v10_rollback_sql(schema)` as the migration role (drops the control schema and the version row; no ledger data changes).
 * **Deployed on 2026-10-11:** no schema change (still **v9**). The server-side **call log** (`docs/call_log.md`; on, `JARVIS_CALL_LOG_ENABLED=1`, in the appdata
   volume and so in the backups) and the **drift watcher** (`docs/drift_watcher.md`; on, `JARVIS_DRIFT_CHECK_ENABLED=1`, operator-only, results go to its own
   log, never to the ledger); `main` at `5a8d756`, an app-only swap. The rollback is the `pre-drift` image tags (remove the flag line, recreate the app only).
@@ -225,7 +226,7 @@ Backup first (`jarvisctl backup`, then `jarvisctl drill`), then pull and `jarvis
 schema version exactly, so **v5 code refuses a v6 database and v6 code refuses a v5 one: rolling back means checking out the old
 code and restoring the backup taken before the upgrade** (writes made after it are lost). After such a rollback the newer
 backup sets and anchors no longer fit the restored database (their blocks "vanished"), so move the newer `jarvis-*` sets and the
-newer `anchors/anchors-*.txt` files aside before the next backup, or it will quarantine its dump. The box is at **v9** now (v6 on 2026-10-06, v7 on
+newer `anchors/anchors-*.txt` files aside before the next backup, or it will quarantine its dump. The box is at **v9** now; `main` is at **v10**, not yet deployed (v6 on 2026-10-06, v7 on
 2026-10-08, v8 and v9 on 2026-10-10). V8 has no down-migration, so the way back across it is the named backup restored under the `pre-v8v9` images. V9 alone
 does have one: run `app.pg_schema.V9_ROLLBACK` as the migration role in one transaction and deploy the V8 code (`docs/ledger_search.md`), which keeps every write made since.
 

docs/POSTGRES.md

@@ -78,7 +78,7 @@ pool is shed with an immediate 503 rather than queued behind a slow database.
 * `GET /health` - **liveness**: the process is up. Never touches the ledger, reports no counts or paths.
 * `GET /ready` - **readiness**: 200 only if all of these hold, checked live on every call (never cached);
   otherwise 503 with `Retry-After` and the failing check names (no details):
-  `database` (`SELECT 1` as the application role), `schema_version` (matches the code exactly), `role`
+  `database` (`SELECT 1` as the application role), `schema_version` (matches the code exactly) (the code is at schema 10; a database at 9 is refused until migrated), `role`
   (not superuser, no `BYPASSRLS`), `rls_tables` (every table of the schema with a `tenant_key` column has row-level security enabled, **forced**, and at least one policy; the role check proves the app role cannot bypass RLS, this proves there is RLS to be subject to, so a migration that adds a tenant table without it, or drops its FORCE, makes `/ready` 503; the offending tables are named in the log only; the one exception is the retained legacy blob table `jarvis_tenant_ledgers`, which the importer leaves as a rollback copy and the row store never reads: `legacy_data` watches it), `history_write_denied` (the role is *proven* unable to write
   `record_history`, `chain_heads` or `history_counters`: the catalog shows no write privilege and attempted
   INSERT/UPDATE/DELETE are refused, inside rolled-back savepoints; `TRUNCATE` is checked in the catalog

docs/RFC-multi-tenant-ledger.md

@@ -391,6 +391,10 @@ concept at all) confirmed the direction and added three requirements. Both are r
     From `skunkworks-powerweave/mcp-multitenant-gateway` (third study, 2026-10-16): its limiter is keyed by the bearer key prefix and runs *before* verification, so brute-force guessing of one key is throttled too,
     and **it refuses to start with more than one worker unless a shared limiter backend is declared**, so a process-local limit cannot silently multiply per replica. Adopt both: throttle by the public `key_id`
     before the HMAC check, and make the process-local limiter a startup refusal when more than one worker or replica is configured without a shared backend.
+    **Built (branch `feat/rate-limit-per-credential`):** `/mcp` requests are charged to the authenticated credential (a key tenant's key id, an OAuth subject, or the operator), the client address only when nobody
+    authenticated; idle buckets are evicted and the table is capped; failed tenant-key attempts are counted per (client address, key id) and, past `JARVIS_KEY_FAIL_LIMIT_PER_MINUTE` (20), refused with 429 before any HMAC
+    or database work (per address as well as key id, so a stranger naming your key id cannot lock you out of it); more than one worker (`WEB_CONCURRENCY`, `UVICORN_WORKERS`, `GUNICORN_WORKERS`) refuses to start unless
+    `JARVIS_RATE_LIMIT_SHARED_BACKEND` names the shared limiter that enforces the real limit. Still process-local and public-mode-only for the `/mcp` limit; a shared limiter is not built.
 18. **Decision transitions are conditional and carry the tenant (section 4).** mcp-gateway decides an approval with `WHERE id = $1 AND status = 'pending' AND tenant_id = $2`, so a decision is idempotent and a guessed id
     from another tenant matches nothing. Our human-decision and draft/verified paths already run under forced row-level security; add an explicit test that a foreign tenant's decision on a guessed id changes nothing
     and a repeated decision is a no-op.
@@ -402,6 +406,14 @@ concept at all) confirmed the direction and added three requirements. Both are r
 20. **Candidate, not scheduled: idempotency keys on writes.** A retried `remember` (a timeout, a client retry) can store a duplicate draft. An optional caller-supplied idempotency key, scoped to the tenant and
     credential, would make a retry return the first result. This is a feature, not an isolation fix; record it, decide later.
 
+21. **Admit only on success; never on "not denied" (from an independent verification of the mcp-zero study, 2026-10-16).** That gateway blocked a call only on an explicit deny, so a hook that crashed before
+    or after the policy ran let the call through; it built a masked payload and then sent the original upstream while the audit said "masked"; and a plugin hook that ran before policy could replace the
+    identity. For us: (a) every gate in `identity_middleware` and the key branch admits only on a positive result, and any exception is a refusal (500 or 503), never a call to the handler; (b) whatever is
+    recorded as the redacted form of a request is what the code path uses where redaction is claimed (call log, replay log); (c) the tenant is bound once, by the identity layer, and nothing downstream (tool
+    code, a future plugin seam) may rewrite the principal. Add Red Team tests: an injected failure in each gate (control schema, pepper, limiter, call log) must produce a refusal and no handler call; a
+    handler must be unable to change `current_principal()` for the rest of the request.
+    Also from it: anonymous operation must stay impossible in production: strict mode already refuses to start without a credential; keep that a startup refusal, not a default.
+
 Also noted from mcp-gateway: its role lookup joined on the agent id without asserting the tenant, relying on globally unique ids and the issuer. Our key resolution takes the tenant from the control-schema row of
 the presented key and never from a request value, which is the stronger rule both studies recommend.