API Rate Limiting & Quota Management System

A backend that limits API usage per user and per plan (Free / Pro / Enterprise) using the Token Bucket algorithm, with hard blocking, soft throttling, automatic midnight quota resets, usage analytics, high-usage alerts, and an admin dashboard.

Built as a compact, readable prototype — no over-engineering, but every moving part is real: real Redis counters, a real Kafka pipeline, a real Next.js dashboard.

This file is the whole-system overview. Component-level docs:

backend/README.md — the Express API, token bucket, Kafka, jobs, full endpoint reference
dashboard/README.md — the Next.js admin UI, pages, data flow, shadcn/Base UI notes

Features

#	Requirement	Where it lives
1	Create plans (Free 100/day, Pro 10 000/day, Enterprise 100 000/day, …)	`plans` table · `services/plans.js` · `GET/POST/PUT /admin/plans` · dashboard Plans
2	Track API calls per user	Redis token bucket per user +`usage_logs` row per request
3	Hard limit (block when quota exceeds)	`middleware/rateLimiter.js` → `429` when the bucket is empty
4	Soft limit (slow down requests)	`rateLimiter.js` → `await sleep(soft_delay_ms)` when few tokens remain
5	Quota reset every midnight	`jobs/quotaReset.js` (`node-cron` `0 0 * * *`) + Redis key TTL
6	Admin dashboard showing usage per user	Next.js dashboard —Users and User detail pages
7	Alerts for high usage	`alerts` table · `services/alerts.js` · dashboard Alerts
8	API usage logs & analytics	`usage_logs` + `services/analytics.js` · dashboard Logs and Overview

Everything else in the codebase (API-key auth, admin token, Redis, Kafka, Docker, shadcn/ui) is either a dependency of one of these features or an explicit stack choice.

Architecture

flowchart LR
    C[API consumer] -- X-API-Key --> API

    subgraph API[Express API :4000]
      A1[auth middleware<br/>resolve key -> user]
      A2[usageLogger<br/>on response finish]
      A3[rateLimiter<br/>token bucket]
      A4[route handler<br/>/api/v1/data, /heavy]
      A1 --> A2 --> A3 --> A4
    end

    A3 <-- INCR / refill<br/>Lua script --> R[(Redis :6379<br/>bucket:userId)]
    A2 -- usage event --> K[[Kafka :9092<br/>api-usage-events]]
    K --> CON[consumer.js]
    CON -- INSERT --> DB[(SQLite<br/>data.db)]
    A3 -- raise alert --> DB
    A1 -- read --> DB

    CRON[node-cron 00:00] -- snapshot + refill --> R
    CRON -- snapshot --> DB

    D[Admin dashboard :3000] -- X-Admin-Token<br/>server-side only --> API

Request path (per call to /api/v1/*):

auth resolves X-API-Key → user (15 s in-memory cache, invalidated on admin changes).
usageLogger registers a res.on("finish") hook (records the event after the response).
rateLimiter runs the token-bucket Lua script in Redis, sets X-RateLimit-* headers, and either blocks (429/403), throttles (sleep then continue), or passes through.
The route handler returns sample data.
On finish, the usage event is published to Kafka (or written directly if USE_KAFKA=false).
consumer.js reads the event and inserts a usage_logs row.

Why Kafka: the request path only produces a lightweight event and returns. All log persistence and analytics happen asynchronously in the consumer, so logging never slows down or breaks the rate-limited API. If Kafka is down, publishes fail softly and the API keeps serving traffic.

Tech stack

Backend (`backend/`) — JavaScript (Node, CommonJS)

Concern	Choice	Notes
Web framework	Express 4	plain routes + middleware, no framework magic
Rate-limit state	Redis via `ioredis`	token bucket state, atomic refill+consume inone Lua script
Persistence	SQLite via `better-sqlite3`	synchronous, no ORM, schema created on boot in`db.js`
Event streaming	Kafka via `kafkajs`	one topic, one producer, one consumer process
Scheduler	`node-cron`	midnight quota reset
Utilities	`dotenv`, `nanoid` (API keys), `morgan` (request logs)

Dashboard (`dashboard/`) — TypeScript

Concern	Choice
Framework	Next.js 16 (App Router, Turbopack, React 19)
UI	shadcn/ui (Base UI primitives) + Tailwind CSS v4
Charts	`recharts` via the shadcn `chart` wrapper
Toasts	`sonner`
Data	server components + server actions;`X-Admin-Token` stays server-side

Infrastructure

Piece	Choice
Local services	Docker Compose — `redis:7-alpine` + `apache/kafka:3.8.0` (KRaft mode, no ZooKeeper)
Container names	`arl-redis`, `arl-kafka`

Prerequisites

Node.js 18+ (developed on Node 23)
Docker + Docker Compose (for Redis and Kafka)
Ports free: 4000 (API), 3000 (dashboard), 6379 (Redis), 9092 (Kafka)

Setup & run

You need three terminals for the full stack (four if you want demo traffic).

# ── 0. Infrastructure ────────────────────────────────────────────────
docker compose up -d                 # starts arl-redis + arl-kafka
docker compose ps                    # both should be "running"

# ── 1. Backend API (terminal A) ─────────────────────────────────────
cd backend
npm install
cp .env.example .env
npm run seed                         # creates 3 plans + 3 users, PRINTS THEIR API KEYS
npm run dev                          # -> http://localhost:4000  (nodemon reload)

# ── 2. Kafka consumer (terminal B) ─────────────────────────────────
cd backend
npm run consumer                     # required while USE_KAFKA=true

# ── 3. Dashboard (terminal C) ──────────────────────────────────────
cd dashboard
npm install
cp .env.local.example .env.local
npm run dev                          # -> http://localhost:3000

# ── 4. Demo traffic (optional, terminal D, from backend/) ──────────
npm run simulate 300                 # fires 300 random requests across all users

Important: with USE_KAFKA=true (the default) the API only publishes usage events — the consumer must be running or usage_logs stays empty (rate limiting still works; only logs/analytics are affected). To run without Kafka entirely, set USE_KAFKA=false in backend/.env and the API writes logs directly.

Make your first API call

npm run seed prints every user's API key. The keys are regenerated on each seed — to look up the current key for a user:

# from backend/ — all users and keys straight from SQLite
node -e "const db=require('better-sqlite3')('data.db',{readonly:true});console.table(db.prepare('SELECT u.name,u.api_key,p.name plan,p.daily_limit FROM users u JOIN plans p ON p.id=u.plan_id').all())"

# or via the admin API (jq optional)
curl -s -H "X-Admin-Token: dev-admin-token" localhost:4000/admin/users | jq '.[] | {name, api_key, plan_name}'

Seeded users: Alice (Free, 100/day), Bob (Pro, 10 000/day), Carol (Enterprise, 100 000/day). Call the API by putting that user's key in the X-API-Key header (a plain browser URL will not work).

PowerShell — call /api/v1/data as Alice and show the status + remaining quota:

$key = "<Alice's key from the command above>"
Invoke-WebRequest -Uri http://localhost:4000/api/v1/data -Headers @{ "X-API-Key" = $key } |
  Select-Object StatusCode, @{n="Remaining";e={$_.Headers["X-RateLimit-Remaining"]}}, Content

curl (Git Bash):

KEY=<Alice's key from the command above>
curl -i -H "X-API-Key: $KEY" localhost:4000/api/v1/data      # rate-limited, costs 1 token
curl -s -H "X-API-Key: $KEY" localhost:4000/api/v1/usage     # quota check, no token cost

See Demo scenarios for hitting the soft / hard limits, blocking a user, and the midnight reset.

npm scripts

backend/

Script	Does
`npm run dev`	start the API with`nodemon`
`npm start`	start the API with`node`
`npm run consumer`	start the standalone Kafka consumer
`npm run seed`	create/refresh plans + sample users (idempotent)
`npm run simulate [N]`	send`N` random requests (default 300) to the API

dashboard/

Script	Does
`npm run dev`	Next.js dev server
`npm run build`	production build
`npm start`	serve the production build

Stopping / resetting

docker compose down                  # stop Redis + Kafka (add -v to wipe their data)

Reset the backend to a clean slate:

# stop the API first, then:
rm backend/data.db backend/data.db-shm backend/data.db-wal
docker exec arl-redis redis-cli FLUSHALL
cd backend && npm run seed

Project structure

api-rate-limiter/
├── docker-compose.yml            Redis + Kafka (KRaft, auto-create topics)
├── README.md
├── CLAUDE.md                     design notes / contributor guide
│
├── backend/
│   ├── .env.example
│   ├── package.json
│   └── src/
│       ├── index.js              Express bootstrap, route mounting, graceful shutdown
│       ├── config.js             env parsing
│       ├── db.js                 better-sqlite3 + CREATE TABLE IF NOT EXISTS schema
│       ├── redis.js              ioredis client + secondsUntilMidnight()
│       ├── tokenBucket.js        inline Lua script + consume() / peek() / refillFull()
│       ├── kafka.js              kafkajs client, producer, ensureTopic()
│       ├── usageRecorder.js      recordUsage(): Kafka publish, else direct SQLite write
│       │
│       ├── middleware/
│       │   ├── auth.js           X-API-Key -> req.user  (+ 15s cache, invalidate())
│       │   ├── adminAuth.js      X-Admin-Token check
│       │   ├── usageLogger.js    res.on('finish') -> recordUsage()
│       │   └── rateLimiter.js    token bucket: 403 / 429 / soft-sleep / headers / alerts
│       │
│       ├── routes/
│       │   ├── api.js            GET /api/v1/data, /api/v1/heavy   (rate limited)
│       │   ├── usage.js          GET /api/v1/usage                 (caller's quota)
│       │   └── admin.js          /admin/plans, /users, /logs, /analytics, /alerts
│       │
│       ├── services/
│       │   ├── plans.js  users.js  logs.js  alerts.js  analytics.js
│       │
│       ├── jobs/
│       │   └── quotaReset.js     node-cron 0 0 * * *  (snapshot + refill)
│       │
│       ├── consumer.js           Kafka consumer -> usage_logs
│       ├── seed.js               3 plans + 3 users
│       └── simulate.js           random traffic generator
│
└── dashboard/
    ├── .env.local.example
    ├── components.json           shadcn config (Base UI)
    ├── app/
    │   ├── layout.tsx            dark shell: <Nav> sidebar + <Toaster>
    │   ├── page.tsx              Overview
    │   ├── users/page.tsx        Users table + New user
    │   ├── users/[id]/page.tsx   User detail
    │   ├── plans/page.tsx        Plans table + New/Edit
    │   ├── alerts/page.tsx       Alerts feed
    │   ├── logs/page.tsx         Usage logs
    │   └── error.tsx             shown when the backend is unreachable
    ├── components/
    │   ├── ui/                   shadcn primitives
    │   ├── nav.tsx  stat-card.tsx  bits.tsx  calls-chart.tsx
    │   ├── users-table.tsx  new-user-dialog.tsx  user-detail-actions.tsx
    │   ├── plans-manager.tsx  alerts-table.tsx  logs-filters.tsx
    └── lib/
        ├── api.ts                server-only fetch<T>() with X-Admin-Token
        ├── actions.ts            "use server" mutations + revalidatePath
        └── format.ts             num / pct / timeAgo / hourLabel

The Token Bucket algorithm

Each user has a bucket of tokens. Every request costs 1 token. Tokens refill continuously over time. An empty bucket means "blocked".

Parameters (derived from the plan)

Bucket parameter	Value	Meaning
`capacity`	`plan.daily_limit`	max tokens the bucket can hold — also the maximum burst
`refill_rate`	`plan.daily_limit / 86 400 000` tokens per ms	sustained allowed rate (a full day's tokens per day)
`soft_cutoff`	`capacity × plan.soft_threshold`	throttle once fewer than this many tokens remain

So a Free user (100/day) can burst up to 100 requests instantly, then is limited to ~1 request every 864 seconds of refill — and once they drop below 20 tokens (0.2 × 100), each request is slowed by soft_delay_ms.

Redis state

key:  bucket:{userId}          (hash)
      tokens        -> float, current token count
      last_refill   -> epoch ms of the last update
TTL:  seconds until next midnight + 1 hour   (self-cleans; also refilled by cron)

The Lua script (atomic)

Refill + consume happen in a single Redis Lua script (tokenBucket.js), so concurrent requests from the same user can never double-spend a token:

elapsed   = now - last_refill
tokens    = min(capacity, tokens + elapsed * refill_rate)   -- refill
if tokens >= 1 then
    tokens  = tokens - 1                                     -- consume
    allowed = 1
    soft    = tokens < soft_cutoff
end
HSET bucket:{userId} tokens <tokens> last_refill <now>
EXPIRE bucket:{userId} <ttl>
return { allowed, tokens, soft }

peek() does the same refill math without consuming — used by GET /api/v1/usage and the dashboard so checking your quota doesn't cost a token.

Rate limiting behaviour

Applied by middleware/rateLimiter.js to every /api/v1/data and /api/v1/heavy request.

Situation	HTTP	Body	Side effects
Valid, tokens available	`200`	endpoint payload	1 token consumed
Valid, but`tokens < soft_cutoff`	`200`	endpoint payload	1 token consumed, response delayed by`soft_delay_ms`, `X-RateLimit-Throttled: true`, `HIGH_USAGE` alert (once/day)
Bucket empty (hard limit)	`429`	`{"error":"Rate limit exceeded","retry_after_seconds":N}`	`Retry-After` header, `LIMIT_REACHED` alert (once/day)
User is blocked by an admin	`403`	`{"error":"Account blocked. Contact support."}`	—
Missing API key	`401`	`{"error":"Missing X-API-Key header"}`	—
Unknown API key	`401`	`{"error":"Invalid API key"}`	—
Redis unavailable	requestpasses through (fail-open) — availability over strictness		logged to console

Response headers (on every rate-limited request)

Header	Example	Meaning
`X-RateLimit-Limit`	`100`	bucket capacity (= plan daily limit)
`X-RateLimit-Remaining`	`73`	whole tokens left right now
`X-RateLimit-Reset`	`1788287400`	Unix seconds of the next local midnight
`X-RateLimit-Throttled`	`true`	present only when the soft limit slowed this request
`Retry-After`	`864`	seconds until the next token (only on`429`)

Retry-After is ceil(86400 / daily_limit) — the time for the bucket to refill one token.

Quota reset

Two mechanisms, on purpose:

Continuous — the token bucket refills every millisecond via refill_rate, so usage recovers gradually without waiting for midnight.
Hard reset at midnight — jobs/quotaReset.js runs on node-cron at 0 0 * * *:
- for each user, count yesterday's usage_logs and upsert a row into daily_usage (user_id, date, call_count, blocked_count) — this is the historical record;
- refill every user's bucket back to full capacity.
- Redis keys also carry a TTL to just past midnight as a backstop.

is_blocked is only ever changed by an admin (dashboard or API). Hitting a 429 never blocks the account — it just means "out of tokens right now".

Timestamps in SQLite are stored in UTC (datetime('now')); the cron fires on the server's local midnight. In a single-timezone deployment this is fine; the two only diverge by your UTC offset.

Alerts

Raised by rateLimiter.js, deduplicated to one alert per type, per user, per day (so a user hammering the API after hitting their limit doesn't create hundreds of rows).

Type	Trigger	Example message
`HIGH_USAGE`	first request after tokens drop below`soft_cutoff`	`Alice is past 80% of their Free quota (19 calls left today)`
`LIMIT_REACHED`	first`429` of the day	`Alice exhausted their Free quota (100/day)`

Alerts are rows in the alerts table, shown in the dashboard Alerts page, and counted in Overview → Open alerts. An admin resolves them (resolved = 1). There is no email/SMS delivery — that's out of scope.

The Kafka usage pipeline

rateLimiter / handler  ──►  res.on('finish')  ──►  usageRecorder.recordUsage(event)
                                                        │
                                       USE_KAFKA=true   │   USE_KAFKA=false
                                                        ▼          ▼
                                          produce → api-usage-events   direct INSERT
                                                        │
                                                        ▼
                                              consumer.js (npm run consumer)
                                                        │
                                                        ▼
                                              INSERT INTO usage_logs

Topic: api-usage-events, 1 partition, created automatically by ensureTopic() on both the producer and consumer side (belt and suspenders with Kafka's own auto.create.topics.enable).

Event shape:

{
  "userId": 1,
  "endpoint": "/api/v1/data",
  "method": "GET",
  "status": 429,
  "responseTimeMs": 2,
  "blocked": true,
  "ts": "2026-09-01T06:18:52.123Z"
}

Fallback: if USE_KAFKA=true but the publish fails (broker down), recordUsage writes the row directly so nothing is lost silently.
Consumer group: usage-consumer. Analytics are computed on-read from usage_logs (SQL in services/analytics.js), so the consumer only does inserts.

Data model

SQLite, defined in backend/src/db.js (created on first run).

`plans`

column	type	notes
`id`	INTEGER PK
`name`	TEXT UNIQUE	`Free`, `Pro`, `Enterprise`, …
`daily_limit`	INTEGER	bucket capacity + refill basis
`soft_threshold`	REAL	fraction of quotaremaining when throttling starts (e.g. `0.2`)
`soft_delay_ms`	INTEGER	delay applied to throttled requests
`price`	REAL	display only — no billing anywhere
`created_at`	TEXT	UTC

`users`

column	type	notes
`id`	INTEGER PK
`name`	TEXT
`email`	TEXT UNIQUE
`api_key`	TEXT UNIQUE	`rl_` + 32 chars (`nanoid`)
`plan_id`	INTEGER FK → plans	set manually (seed / admin API / DB edit) — no upgrade flow
`is_blocked`	INTEGER	`0`/`1`, admin-controlled only
`created_at`	TEXT	UTC

`usage_logs` (one row per API request)

column	type
`id`	INTEGER PK
`user_id`	INTEGER FK → users
`endpoint`	TEXT
`method`	TEXT
`status_code`	INTEGER
`response_time_ms`	INTEGER
`blocked`	INTEGER (`1` for `429`/`403`)
`created_at`	TEXT (UTC)

Indexed on user_id and created_at.

`daily_usage` (midnight snapshots)

column	type	notes
`id`	INTEGER PK
`user_id`	INTEGER FK → users
`date`	TEXT	`YYYY-MM-DD` of the day summarised
`call_count`	INTEGER
`blocked_count`	INTEGER
		`UNIQUE(user_id, date)`

`alerts`

column	type	notes
`id`	INTEGER PK
`user_id`	INTEGER FK → users
`type`	TEXT	`HIGH_USAGE` \| `LIMIT_REACHED`
`message`	TEXT
`resolved`	INTEGER	`0`/`1`
`created_at`	TEXT	UTC

API reference

Base URL: http://localhost:4000

Authentication

Audience	Header	Value
API consumer	`X-API-Key`	a user's key (printed by`npm run seed`, or from `GET /admin/users`)
Admin / dashboard	`X-Admin-Token`	`ADMIN_TOKEN` from `backend/.env` (default `dev-admin-token`)

Consumer API

`GET /health`

No auth. { "ok": true }

`GET /api/v1/data` (rate limited)

curl -i -H "X-API-Key: rl_xxx" http://localhost:4000/api/v1/data

HTTP/1.1 200 OK
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 98
X-RateLimit-Reset: 1788287400

{
  "message": "Here is your data",
  "user": "Alice",
  "items": [{ "id": 1, "value": "alpha" }, { "id": 2, "value": "beta" }, { "id": 3, "value": "gamma" }],
  "served_at": "2026-09-01T06:00:00.000Z"
}

`GET /api/v1/heavy` (rate limited)

Same auth/headers; simulates ~200 ms of work. Returns { message, user, result, served_at }.

`GET /api/v1/usage` (auth only — does NOT consume a token)

curl -s -H "X-API-Key: rl_xxx" http://localhost:4000/api/v1/usage

{
  "user": "Alice",
  "plan": "Free",
  "daily_limit": 100,
  "remaining": 98,
  "used_estimate": 2,
  "calls_today": 2,
  "is_blocked": false
}

Admin API — plans

Method	Path	Body	Notes
`GET`	`/admin/plans`	—	array of plans, ordered by`daily_limit`
`POST`	`/admin/plans`	`{ name, daily_limit, soft_threshold?, soft_delay_ms?, price? }`	`name` + `daily_limit` required; `409` if name exists
`PUT`	`/admin/plans/:id`	any subset of`{ name, daily_limit, soft_threshold, soft_delay_ms, price }`	`404` if not found

curl -s -X POST http://localhost:4000/admin/plans \
  -H "X-Admin-Token: dev-admin-token" -H "Content-Type: application/json" \
  -d '{"name":"Trial","daily_limit":25,"soft_delay_ms":300}'

Admin API — users

Method	Path	Body	Notes
`GET`	`/admin/users`	—	each user +`plan_name`, `daily_limit`, `calls_today`
`GET`	`/admin/users/:id`	—	user +`plan` object + `calls_today` + `recent_logs` (25)
`POST`	`/admin/users`	`{ name, email, plan_id }`	returns the new user incl.`api_key`; `409` on duplicate email
`PUT`	`/admin/users/:id`	`{ name?, plan_id? }`	change name / plan
`POST`	`/admin/users/:id/block`	—	sets`is_blocked = 1`
`POST`	`/admin/users/:id/unblock`	—	sets`is_blocked = 0`
`POST`	`/admin/users/:id/regenerate-key`	—	issues a fresh`api_key`

All mutating user routes invalidate the auth cache immediately.

Admin API — logs, analytics, alerts

Method	Path	Query	Returns
`GET`	`/admin/logs`	`user_id`, `limit` (def 100), `offset` (def 0)	`usage_logs` rows (newest first) + `user_name`
`GET`	`/admin/analytics`	—	see below
`GET`	`/admin/alerts`	`resolved` (`true`/`false`, omit for all)	alert rows +`user_name`
`POST`	`/admin/alerts/:id/resolve`	—	`{ ok: true }`

GET /admin/analytics response:

{
  "overview": {
    "calls_today": 311, "blocked_today": 42, "active_users_today": 3,
    "total_users": 3, "blocked_users": 0, "open_alerts": 2
  },
  "calls_over_time": [{ "bucket": "2026-09-01 06:00", "calls": 311, "blocked": 42 }],
  "top_users": [
    { "id": 1, "name": "Alice", "plan_name": "Free", "daily_limit": 100,
      "calls_today": 143, "blocked_today": 42 }
  ],
  "status_breakdown": [{ "status_code": 200, "n": 269 }, { "status_code": 429, "n": 42 }]
}

Admin dashboard

http://localhost:3000 — dark admin UI (Next.js 16 + shadcn/ui). Reads the backend only through server-side calls, so ADMIN_TOKEN never reaches the browser. Every page fetches live (cache: "no-store", dynamic = "force-dynamic").

Full frontend docs: dashboard/README.md · Full backend docs: backend/README.md

What was built — `dashboard/` (Next.js 16 + shadcn/ui)

Every page and interaction below was verified working in a browser against the live backend:

Page	Route	Verified working
Overview	`/`	stat cards (calls / blocked / active / alerts), 24 h calls area chart, top-users table with quota bars, status-code breakdown
Users	`/users`	table with quota bars + status badges; row menu →Block (toast + badge flips to red), Unblock, Regenerate key; New user dialog
User detail	`/users/[id]`	plan`<Select>`, block / regenerate, reveal-API-key toggle, stat row, quota bar, recent requests
Plans	`/plans`	table;New plan + Edit dialogs (name locked on edit)
Alerts	`/alerts`	feed withOpen / All filter; Resolve (toast + row leaves the Open view)
Logs	`/logs`	every request, filter by user (URL-driven),Previous / Next pagination

Checks passed: tsc --noEmit clean · next build clean · browser smoke test of all six pages plus block / unblock / resolve / dialogs / filter · zero app errors or warnings in the console.

Mutations are Next.js server actions (lib/actions.ts) → call the admin API → revalidatePath so tables refresh. Success/failure surface as sonner toasts. If the backend is unreachable, app/error.tsx explains what to check.

shadcn note: shadcn init selected Base UI (not Radix). Compose with the render={<Comp/>} prop instead of asChild, and pass items to <Select> so the trigger shows the selected label.

Environment variables

`backend/.env` (copy from `.env.example`)

Var	Default	Purpose
`PORT`	`4000`	API port
`DATABASE_FILE`	`./data.db`	SQLite file path
`REDIS_URL`	`redis://localhost:6379`	Redis connection
`KAFKA_BROKERS`	`localhost:9092`	comma-separated broker list
`KAFKA_TOPIC`	`api-usage-events`	usage event topic
`KAFKA_CLIENT_ID`	`api-rate-limiter`
`KAFKA_CONSUMER_GROUP`	`usage-consumer`
`USE_KAFKA`	`true`	`false` → API writes `usage_logs` directly, no consumer needed
`ADMIN_TOKEN`	`dev-admin-token`	shared secret for`/admin/*` and the dashboard

`dashboard/.env.local` (copy from `.env.local.example`)

Var	Default	Purpose
`BACKEND_URL`	`http://localhost:4000`	where the dashboard reaches the API
`ADMIN_TOKEN`	`dev-admin-token`	must match the backend's `ADMIN_TOKEN`

Demo scenarios

With the stack running (docker compose up -d, backend npm run dev, consumer npm run consumer, dashboard npm run dev) and npm run seed done.

0. Get the API keys

npm run seed prints them. To fetch them again later, use either:

# via the admin API
curl -s -H "X-Admin-Token: dev-admin-token" localhost:4000/admin/users

# or straight from SQLite (from backend/)
node -e "const db=require('better-sqlite3')('data.db',{readonly:true});console.table(db.prepare('SELECT u.name,u.api_key,p.name plan,p.daily_limit FROM users u JOIN plans p ON p.id=u.plan_id').all())"

The dashboard also shows a key on Users → (a user) → reveal API key.

Seeded users: Alice (Free, 100/day), Bob (Pro, 10 000/day), Carol (Enterprise, 100 000/day). Examples below use Alice — substitute her current key.

1. Call the API manually as Alice

Every consumer call sends the key in the X-API-Key header to http://localhost:4000. A plain browser URL will not work (no header) — use curl, PowerShell, or an API client.

curl (Git Bash):

KEY=<Alice's key>

# rate-limited call — costs 1 token, returns sample data
curl -i -H "X-API-Key: $KEY" localhost:4000/api/v1/data

# the other sample endpoint (~200 ms simulated work)
curl -i -H "X-API-Key: $KEY" localhost:4000/api/v1/heavy

# check Alice's quota — does NOT cost a token
curl -s -H "X-API-Key: $KEY" localhost:4000/api/v1/usage

-i prints the response headers — watch X-RateLimit-Remaining count down from X-RateLimit-Limit: 100, and X-RateLimit-Reset (next-midnight epoch seconds).

PowerShell (one line — status + remaining quota + body):

$key = "<Alice's key>"
Invoke-WebRequest -Uri http://localhost:4000/api/v1/data -Headers @{ "X-API-Key" = $key } |
  Select-Object StatusCode, @{n="Remaining";e={$_.Headers["X-RateLimit-Remaining"]}}, Content

Add -SkipHttpErrorCheck so a 429 / 403 prints instead of throwing.

Postman / Thunder Client / Bruno / Insomnia: GET http://localhost:4000/api/v1/data, add a header X-API-Key = Alice's key. Send.

Expected responses:

Header / body	Meaning
`200` + `{ "message": "Here is your data", "user": "Alice", ... }`	success, 1 token spent
`X-RateLimit-Throttled: true` (still `200`, response delayed ~800 ms)	soft limit — Alice is below 20 % of quota
`429` + `{ "error": "Rate limit exceeded", "retry_after_seconds": 864 }`	hard limit — bucket empty
`403` + `{ "error": "Account blocked. Contact support." }`	an admin blocked Alice
`401` + `{ "error": "Missing X-API-Key header" }` / `Invalid API key`	no / wrong key

2. Watch Alice hit every limit stage (manual burst)

KEY=<Alice's key>
for i in $(seq 1 130); do
  curl -s -o /dev/null -w "%{http_code} " -H "X-API-Key: $KEY" localhost:4000/api/v1/data
done; echo

PowerShell equivalent:

$key = "<Alice's key>"
1..130 | ForEach-Object {
  (Invoke-WebRequest http://localhost:4000/api/v1/data -Headers @{ "X-API-Key" = $key } `
     -SkipHttpErrorCheck).StatusCode
}

You'll see:

200 for roughly the first 80 requests;
200 but visibly slower for ~81–100 (soft limit, X-RateLimit-Throttled: true);
a run of 429 from ~101 onward (hard limit).

Then check the dashboard (localhost:3000): Alerts has HIGH_USAGE and LIMIT_REACHED for Alice, Overview → Blocked today is non-zero, and Logs lists every request (filter by Alice).

3. Generate bulk traffic across all users

For a populated dashboard without hand-firing requests:

cd backend
npm run simulate 300        # 300 random GETs spread across Alice, Bob, Carol

It prints a status-code tally when done (e.g. { '200': 271, '429': 29 }). Increase the number for more data: npm run simulate 1000.

4. Block / unblock a user

Dashboard → Users → row menu → Block (or curl -X POST -H "X-Admin-Token: dev-admin-token" localhost:4000/admin/users/1/block). Alice's calls now return 403 regardless of quota. Unblock to restore — effective immediately (the admin action clears the auth cache; otherwise it would take up to 15 s).

5. Simulate the midnight reset

cd backend
node -e "require('./src/jobs/quotaReset').runReset().then(()=>process.exit(0))"

Snapshots the previous day into daily_usage and refills every token bucket. Alice (previously 429ing) can call again.

Troubleshooting

Symptom	Cause / fix
Dashboard pages show "Couldn't load this page"	Backend not running, or`ADMIN_TOKEN` mismatch between `backend/.env` and `dashboard/.env.local`
`/logs` and Overview are empty but rate limiting works	`USE_KAFKA=true` and the consumer isn't running — run `npm run consumer`, or set `USE_KAFKA=false`
Consumer exits with`UNKNOWN_TOPIC_OR_PARTITION` / `group coordinator not available`	Kafka still starting up — wait ~30 s after`docker compose up -d`, then restart the consumer
`TimeoutNegativeWarning` printed once at startup	Harmless`kafkajs` quirk on newer Node when `USE_KAFKA=true`; ignore
`EADDRINUSE :4000` / `:3000`	A previous process is still bound — kill it (`npx kill-port 4000`)
`better-sqlite3` fails to install	Needs build tools; usually prebuilt binaries cover Node 18–23. Retry`npm install`.
Quota "resets" don't line up with your local midnight	SQLite stores UTC; cron uses server-local time — seeQuota reset
All requests return`200`, never `429`, during a load test	The bucket refills as you go; to force a block, burst faster than`daily_limit / 86400` per second, or lower the plan's `daily_limit`

Design decisions & scope

Deliberately included (dependencies of the required features, or explicit choices): API-key auth, a single admin token, Redis, Kafka + a consumer, Docker Compose, shadcn/ui, seed.js, simulate.js, two sample endpoints to rate-limit.

Deliberately out of scope:

Payments / billing / checkout / plan-upgrade flow — plans are assigned manually (seed, admin API, or a direct DB edit). price is a display-only field.
Email / SMS alert delivery — alerts are DB rows shown in the dashboard.
End-user signup UI, JWT/OAuth — API keys only.
Per-endpoint quotas — the quota is per user per day.
Multi-node coordination beyond what Redis already provides.
A test suite — verification is via simulate.js and manual/browser checks.

Key trade-offs:

Fail-open on Redis outage — if the token bucket can't be evaluated, the request is allowed. Availability is prioritised over strict enforcement for a prototype.
Auth cache (15 s) — a burst from one key doesn't hit SQLite every time; admin mutations invalidate the entry so blocks/plan-changes apply immediately.
Analytics computed on-read — simple SQL over usage_logs instead of maintaining rollups; fine at prototype data volumes.
Token bucket over fixed-window — gives smooth throttling and natural burst handling, and maps cleanly onto "N calls/day" as the bucket capacity.

Name		Name	Last commit message	Last commit date
backend		backend
dashboard		dashboard
CLAUDE.md		CLAUDE.md
README.md		README.md
docker-compose.yml		docker-compose.yml

Latest commit

History

Folders and files

Repository files navigation

API Rate Limiting & Quota Management System

Table of contents

Features

Architecture

Tech stack

Backend (backend/) — JavaScript (Node, CommonJS)

Dashboard (dashboard/) — TypeScript

Infrastructure

Prerequisites

Setup & run

Make your first API call

npm scripts

Stopping / resetting

Project structure

The Token Bucket algorithm

Parameters (derived from the plan)

Redis state

The Lua script (atomic)

Rate limiting behaviour

Response headers (on every rate-limited request)

Quota reset

Alerts

The Kafka usage pipeline

Data model

plans

users

usage_logs (one row per API request)

daily_usage (midnight snapshots)

alerts

API reference

Authentication

Consumer API

GET /health

GET /api/v1/data (rate limited)

GET /api/v1/heavy (rate limited)

GET /api/v1/usage (auth only — does NOT consume a token)

Admin API — plans

Admin API — users

Admin API — logs, analytics, alerts

Admin dashboard

What was built — dashboard/ (Next.js 16 + shadcn/ui)

Environment variables

backend/.env (copy from .env.example)

dashboard/.env.local (copy from .env.local.example)

Demo scenarios

0. Get the API keys

1. Call the API manually as Alice

2. Watch Alice hit every limit stage (manual burst)

3. Generate bulk traffic across all users

4. Block / unblock a user

5. Simulate the midnight reset

Troubleshooting

Design decisions & scope

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages