diff --git a/CLAUDE.md b/CLAUDE.md index 251635a..44d6f75 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -52,10 +52,14 @@ Copy `.env.example` to `.env` for local development. ### Connection slots and session lifetime -The concurrent-connection limit (20, per API token, `src/ratelimit/index.ts`) is acquired on session creation and released on `DELETE`, transport close, or by the idle sweeper. Two rules keep the pool from silting up, both covered by `tests/connectionSlots.spec.ts`: +The connection limit (20, per API token, `src/ratelimit/index.ts`) is acquired on session creation and released on `DELETE`, transport close, eviction, or by the idle sweeper (10 minutes). It caps how many sessions of one token may **work** at once, not how many it has abandoned: -- An initialize request that never produces a session (malformed body, aborted client) releases its slot from `res.on('close')`. Without that the slot is unreachable — the transport never opened, so `onclose` never fires, and the sweeper never sees a session that was never registered. -- A session that initialized but never made a call is reclaimed after **30 s** (`unusedSessionTimeoutMs`) instead of the ordinary 10-minute idle timeout. Reconnect-happy clients leave these behind by the dozen; each one holds a slot. +- When the pool is full, `evictIdlestSession()` closes the session of the **same token** that has been idle longest and has no POST in flight (`inFlight`). Only if all 20 are mid-request does the client get `429`. A client returning to an evicted session gets `404` and must re-initialize (MCP spec). A GET stream does not count as in flight. Coverage: `tests/sessionEviction.spec.ts`, which replays the 15 November 2026 incident. +- An initialize request that never produces a session (malformed body, aborted client) releases its slot from `res.on('close')`. Without that the slot is unreachable — the transport never opened, so `onclose` never fires, and nothing can evict a session that was never registered. Coverage: `tests/connectionSlots.spec.ts`. + +ChatGPT opens a new session for every tool call (initialize, `notifications/initialized`, a GET it aborts after a millisecond, one `tools/call`) and never sends `DELETE`. Twice the response was to treat the symptom: raising the limit from 5 to 20, then a 30-second timeout for "unused" sessions, which never fired because the `initialized` notification and the GET already counted as use. Do not raise the limit or add per-client timeouts again; eviction is what makes the pool independent of how many sessions a client leaves behind. + +`Session initialized`, `Session evicted` and every 429 that knows its token log the token fingerprint, so a burst can be attributed to a tenant. All 429s are counted in `mcp_rate_limit_exceeded_total{type}`, evictions in `mcp_sessions_evicted_total`. Client IP for rate limiting comes from the **rightmost** `X-Forwarded-For` entry — the hop our own proxy appended. The leftmost entry is client-supplied and would let a caller pick its own rate-limit bucket. `docker-compose.yml` additionally has Caddy overwrite the header rather than append to it. diff --git a/README.md b/README.md index 6a8ea73..d748507 100644 --- a/README.md +++ b/README.md @@ -217,7 +217,7 @@ The security characteristics differ significantly between transport modes: | **Network exposure** | None (local process) | TCP port, **no auth** | TCP port, Bearer auth | | **Authentication** | Not needed (local) | None | Bearer token or OAuth 2.0, validated against SeaTable | | **Rate limiting** | None | None | Per-token, per-IP, global | -| **Connection limits** | N/A | None | 20 concurrent sessions per token | +| **Connection limits** | N/A | None | 20 sessions per token; the longest-idle one makes room | | **Data scope** | All configured bases | All configured bases | One base per client token | > **⚠️ Warning:** Selfhosted HTTP mode (`--sse` / `--http`) has **no authentication**. Anyone who can reach the port gets full access to all configured bases, including write and delete operations. Only run it in trusted networks (localhost, Docker-internal) or behind a reverse proxy that handles authentication. For untrusted networks, use **managed mode** instead. @@ -226,7 +226,7 @@ The security characteristics differ significantly between transport modes: SeaTable's own API gateway enforces rate limits **per base** (default: 500 requests/minute per `base_uuid`) and **per organization** (monthly quota). These limits apply regardless of whether requests come from the MCP server, the web UI, or direct API calls. The MCP server does not duplicate these limits — instead, it retries automatically with exponential backoff when SeaTable returns `429 Too Many Requests`. -In **managed mode**, the MCP server adds its own rate limits to protect the server process itself (not the SeaTable backend): 60 req/min per token, 120/min per IP, 30/min for new session creation, and 20 concurrent connections per token. +In **managed mode**, the MCP server adds its own rate limits to protect the server process itself (not the SeaTable backend): 60 req/min per token, 120/min per IP, 30/min for new session creation, and 20 sessions per token. When a token's 20 sessions are all taken, the one idle longest is closed to make room (a client returning to it gets `404` and initializes again, as the MCP spec requires); only if all 20 are in the middle of a request is the new one rejected with `429`. ChatGPT opens a new session for every tool call and never closes one, so it relies on this. ### Input Validation @@ -279,7 +279,8 @@ Available metrics: | `mcp_tool_calls_by_tool_total{tool}` | Counter | Total calls per tool (regardless of outcome) | | `mcp_tool_duration_seconds{tool}` | Histogram | Tool execution time | | `mcp_http_requests_total{method, status}` | Counter | HTTP requests by method and status code | -| `mcp_rate_limit_exceeded_total{type}` | Counter | Rate limit rejections (global/per_ip/per_token) | +| `mcp_rate_limit_exceeded_total{type}` | Counter | Rate limit rejections (`global`, `per_ip`, `per_token`, `pre_auth`, `oauth`, `connections`) | +| `mcp_sessions_evicted_total` | Counter | Idle sessions closed to make room for a new session of the same token | | `mcp_auth_validations_total{result}` | Counter | Auth validations (success/failure/cache_hit) | | `mcp_sql_failures_total{tool, reason}` | Counter | Failed `query_sql`/`execute_sql` calls (e.g. `subquery`, `join`, `wrong_statement_update`, `unknown_name`, `invalid_value`, `seatable_error`) | | `mcp_active_sessions` | Gauge | Currently active HTTP sessions | diff --git a/docs/openai-tunnel.md b/docs/openai-tunnel.md index 41b8c26..3570139 100644 --- a/docs/openai-tunnel.md +++ b/docs/openai-tunnel.md @@ -137,7 +137,7 @@ Done by someone in the workspace with developer mode access and *Tunnels Read + ## Open points -- Not yet verified end to end: whether `tunnel-client` passes the `mcp-session-id` header through correctly. The server is stateful and reclaims unused sessions after 30 seconds; problems would show up in the `seatable-mcp` logs on the first connection. +- Not yet verified end to end: whether `tunnel-client` passes the `mcp-session-id` header through correctly. The server is stateful; idle sessions are closed after 10 minutes or earlier when a token's 20 slots are full; problems would show up in the `seatable-mcp` logs on the first connection. - The `tunnel-client` image's entrypoint (`tunnel-client run`) and environment variable names are taken from OpenAI's repository and documentation, not from a running deployment. Update this file once a deployment has confirmed or corrected these points. diff --git a/package-lock.json b/package-lock.json index 83bd18b..2d8f915 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "@seatable/mcp-seatable", - "version": "1.9.6", + "version": "1.9.7", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@seatable/mcp-seatable", - "version": "1.9.6", + "version": "1.9.7", "license": "MIT", "dependencies": { "@modelcontextprotocol/sdk": "^1.17.4", diff --git a/package.json b/package.json index 9c1d3f7..d092108 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@seatable/mcp-seatable", - "version": "1.9.6", + "version": "1.9.7", "type": "module", "license": "MIT", "mcpName": "io.github.seatable/seatable", diff --git a/src/fingerprint.ts b/src/fingerprint.ts new file mode 100644 index 0000000..3c13d94 --- /dev/null +++ b/src/fingerprint.ts @@ -0,0 +1,15 @@ +import { createHash } from 'node:crypto' + +function fingerprint(value: string): string { + return createHash('sha256').update(value).digest('hex').slice(0, 12) +} + +/** Session IDs are credentials-adjacent routing values — log a fingerprint, never the value. */ +export function sessionFingerprint(sessionId: string): string { + return fingerprint(sessionId) +} + +/** Same rule for API tokens: enough to correlate one tenant's lines, never the credential. */ +export function tokenFingerprint(apiToken: string): string { + return fingerprint(apiToken) +} diff --git a/src/http/httpServer.ts b/src/http/httpServer.ts index b0a379b..da93eaa 100644 --- a/src/http/httpServer.ts +++ b/src/http/httpServer.ts @@ -6,9 +6,10 @@ import { StreamableHTTPServerTransport } from '@modelcontextprotocol/sdk/server/ import { MCP_RESOURCE_PATH, OAuthProvider, PROTECTED_RESOURCE_PATH } from '../auth/oauthProvider.js' import { TokenValidator } from '../auth/tokenValidator.js' import { getEnv, type ServerMode, VERSION } from '../config/env.js' +import { sessionFingerprint, tokenFingerprint } from '../fingerprint.js' import { logger } from '../logger.js' import { buildServer, getStaticToolDefinitions } from '../mcp/server.js' -import { activeConnections, activeSessions, httpRequestsTotal } from '../metrics/index.js' +import { activeConnections, activeSessions, httpRequestsTotal, rateLimitExceededTotal, sessionsEvictedTotal } from '../metrics/index.js' import { startMetricsServer } from '../metrics/metricsServer.js' import { RateLimitManager } from '../ratelimit/index.js' @@ -17,12 +18,6 @@ export interface StartHttpServerOptions { port?: number /** Session idle timeout in ms (default: 10 minutes) */ sessionIdleTimeoutMs?: number - /** - * Idle timeout for a session that initialized but never made a call - * (default: 30 seconds). These are the sessions a reconnect-happy client - * leaves behind, and each one holds a connection slot while it lives. - */ - unusedSessionTimeoutMs?: number /** Interval for checking idle sessions in ms (default: 60 seconds) */ sessionCheckIntervalMs?: number } @@ -32,8 +27,8 @@ type ActiveSession = { apiToken?: string /** Digest of the SeaTable API token that created this session; every later request must resolve to the same one. */ apiTokenDigest?: Buffer - /** False until the client makes its first call. Sessions that never do are reclaimed far sooner. */ - used: boolean + /** POSTs on this session still being worked on. A session with one is never evicted. */ + inFlight: number lastActivity: number close: () => Promise } @@ -42,16 +37,6 @@ function digest(value: string): Buffer { return createHash('sha256').update(value).digest() } -/** Session IDs are credentials-adjacent routing values — log a fingerprint, never the value. */ -function sessionFingerprint(sessionId: string): string { - return createHash('sha256').update(sessionId).digest('hex').slice(0, 12) -} - -/** Same rule for API tokens: enough to correlate one tenant's lines, never the credential. */ -function tokenFingerprint(apiToken: string): string { - return createHash('sha256').update(apiToken).digest('hex').slice(0, 12) -} - const MAX_BODY_SIZE = 10 * 1024 * 1024 // 10 MB async function parseJsonBody(req: IncomingMessage): Promise { @@ -167,6 +152,45 @@ export async function startHttpServer(options: StartHttpServerOptions = {}) { res.writeHead(401, headers).end(message) } + /** Sessions of this token that are in the middle of a request. */ + function busySessions(apiToken: string): number { + let busy = 0 + for (const session of sessions.values()) { + if (session.apiToken === apiToken && session.inFlight > 0) busy++ + } + return busy + } + + /** + * Close the session of this token that has been idle longest and has no + * request in flight. Never touches another token's sessions. A client that + * comes back to it gets 404, on which the MCP spec requires it to initialize + * a new session. + */ + async function evictIdlestSession(apiToken: string): Promise { + let victimId: string | undefined + let victim: ActiveSession | undefined + for (const [id, session] of sessions.entries()) { + if (session.apiToken !== apiToken || session.inFlight > 0) continue + if (!victim || session.lastActivity < victim.lastActivity) { + victimId = id + victim = session + } + } + if (!victim || !victimId) return false + logger.info( + { + session: sessionFingerprint(victimId), + token: tokenFingerprint(apiToken), + idleMs: Date.now() - victim.lastActivity, + }, + 'Session evicted' + ) + sessionsEvictedTotal.inc() + await victim.close() + return true + } + const trustProxy = env.TRUST_PROXY ?? true /** @@ -201,6 +225,7 @@ export async function startHttpServer(options: StartHttpServerOptions = {}) { for (const result of checks) { if (!result.allowed) { logger.warn({ ip, endpoint: req.url }, 'OAuth rate limit exceeded') + rateLimitExceededTotal.inc({ type: 'oauth' }) res.writeHead(429, { 'content-type': 'text/plain', 'retry-after': String(Math.max(1, Math.ceil(result.retryAfterMs / 1000))), @@ -243,6 +268,7 @@ export async function startHttpServer(options: StartHttpServerOptions = {}) { const preAuthResult = rateLimiter.preAuth.check(getClientIp(req)) if (!preAuthResult.allowed) { logger.warn({ ip: getClientIp(req) }, 'Pre-auth rate limit exceeded') + rateLimitExceededTotal.inc({ type: 'pre_auth' }) const retryAfter = Math.ceil(preAuthResult.retryAfterMs / 1000) res.writeHead(429, { 'content-type': 'text/plain', @@ -272,40 +298,56 @@ export async function startHttpServer(options: StartHttpServerOptions = {}) { // Before the slot is taken, so that a body that doesn't parse can't hold one. const body = await parseJsonBody(req) - // Connection limit (managed mode) + /* + * Connection limit (managed mode). The pool caps how many sessions of + * one token may work at the same time, not how many it has walked + * away from: ChatGPT opens a session per tool call and never closes + * one. When the pool is full, the session of this token that has been + * idle longest makes room. Only when every one of them is in the + * middle of a request is the client turned away. + */ if (rateLimiter && apiToken) { if (!rateLimiter.connections.acquire(apiToken)) { - logger.warn( - { - ip: getClientIp(req), - token: tokenFingerprint(apiToken), - active: rateLimiter.connections.active(apiToken), - limit: rateLimiter.connections.maxConnections, - }, - 'Connection limit exceeded' - ) - res.writeHead(429, { 'content-type': 'text/plain' }).end('Too many concurrent connections') - return + const evicted = await evictIdlestSession(apiToken) + if (!evicted || !rateLimiter.connections.acquire(apiToken)) { + logger.warn( + { + ip: getClientIp(req), + token: tokenFingerprint(apiToken), + active: rateLimiter.connections.active(apiToken), + busy: busySessions(apiToken), + limit: rateLimiter.connections.maxConnections, + }, + 'Connection limit exceeded' + ) + rateLimitExceededTotal.inc({ type: 'connections' }) + res.writeHead(429, { 'content-type': 'text/plain' }).end('Too many concurrent connections') + return + } } activeConnections.inc() } const mcpServer = buildServer(apiToken ? { apiToken } : undefined) - let sessionEstablished = false + let established: ActiveSession | undefined const transport = new StreamableHTTPServerTransport({ sessionIdGenerator: () => randomUUID(), onsessioninitialized: (id) => { - sessionEstablished = true mcpServer.setSessionId(id) - logger.info({ session: sessionFingerprint(id) }, 'Session initialized') - sessions.set(id, { + logger.info( + { session: sessionFingerprint(id), ...(apiToken && { token: tokenFingerprint(apiToken) }) }, + 'Session initialized' + ) + established = { transport, apiToken, apiTokenDigest: apiToken ? digest(apiToken) : undefined, - used: false, + // The initialize request itself is still being answered. + inFlight: 1, lastActivity: Date.now(), close: cleanup, - }) + } + sessions.set(id, established) activeSessions.inc() }, }) @@ -314,7 +356,7 @@ export async function startHttpServer(options: StartHttpServerOptions = {}) { const cleanup = async () => { if (cleaned) return cleaned = true - if (sessionEstablished) activeSessions.dec() + if (established) activeSessions.dec() if (apiToken && rateLimiter) { rateLimiter.connections.release(apiToken) activeConnections.dec() @@ -346,7 +388,15 @@ export async function startHttpServer(options: StartHttpServerOptions = {}) { * above was held until the process restarted. */ res.on('close', () => { - if (!sessionEstablished) void cleanup() + if (established) { + established.inFlight-- + return + } + logger.info( + { ip: getClientIp(req), status: res.statusCode, ...(apiToken && { token: tokenFingerprint(apiToken) }) }, + 'Initialize request ended without a session' + ) + void cleanup() }) await mcpServer.connect(transport) @@ -394,6 +444,19 @@ export async function startHttpServer(options: StartHttpServerOptions = {}) { } } + session.lastActivity = Date.now() + + // A POST is work in progress from the moment its body starts arriving + // until the response is done. A GET stream is not: ChatGPT opens one + // and drops it at once, other clients hold one for the whole session. + if (req.method === 'POST') { + session.inFlight++ + res.on('close', () => { + session.inFlight-- + session.lastActivity = Date.now() + }) + } + const body = req.method === 'POST' ? await parseJsonBody(req) : undefined // The session can end while a large body is on its way. @@ -402,8 +465,6 @@ export async function startHttpServer(options: StartHttpServerOptions = {}) { return } - session.used = true - session.lastActivity = Date.now() await session.transport.handleRequest(req, res, body) return } @@ -544,19 +605,15 @@ export async function startHttpServer(options: StartHttpServerOptions = {}) { logger.info({ host, port, endpoint: '/mcp' }, 'Streamable HTTP server listening') - // Idle session cleanup + // Idle session cleanup. Sessions that crowd the connection pool sooner than + // this are evicted on demand, see evictIdlestSession(). const sessionIdleTimeoutMs = options.sessionIdleTimeoutMs ?? 10 * 60 * 1000 const sessionCheckIntervalMs = options.sessionCheckIntervalMs ?? 60 * 1000 - // A session that never made a call gets a much shorter leash than one doing - // real work — it is holding a connection slot for nothing. Never longer - // than the ordinary idle timeout, whatever the caller passes. - const unusedSessionTimeoutMs = Math.min(options.unusedSessionTimeoutMs ?? 30 * 1000, sessionIdleTimeoutMs) const idleCheckInterval = setInterval(() => { const now = Date.now() for (const [sessionId, session] of sessions.entries()) { - const timeout = session.used ? sessionIdleTimeoutMs : unusedSessionTimeoutMs - if (now - session.lastActivity > timeout) { - logger.info({ session: sessionFingerprint(sessionId), used: session.used }, 'Closing idle session') + if (session.inFlight === 0 && now - session.lastActivity > sessionIdleTimeoutMs) { + logger.info({ session: sessionFingerprint(sessionId) }, 'Closing idle session') void session.close() } } diff --git a/src/mcp/server.ts b/src/mcp/server.ts index 02672eb..7cdf8b0 100644 --- a/src/mcp/server.ts +++ b/src/mcp/server.ts @@ -12,6 +12,7 @@ import { z } from 'zod' import { getEnv, parseBases, VERSION } from '../config/env.js' import type { CodedError } from '../errors.js' +import { sessionFingerprint } from '../fingerprint.js' import { logger } from '../logger.js' import { toolCallsByToolTotal, toolCallsTotal, toolDurationSeconds } from '../metrics/index.js' import { createClientFromEnv, createClientFromToken,SeaTableClient } from '../seatable/client.js' @@ -258,7 +259,7 @@ export class SeaTableMCPServer { // Common log context const logCtx = { requestId, - ...(this.sessionId && { sessionId: this.sessionId }), + ...(this.sessionId && { session: sessionFingerprint(this.sessionId) }), tool: toolName, } diff --git a/src/metrics/index.ts b/src/metrics/index.ts index 4fb7d59..efb0d4b 100644 --- a/src/metrics/index.ts +++ b/src/metrics/index.ts @@ -42,6 +42,12 @@ export const rateLimitExceededTotal = new Counter({ registers: [register], }) +export const sessionsEvictedTotal = new Counter({ + name: 'mcp_sessions_evicted_total', + help: 'Idle sessions closed to make room for a new one of the same token at the connection limit', + registers: [register], +}) + export const authValidationsTotal = new Counter({ name: 'mcp_auth_validations_total', help: 'Total number of auth token validations', diff --git a/src/ratelimit/index.ts b/src/ratelimit/index.ts index d5cba9c..f862d63 100644 --- a/src/ratelimit/index.ts +++ b/src/ratelimit/index.ts @@ -1,3 +1,4 @@ +import { tokenFingerprint } from '../fingerprint.js' import { logger } from '../logger.js' import { rateLimitExceededTotal } from '../metrics/index.js' import { ConnectionCounter } from './connectionCounter.js' @@ -64,7 +65,7 @@ export class RateLimitManager { if (opts.token) { const tokenResult = this.perToken.check(opts.token) if (!tokenResult.allowed) { - logger.warn({ type: 'per_token', ip: opts.ip }, 'Rate limit exceeded') + logger.warn({ type: 'per_token', ip: opts.ip, token: tokenFingerprint(opts.token) }, 'Rate limit exceeded') rateLimitExceededTotal.inc({ type: 'per_token' }) return { allowed: false, retryAfterMs: tokenResult.retryAfterMs, reason: 'Token rate limit exceeded' } } diff --git a/tests/connectionSlots.spec.ts b/tests/connectionSlots.spec.ts index 438f378..037fd1d 100644 --- a/tests/connectionSlots.spec.ts +++ b/tests/connectionSlots.spec.ts @@ -1,6 +1,6 @@ import type { AddressInfo } from 'node:net' -import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest' +import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest' vi.mock('../src/metrics/metricsServer', () => ({ startMetricsServer: vi.fn().mockResolvedValue(undefined), @@ -32,9 +32,9 @@ import { startHttpServer } from '../src/http/httpServer' /** * The production incident these tests encode: a client opened 14 sessions in 51 * seconds, made one tool call, and then met "Connection limit exceeded" for - * every further attempt. Two separate defects fed it — slots that a failed - * initialize never gave back, and slots that a successful-but-unused session - * held for the full 10-minute idle timeout. + * every further attempt. One defect fed it: slots that a failed initialize + * never gave back. Sessions that are idle but alive are handled by eviction, + * see tests/sessionEviction.spec.ts. */ const JSON_HEADERS = { @@ -121,14 +121,7 @@ describe('Connection slot accounting', () => { ids.push(res.headers.get('mcp-session-id')!) } - // Pool is full. - const blocked = await fetch(`${baseUrl}/mcp`, { - method: 'POST', - headers: { ...JSON_HEADERS, authorization: `Bearer ${token}`, 'x-forwarded-for': ip }, - body: INIT_BODY, - }) - expect(blocked.status).toBe(429) - + // Pool is full; DELETE frees a slot, so the next session needs no eviction. await fetch(`${baseUrl}/mcp`, { method: 'DELETE', headers: { 'mcp-session-id': ids[0], authorization: `Bearer ${token}`, 'x-forwarded-for': ip }, @@ -140,6 +133,8 @@ describe('Connection slot accounting', () => { body: INIT_BODY, }) expect(allowed.status).toBe(200) + expect(find('Session evicted')).toHaveLength(0) + ids.push(allowed.headers.get('mcp-session-id')!) for (const id of ids.slice(1)) { await fetch(`${baseUrl}/mcp`, { @@ -148,115 +143,6 @@ describe('Connection slot accounting', () => { }) } }) - - /** - * "Connection limit exceeded" used to log only an IP — and behind a proxy - * that IP is the same for everyone, so the line could not answer "whose - * pool is full?" or "how full?". - */ - it('logs which pool is exhausted and how full it is', async () => { - const token = 'slot-token-3' - const ip = '198.51.100.3' - const ids: string[] = [] - - for (let i = 0; i < 20; i++) { - const res = await fetch(`${baseUrl}/mcp`, { - method: 'POST', - headers: { ...JSON_HEADERS, authorization: `Bearer ${token}`, 'x-forwarded-for': ip }, - body: INIT_BODY, - }) - ids.push(res.headers.get('mcp-session-id')!) - } - - await fetch(`${baseUrl}/mcp`, { - method: 'POST', - headers: { ...JSON_HEADERS, authorization: `Bearer ${token}`, 'x-forwarded-for': ip }, - body: INIT_BODY, - }) - - const denied = find('Connection limit exceeded') - expect(denied.length).toBeGreaterThan(0) - const fields = denied[denied.length - 1].fields - expect(fields).toHaveProperty('token') - expect(String(fields.token)).not.toContain(token) - expect(fields).toHaveProperty('active', 20) - expect(fields).toHaveProperty('limit', 20) - - for (const id of ids) { - await fetch(`${baseUrl}/mcp`, { - method: 'DELETE', - headers: { 'mcp-session-id': id, authorization: `Bearer ${token}`, 'x-forwarded-for': ip }, - }) - } - }) -}) - -/** - * A session that initializes and is then abandoned is the common client bug. - * It should not hold a slot for the full idle timeout that working sessions get. - */ -describe('Unused session timeout', () => { - let server: ReturnType - let baseUrl: string - - beforeEach(async () => { - process.env.SEATABLE_SERVER_URL = 'http://localhost' - process.env.SEATABLE_API_TOKEN = 'test-token' - process.env.SEATABLE_MOCK = 'true' - delete process.env.SEATABLE_MODE - - server = await startHttpServer({ - port: 0, - sessionIdleTimeoutMs: 60_000, - unusedSessionTimeoutMs: 150, - sessionCheckIntervalMs: 50, - }) - baseUrl = `http://127.0.0.1:${(server.address() as AddressInfo).port}` - }) - - afterEach(async () => { - if (server) await new Promise((resolve) => server.close(() => resolve())) - }) - - async function init(): Promise { - const res = await fetch(`${baseUrl}/mcp`, { method: 'POST', headers: JSON_HEADERS, body: INIT_BODY }) - expect(res.status).toBe(200) - return res.headers.get('mcp-session-id')! - } - - it('reclaims a session that never made a call', async () => { - const sessionId = await init() - - await new Promise((r) => setTimeout(r, 300)) - - const res = await fetch(`${baseUrl}/mcp`, { - method: 'POST', - headers: { ...JSON_HEADERS, 'mcp-session-id': sessionId }, - body: JSON.stringify({ jsonrpc: '2.0', id: 2, method: 'tools/list', params: {} }), - }) - expect(res.status).toBe(404) - }) - - it('keeps a session that has done real work on the full idle timeout', async () => { - const sessionId = await init() - - const worked = await fetch(`${baseUrl}/mcp`, { - method: 'POST', - headers: { ...JSON_HEADERS, 'mcp-session-id': sessionId }, - body: JSON.stringify({ jsonrpc: '2.0', id: 2, method: 'tools/list', params: {} }), - }) - expect(worked.status).toBe(200) - - // Well past the unused timeout, far short of the idle timeout. - await new Promise((r) => setTimeout(r, 300)) - - const res = await fetch(`${baseUrl}/mcp`, { - method: 'POST', - headers: { ...JSON_HEADERS, 'mcp-session-id': sessionId }, - body: JSON.stringify({ jsonrpc: '2.0', id: 3, method: 'tools/list', params: {} }), - }) - expect(res.status).toBe(200) - }) }) /** diff --git a/tests/sessionEviction.spec.ts b/tests/sessionEviction.spec.ts new file mode 100644 index 0000000..ecbc8d2 --- /dev/null +++ b/tests/sessionEviction.spec.ts @@ -0,0 +1,324 @@ +import { request as httpRequest } from 'node:http' +import type { AddressInfo } from 'node:net' + +import { afterAll, beforeAll, beforeEach, describe, expect, it, vi } from 'vitest' + +vi.mock('../src/metrics/metricsServer', () => ({ + startMetricsServer: vi.fn().mockResolvedValue(undefined), +})) + +const VALID_TOKENS = new Set(['chatgpt-token', 'evicted-token', 'idle-token', 'visible-token', 'busy-token', 'tenant-a', 'tenant-b']) + +vi.mock('../src/auth/tokenValidator', () => ({ + TokenValidator: class { + async validate(token: string): Promise { + return VALID_TOKENS.has(token) + } + cleanup(): void {} + destroy(): void {} + }, +})) + +const logCalls: { fields: Record; msg: string }[] = [] +vi.mock('../src/logger', () => { + const record = (a: unknown, b?: unknown) => { + if (typeof a === 'object' && a !== null) logCalls.push({ fields: a as Record, msg: String(b ?? '') }) + else logCalls.push({ fields: {}, msg: String(a) }) + } + return { logger: { fatal: record, error: record, warn: record, info: record, debug: record, trace: record } } +}) + +import { startHttpServer } from '../src/http/httpServer' +import { rateLimitExceededTotal, sessionsEvictedTotal } from '../src/metrics/index' + +/** + * ChatGPT opens a fresh MCP session for every tool call and never sends DELETE. + * Each session goes initialize → notifications/initialized → a GET it aborts + * within milliseconds → one tools/call, and is then never touched again. + * + * On 15 November 2026 one ChatGPT Business user opened 29 sessions in about two + * minutes this way. Every one of them had done real work, so each held its slot + * for the full 10-minute idle timeout, and the 21st call met "429 Too many + * concurrent connections". The pool limits how many sessions may *work* at + * once; it must not limit how many a client has walked away from. + */ + +const JSON_HEADERS = { + 'content-type': 'application/json', + accept: 'application/json, text/event-stream', +} + +const INIT_BODY = JSON.stringify({ + jsonrpc: '2.0', + id: 1, + method: 'initialize', + params: { protocolVersion: '2025-03-26', capabilities: {}, clientInfo: { name: 'openai-mcp', version: '1.0.0' } }, +}) + +const find = (needle: string) => logCalls.filter((c) => c.msg.includes(needle)) + +async function counterValue(counter: { get(): Promise<{ values: { value: number; labels: Record }[] }> }, labels: Record = {}): Promise { + const { values } = await counter.get() + return values + .filter((v) => Object.entries(labels).every(([k, val]) => v.labels[k] === val)) + .reduce((sum, v) => sum + v.value, 0) +} + +describe('Session eviction at the connection limit', () => { + let server: ReturnType + let baseUrl: string + let port: number + + beforeAll(async () => { + process.env.SEATABLE_SERVER_URL = 'http://localhost' + process.env.SEATABLE_MODE = 'managed' + process.env.SEATABLE_MOCK = 'true' + process.env.SEATABLE_TOKEN_SECRET = 'session-eviction-spec-secret-long-enough' + delete process.env.SEATABLE_API_TOKEN + + // Only the clock is faked, so the per-token request limit (60/min) sees + // the requests spread over time as they were, not crammed into one burst. + vi.useFakeTimers({ toFake: ['Date'] }) + + server = await startHttpServer({ port: 0 }) + port = (server.address() as AddressInfo).port + baseUrl = `http://127.0.0.1:${port}` + }) + + afterAll(async () => { + vi.useRealTimers() + delete process.env.SEATABLE_MODE + delete process.env.SEATABLE_TOKEN_SECRET + if (server) await new Promise((resolve) => server.close(() => resolve())) + }) + + beforeEach(() => { + logCalls.length = 0 + // A fresh rate-limit window for every test. + vi.setSystemTime(Date.now() + 61_000) + }) + + // Every session gets its own IP: ChatGPT's egress addresses rotate, and the + // per-IP limits are not what this test is about. + let ipCounter = 0 + const nextIp = () => `203.0.113.${(ipCounter++ % 250) + 1}` + + function headers(token: string, ip: string, sessionId?: string): Record { + return { + ...JSON_HEADERS, + authorization: `Bearer ${token}`, + 'x-forwarded-for': ip, + ...(sessionId && { 'mcp-session-id': sessionId }), + } + } + + async function init(token: string, ip = nextIp()): Promise { + return fetch(`${baseUrl}/mcp`, { method: 'POST', headers: headers(token, ip), body: INIT_BODY }) + } + + /** The full ChatGPT sequence for one tool call. Returns the status of each step. */ + async function chatgptToolCall(token: string): Promise<{ init: number; call?: number; sessionId?: string }> { + const ip = nextIp() + const initRes = await init(token, ip) + await initRes.text() + const sessionId = initRes.headers.get('mcp-session-id') ?? undefined + if (initRes.status !== 200 || !sessionId) return { init: initRes.status } + + const notified = await fetch(`${baseUrl}/mcp`, { + method: 'POST', + headers: headers(token, ip, sessionId), + body: JSON.stringify({ jsonrpc: '2.0', method: 'notifications/initialized' }), + }) + expect(notified.status).toBe(202) + + // Opened and dropped at once — what Caddy logged as "aborting with incomplete response". + const abort = new AbortController() + const stream = await fetch(`${baseUrl}/mcp`, { + method: 'GET', + headers: { ...headers(token, ip, sessionId), accept: 'text/event-stream' }, + signal: abort.signal, + }) + expect(stream.status).toBe(200) + abort.abort() + + const call = await fetch(`${baseUrl}/mcp`, { + method: 'POST', + headers: headers(token, ip, sessionId), + body: JSON.stringify({ jsonrpc: '2.0', id: 2, method: 'tools/call', params: { name: 'list_tables', arguments: {} } }), + }) + await call.text() + return { init: initRes.status, call: call.status, sessionId } + } + + /** + * Holds a POST on the session in flight: the body is never finished, so the + * request is still being worked on from the server's point of view. + */ + function holdRequestOpen(token: string, sessionId: string): { release: () => Promise } { + const req = httpRequest({ + host: '127.0.0.1', + port, + path: '/mcp', + method: 'POST', + headers: { ...headers(token, nextIp(), sessionId), 'transfer-encoding': 'chunked' }, + }) + req.on('error', () => {}) + req.write('{"jsonrpc":"2.0","id":9,') + return { + release: () => + new Promise((resolve) => { + req.on('close', () => resolve()) + req.destroy() + }), + } + } + + async function close(token: string, sessionId: string): Promise { + await fetch(`${baseUrl}/mcp`, { method: 'DELETE', headers: headers(token, nextIp(), sessionId) }) + } + + it('serves the 15 November pattern: 29 sessions, each used for one tool call', async () => { + // 29 sessions over two minutes, as on 15 November. + const results = [] + for (let i = 0; i < 29; i++) { + results.push(await chatgptToolCall('chatgpt-token')) + vi.setSystemTime(Date.now() + 4_000) + } + + expect(results.map((r) => r.init)).toEqual(Array(29).fill(200)) + expect(results.map((r) => r.call)).toEqual(Array(29).fill(200)) + expect(find('Connection limit exceeded')).toHaveLength(0) + + // The oldest sessions made room; the newest are the ones still alive. + const evicted = find('Session evicted') + expect(evicted).toHaveLength(9) + const stillAlive = await fetch(`${baseUrl}/mcp`, { + method: 'POST', + headers: headers('chatgpt-token', nextIp(), results[28].sessionId), + body: JSON.stringify({ jsonrpc: '2.0', id: 3, method: 'tools/list', params: {} }), + }) + expect(stillAlive.status).toBe(200) + + for (const r of results) await close('chatgpt-token', r.sessionId!) + }) + + it('answers 404 on an evicted session, so a client knows to initialize again', async () => { + const ids: string[] = [] + for (let i = 0; i < 20; i++) { + const res = await init('evicted-token') + ids.push(res.headers.get('mcp-session-id')!) + } + expect((await init('evicted-token')).status).toBe(200) + + const res = await fetch(`${baseUrl}/mcp`, { + method: 'POST', + headers: headers('evicted-token', nextIp(), ids[0]), + body: JSON.stringify({ jsonrpc: '2.0', id: 2, method: 'tools/list', params: {} }), + }) + expect(res.status).toBe(404) + + for (const id of ids) await close('evicted-token', id) + }) + + it('evicts the session that has been idle longest, not the oldest one', async () => { + const ids: string[] = [] + for (let i = 0; i < 20; i++) { + const res = await init('idle-token') + ids.push(res.headers.get('mcp-session-id')!) + vi.setSystemTime(Date.now() + 1_000) + } + // The first session is still in use; the second is now the stalest. + const touched = await fetch(`${baseUrl}/mcp`, { + method: 'POST', + headers: headers('idle-token', nextIp(), ids[0]), + body: JSON.stringify({ jsonrpc: '2.0', id: 2, method: 'tools/list', params: {} }), + }) + expect(touched.status).toBe(200) + + expect((await init('idle-token')).status).toBe(200) + + const probe = async (id: string) => + (await fetch(`${baseUrl}/mcp`, { + method: 'POST', + headers: headers('idle-token', nextIp(), id), + body: JSON.stringify({ jsonrpc: '2.0', id: 3, method: 'tools/list', params: {} }), + })).status + expect(await probe(ids[0])).toBe(200) + expect(await probe(ids[1])).toBe(404) + + for (const id of ids) await close('idle-token', id) + }) + + it('never closes a session that has a request in flight, and then answers 429', async () => { + const ids: string[] = [] + for (let i = 0; i < 20; i++) { + const res = await init('busy-token') + ids.push(res.headers.get('mcp-session-id')!) + } + const held = ids.map((id) => holdRequestOpen('busy-token', id)) + // Let the server see every held request before the next initialize. + await new Promise((r) => setTimeout(r, 100)) + + const rejectedBefore = await counterValue(rateLimitExceededTotal, { type: 'connections' }) + const blocked = await init('busy-token') + expect(blocked.status).toBe(429) + expect(find('Session evicted')).toHaveLength(0) + expect(await counterValue(rateLimitExceededTotal, { type: 'connections' })).toBe(rejectedBefore + 1) + + // The rejection names whose pool is full, without the credential. + const denied = find('Connection limit exceeded').at(-1)!.fields + expect(denied).toMatchObject({ active: 20, busy: 20, limit: 20 }) + expect(String(denied.token)).toMatch(/^[0-9a-f]{12}$/) + + // Once the work is done, the same sessions become evictable again. + await Promise.all(held.map((h) => h.release())) + await new Promise((r) => setTimeout(r, 50)) + expect((await init('busy-token')).status).toBe(200) + + for (const id of ids) await close('busy-token', id) + }) + + it("never evicts another tenant's session", async () => { + const a: string[] = [] + for (let i = 0; i < 20; i++) a.push((await init('tenant-a')).headers.get('mcp-session-id')!) + const b: string[] = [] + for (let i = 0; i < 25; i++) { + const res = await init('tenant-b') + expect(res.status).toBe(200) + b.push(res.headers.get('mcp-session-id')!) + } + + for (const id of a) { + const res = await fetch(`${baseUrl}/mcp`, { + method: 'POST', + headers: headers('tenant-a', nextIp(), id), + body: JSON.stringify({ jsonrpc: '2.0', id: 2, method: 'tools/list', params: {} }), + }) + expect(res.status).toBe(200) + } + + for (const id of a) await close('tenant-a', id) + for (const id of b) await close('tenant-b', id) + }) + + it('makes evictions and the tenant behind each session visible', async () => { + const before = await counterValue(sessionsEvictedTotal) + const ids: string[] = [] + for (let i = 0; i < 21; i++) ids.push((await init('visible-token')).headers.get('mcp-session-id')!) + + expect(await counterValue(sessionsEvictedTotal)).toBe(before + 1) + + const evicted = find('Session evicted').at(-1)!.fields + expect(String(evicted.token)).toMatch(/^[0-9a-f]{12}$/) + expect(evicted).toHaveProperty('session') + expect(evicted).toHaveProperty('idleMs') + + // "Session initialized" carries the tenant, so a burst can be attributed. + const initialized = find('Session initialized') + expect(initialized).toHaveLength(21) + for (const line of initialized) expect(line.fields.token).toBe(evicted.token) + expect(JSON.stringify(logCalls)).not.toContain('visible-token') + + for (const id of ids) await close('visible-token', id) + }) +})