[Bug]: Codex App repeatedly reconnects with "Codex WebSockets capacity reached" #7090

Client or integration

Codex App

Area

Proxy and routing

Summary

Long-lived idle incoming Responses WebSockets can consume the entire process-wide 128-connection budget. New Codex sessions then receive HTTP 503 at the local WebSocket upgrade boundary, retry, and only continue after falling back to HTTP. The affected proxy can still report a healthy /healthz.

Evidence and version boundary

• Observed deployment: 2.83.0. Codex App (bundled runtime 0.162.0-alpha.17.2) reported retries 1/5 through 5/5 with 503 Service Unavailable: Codex WebSockets capacity reached from the local /v1/responses endpoint, followed by falling back to HTTP and successful continuation.
• Still present in 2.84.0: source comparison against tag v2.83.0 and an isolated component reproduction against v2.84.0 / 6955343008e76e96d1c89bb32a9fc49f50e44fb9 confirm the same admission/idle lifecycle. This is not a claim that the complete 2.84.0 desktop deployment was run and produced the five-retry UI sequence.
• This report concerns Codex → OpenCodex incoming connections, not OpenCodex → provider connections or active-turn/session-lane admission. It does not establish a memory leak or a leaked reservation. Legitimate open but idle connections alone are sufficient to reach the limit.

Relevant 2.84.0 code

1. websocket-registry.ts, lines 5–24: global limit 128; reservations remain active until explicitly released.
2. serve-options.ts, lines 603–618: reserve before upgrade; when full, return Codex WebSockets capacity reached with 503.
3. websocket-handler.ts, lines 273–282: generate:false completes locally while keeping the socket reservation. Normal turn cleanup releases the turn lease; the incoming socket lease is released in close, lines 405–408.
4. live-sideband.ts, lines 25–48: idle timeout is 0; the same setting covers Responses and live sideband, and the comment documents the continuation-state coupling.

These lifecycle files are unchanged between v2.83.0 and v2.84.0; the upgrade branch is also unchanged. #6850 fixes active-turn cancellation and was already in 2.83.0. #6860 concerns upstream connection reuse and was also already present. The 2.84.0 changes in #6974 (upstream error logging) and #6997 (upstream pre-output refusal recovery) do not recover incoming connection slots before upgrade.

Expected behavior and proposed direction

A long-running desktop proxy should have a bounded, explicit lifecycle for inactive Responses connections so idle cached connections do not indefinitely deny new sessions. The connection cap should remain meaningful.

• Consider a Responses-specific idle reclamation policy, possibly with capacity-pressure reclamation of genuinely inactive connections. Protect ongoing generation, tool execution, and native steering/injection.
• Do not simply lower the shared Bun timeout: quiet live/voice sessions use the same handler and need a separate policy.
• Verify that an evicted connection reconnects and restores the full context on the next turn, including expired previous_response_id handling; avoid silent context loss or ambiguous replay.
• Expose content-free in-process admission/open/idle counts and close reasons so retained live connections can be distinguished from unreleased reservations.
• Add regression coverage for capacity recovery after idle reclamation and for normal close, abrupt disconnect, and unsuccessful upgrade cleanup.

Increasing 128 alone only postpones this condition. Releasing the connection lease at response.completed while the socket remains open would invalidate the connection limit and is not the proposed fix.

Reproduction

Observed desktop sequence (2.83.0)

1. Run the local proxy with websockets: true, and use Codex App against its local /v1/responses endpoint over a long-running desktop session.
2. Once incoming WebSocket admission is exhausted, start work in a session that attempts a new WebSocket. The proxy rejects the upgrade with 503: Codex WebSockets capacity reached.
3. In the observed Codex path, the UI/log shows retries 1/5 through 5/5, then HTTP fallback succeeds. The proxy's /healthz remains healthy.

The exact UI steps that accumulated all production connections have not been isolated. This is why the deterministic component reproduction below is provided separately.

Deterministic component reproduction against 2.84.0

Use an existing v2.84.0 checkout (6955343008e76e96d1c89bb32a9fc49f50e44fb9) and Bun 1.4.2. Save the following as /tmp/ocx-ws-capacity.ts. From that checkout, run:

bun /tmp/ocx-ws-capacity.ts "$PWD"

It imports only the real registry/admission code, listens on a random loopback port, sends no model requests, and stops its temporary server in finally. No production config or service is used. The upgrade/open/close plumbing is a minimal representation of the production ownership pattern, not the complete OpenCodex server.

import assert from 'node:assert/strict';
const root = process.argv[2];
const {
  tryReserveCodexWebSocket: reserve,
  codexWebSocketAdmissionMetrics: metrics,
  registerCodexWebSocket: register,
  unregisterCodexWebSocket: unregister,
} = await import(`${root}/src/codex/websocket-registry.ts`);
const clients: WebSocket[] = [];
const waitFor = async (check: () => boolean) => {
  const until = Date.now() + 5000;
  while (!check()) {
    if (Date.now() > until) throw new Error('timed out');
    await Bun.sleep(10);
  }
};
const server = Bun.serve({
  hostname: '127.0.0.1', port: 0,
  fetch(req, receivingServer) {
    const lease = reserve();
    if (!lease) return new Response(
      'Codex WebSockets capacity reached', { status: 503 });
    if (receivingServer.upgrade(req, {
      data: { admissionLease: lease },
    })) return undefined;
    lease.release();
    return new Response('WebSocket upgrade failed', { status: 426 });
  },
  websocket: {
    idleTimeout: 0,
    open(ws) { ws.data.admissionLease.bind(ws); register(ws); },
    message() {},
    close(ws) {
      unregister(ws);
      ws.data.admissionLease?.release();
      ws.data.admissionLease = undefined;
    },
  },
});
const connect = () => new Promise<WebSocket>((resolve, reject) => {
  const ws = new WebSocket(
    `ws://127.0.0.1:${server.port}/v1/responses`);
  clients.push(ws);
  ws.onopen = () => resolve(ws);
  ws.onerror = () => reject(new Error('connection failed'));
});
try {
  assert.equal(metrics().active, 0);
  for (let i = 0; i < 128; i++) await connect();
  assert.equal(metrics().active, 128);
  const rejected = await fetch(
    `http://127.0.0.1:${server.port}/v1/responses`,
    { headers: { Upgrade: 'websocket' } },
  );
  assert.equal(rejected.status, 503);
  assert.equal(await rejected.text(),
    'Codex WebSockets capacity reached');
  await Bun.sleep(2200);
  assert.equal(metrics().active, 128);
  clients[0].close(1000, 'synthetic normal close');
  await waitFor(() => metrics().active === 127);
  await connect();
  assert.equal(metrics().active, 128);
  for (const ws of clients) ws.close(1000, 'synthetic cleanup');
  await waitFor(() => metrics().active === 0);
  console.log('128 idle sockets reject the next upgrade; close restores capacity');
} finally {
  for (const ws of clients) { try { ws.close(); } catch {} }
  await server.stop(true);
}

No response.create is required to consume a connection slot. Separately, source inspection confirms generate:false also retains the socket slot while bypassing real inference. An equivalent, broader harness was run against both 2.83.0 and 2.84.0; its additional disconnect results are listed under Logs.

Suggested regression/acceptance cases

• Keep the hard cap for genuinely occupied connections; recover capacity without restarting once the chosen Responses idle policy expires or reclaims an eligible connection.
• Protect active responses, tool waits, and native control channels from idle eviction.
• After eviction, verify the next turn reconnects with correct context and expired-reference recovery.
• Preserve quiet live/voice sessions.
• Normal close, abrupt disconnect, and unsuccessful upgrades return reservations to baseline.

These are proposed acceptance cases, not claims that a fix has been implemented or validated.

Version

Source/component reproduction: 2.84.0, commit 6955343. Observed running deployment: 2.83.0. Bun 1.4.2; Codex runtime 0.162.0-alpha.17.2.

Operating system

macOS 26.6.2

Provider and model

Provider-independent incoming admission; observed with gpt-6-astra. Isolated reproduction uses no provider/model.

Logs or error output

Redacted observed deployment log (2.83.0, 2026-10-14; relative times):
T+0s  retrying sampling request (1/5) ... unexpected status 503 Service Unavailable: Codex WebSockets capacity reached, url: ws://127.0.0.1:10100/v1/responses
T+1s  retrying sampling request (2/5) ... same 503
T+2s  retrying sampling request (3/5) ... same 503
T+3s  retrying sampling request (4/5) ... same 503
T+4s  retrying sampling request (5/5) ... same 503
T+5s  falling back to HTTP
HTTP continuation then succeeded.

2.84.0 isolated component results (Bun 1.4.2):
128_held_129th_rejected: status=503, active=128
idle_2200ms_retains_all: active=128, closed=0
normal_close_returns_slot_replacement_accepted: active=128
all_normal_closes_released: active=0
actual_upgrade_false_released: status=426, active=0
tcp_destroy_without_ws_close_released: active=0
200_disconnects_immediately_after_handshake_write_no_leak: active=0; cumulative opened=330, closed=330

Thread IDs, request IDs, tracing context, local user paths and account details are omitted.

Screenshots and supporting files

No screenshots or raw logs are attached, to avoid disclosing unrelated desktop/account information. Source links and the self-contained component reproducer are included above.

Existing coverage checked: tests/codex-integration/active-registry-admission.test.ts manually reserves 128 leases before checking upgrade rejection; it does not exercise 128 real idle sockets and policy-driven recovery. tests/responses/ws-endpoint.test.ts pins idleTimeout=0 and the expired-continuation contract. These tests are unchanged in v2.84.0.

Scope limits: the field observations establish local admission rejection, but do not identify the lifecycle of all held production sockets. The isolated test imports the real registry/admission modules and uses minimal Bun upgrade/open/close plumbing; it does not execute the full OpenCodex handler. The 2.2-second observation is only a short retention check; the absence of an idle deadline comes from source inspection.

Redacted configuration

{
  "websockets": true,
  "port": 10100
}

Checks

• I searched existing issues and documentation.
• I removed secrets, tokens, account details, request credentials, and personal data.