Repository navigation
[Enhancement]: Mid-stream OpenAI socket resets can break Codex account affinity #919
Description
Client or integration
Codex App
Area
Proxy and routing
Summary
A native OpenAI Responses stream can return HTTP 200 and partial SSE data, then fail while OpenCodex reads the upstream body with a socket reset such as ECONNRESET.
Codex correctly treats the resulting stream disconnected before completion: Transport error: network error: error decoding response body as retryable and replays the logical request. OpenCodex should not replay a partially delivered stream because that could duplicate tool calls or other side effects.
The routing side effect is incorrect: the passthrough inspection path converts a mid-stream body-read failure into synthetic status 502 and records it through recordCodexUpstreamOutcome. Repeated client retries or concurrent streams can then reach upstreamFailoverThreshold, soft-avoid a healthy credential, clear thread affinity, and promote another account even though no account-specific HTTP rejection occurred.
Expected behavior:
- Keep the synthetic mid-stream 502 in request diagnostics.
- Do not apply that synthetic transport terminal to account failure streak, cooldown, affinity, or active-account selection.
- Release only a quota-probe lease owned by the interrupted request.
- Continue recording actual HTTP
429/402/5xxand semantic SSEresponse.failedoutcomes. - Do not replay the partially delivered request inside OpenCodex; leave bounded retry to Codex.
- Preserve the selected account for the Codex retry unless an actual account-specific rejection occurs.
- Record enough correlation-safe metadata to distinguish upstream body-read failure, downstream pipe failure, client cancellation, and terminal delivery.
Relevant source:
src/server/relay.ts: passthrough inspection maps body-read failure to a synthetic 502 terminal.src/server/responses/core.ts: native passthrough terminal recording forwards that synthetic status to account routing.src/codex/routing.ts: repeated transient outcomes can clear affinity and promote another account.structure/04_transports-and-sidecars.md: mid-stream SSE failures are intentionally not replayed by the proxy.
Required regression coverage:
- HTTP 200 SSE emits a partial delta, then its body throws
ECONNRESETthree times for the same thread. - Each request remains observable as a synthetic mid-stream failure, but account health, affinity, and active selection remain unchanged.
- A held scoped or account-wide probe lease is released exactly once.
- An actual SSE
response.failedwith status 502 still records an account failure. - A client cancellation remains distinct and does not become an upstream failure.
- No extra upstream request is sent by OpenCodex after partial delivery.
Related: #914 handles DNS/reachability failures before response headers; this issue handles failures after HTTP 200 and partial body delivery. #913 covers compact failover only after an actual upstream 429/402. The same client-visible error is tracked upstream in openai/codex#29087.
Reproduction
- Configure OpenAI Codex-login pool routing and bind a thread to account A.
- Return HTTP 200 with
text/event-streamfrom the OpenAI upstream. - Deliver at least one valid SSE delta, then make the upstream body reader throw an
ECONNRESET-shaped error beforeresponse.completed. - Let Codex retry the request several times or reproduce the same failure concurrently.
Current: OpenCodex records synthetic 502 account failures. At the configured threshold it can clear A's affinity or promote another account without an actual HTTP rejection.
Expected: Codex can retry, but OpenCodex keeps account routing state unchanged and never replays the partial stream itself.
Version
2.10.0; the relevant passthrough terminal and routing behavior remained on dev at c72acb3e368d2613c41d6d2b15a771895843f03f when checked on 2026-08-03.
Operating system
Windows 11 Pro 10.0.26200. The account-outcome classification issue is platform-independent; the observed resets occurred in the Windows native passthrough path.
Provider and model
OpenAI (Codex login / forward account pool); native Codex Responses streaming.
Logs or error output
stream disconnected before completion: Transport error: network error: error decoding response body
TypeError: The socket connection was closed unexpectedly
code: ECONNRESET
Screenshots and supporting files
None. Exact request IDs, thread IDs, account identifiers, quota values, and timestamps are intentionally omitted.
Redacted configuration
{
"providers": {
"openai": {
"adapter": "openai-responses",
"authMode": "forward",
"codexAccountMode": "pool"
}
},
"upstreamFailoverThreshold": 3
}Checks
- I searched existing issues and documentation.
- I removed secrets, tokens, account details, request credentials, and personal data.
Activity
github-actions commentedon Aug 3, 2026
Automated translation bookkeeping — detected language: unknown.
lidge-jun commentedon Aug 3, 2026
Triaged, and the answer is more interesting than a yes or no. Two corrections and a reframe.
OpenCodex does not replay a partially delivered stream. Retry helpers operate before a
Responseis returned (src/lib/upstream-retry.ts:203-238), and the failed-tail relay documents and implements "not a resend" (src/server/relay.ts:98-137). Any logical replay you observed is the client's own subsequent request. That half of the report is already satisfied.The routing side effect is real.
consumeForInspection()maps an upstream body-read exception to a syntheticfailed, 502labeledtransportPhase="mid_stream",terminalSource="synthetic"(src/server/relay.ts:988), andcodexForwardTerminalOutcomeRecorder()(src/server/responses/core.ts:427-469) records it as an account failure. Enough of them soft-avoid the account and clear affinity.But it was designed that way, deliberately.
devlog/_fin/260722_issue_bug_sweep/030_patch_s_sticky_502.mdintroduced this exact behavior, and its test matrix listsupstream read rejection (200 후 socket reset)with the expected outcometransient 실패 기록, affinity 해제. The comment is still in the source. So this is not a bug that slipped in — it is a policy someone chose, and reversing it is a policy decision.I spent four audit rounds trying to make it a local fix and failed each time.
terminalSource: "synthetic"records who manufactured the terminal event, not who caused the failure — and the eager path's catch (src/server/relay-eager.ts:256) additionally wraps inspection and rewriting, so the same label can mean an upstream body failure or a local exception in our own relay.The real question is whether post-200 transport reliability should affect account routing at all. Arguments both ways: every pool account shares the host, so rotating cannot repair a host-side drop — but account health tracks transient reliability generally, which is why ordinary 5xx responses count.
Recorded with #914 in
devlog/_plan/260803_transport_attribution/000_plan.md, since they are the same question before and after HTTP 200. Leaving open until that policy is decided rather than shipping a guard that suppresses real signal.luvs01 commentedon Aug 3, 2026
A new live reproduction on installed opencodex
2.10.0confirms the routing side effect described here.Observed on 2026-08-03 UTC, with account identifiers and request contents omitted:
12:34:31and12:36:24. Multiple records had a non-nullfirstOutputMs, so at least part of the burst occurred after output had started.stream disconnected before completion: Transport error: network error: error decoding response bodyand an ordinary sampling retry againsthttp://127.0.0.1:10100/v1/responses.openaiprovider label to a different privacy-safe pool label. That pool account returned 429 twice, followed by additional router-level 429 responses. Codex turns then terminated withexceeded retry limit, last status: 429 Too Many Requests.fill-firstwithautoSwitchThreshold: 100. Ongoing affinity should not rotate for ordinary fill-first selection, which leaves the transient-failure path as the matching trigger. The installed routing code defaultsupstreamFailoverThresholdto 3 and records synthetic mid-stream 502 as an account transient failure that can clear affinity and promote an alternate.This evidence does not establish that every post-200 transport failure should be account-neutral. It does establish that the current synthetic-502 policy can break ongoing affinity and turn a recoverable transport failure into account rotation plus 429 exhaustion. A regression test should cover both relay modes and assert that client cancellation remains neutral, while policy work separates host/transport health from credential/account health.
lidge-jun commentedon Aug 3, 2026
Re-labeled
bug→enhancement, keepingproxy,streaming,tools. The behavior you describe is real and reproduces; the reclassification is about what kind of change would resolve it, not about whether it happens.The mid-stream reset clearing account affinity was introduced deliberately.
devlog/_fin/260722_issue_bug_sweep/030_patch_s_sticky_502.mdrecords the intended outcome astransient 실패 기록, affinity 해제— a mid-stream socket reset is recorded as a transient failure precisely so account health stops treating that account as sticky. Escalated transient health then requires two successes to recover.So the current behavior is the policy working as written, and changing it is a policy decision about how aggressively affinity should survive a transport failure. That is an enhancement request against a recorded design, not a defect against an intended one.
Two things worth knowing for whoever picks this up:
The synthetic/mid-stream label is not a shortcut to the answer. It records who manufactured the terminal event, not who caused the failure, and the eager path's catch wraps our own inspection and rewriting too — so keying the new behavior off it would produce a guard that looks right and misattributes.
#922 is adjacent but not this. It makes shared network failures account-neutral, which is a different question from whether a confirmed mid-stream reset should drop affinity.
Keeping it open with the label corrected so it sorts with the other accepted behavior changes rather than sitting in the bug queue.
[-][Bug]: Mid-stream OpenAI socket resets can break Codex account affinity[/-][+][Enhancement]: Mid-stream OpenAI socket resets can break Codex account affinity[/+]lidge-jun commentedon Aug 6, 2026
Closing after a policy review. What the report shows is real and was reproduced, but the behavior is the intended account-health policy: post-200 transport failures count against the account so that persistent upstream trouble rotates traffic away. Treating a mid-stream socket reset as never-account-attributable would mask genuinely unhealthy accounts, and
terminalSource="synthetic"alone does not establish that the reset was network-local — especially on the eager relay path.The right evolution here is a transport-attribution policy (classifying post-200 failures before they touch affinity), which is an enhancement, not a defect fix. Happy to reopen against a concrete attribution proposal, or a repro isolating a non-network cause for the resets.
luvs01 commentedon Sep 8, 2026
Concrete attribution proposal against
dev7797586a8899c673eab48886a490e85b480c6d72: keep post-200 transport reliability in account health, but distinguish a locally enforced relay budget from a transport failure before recording the account outcome.I isolated one non-network case using the unchanged eager relay, its real SSE inspector, an identity rewrite, and
createTranslatorBudget. All input is generated in memory; there is no fetch or socket. The first valid text delta is delivered in every case. The healthy and local-budget cases receive identical valid SSE chunks; only the local budget changes.onSyntheticargumentscompleted; no synthetic failuretranslation_buffer_limit; budget overflow count 1failed, no reasonECONNRESETupstream_reset; budget overflow count 0failed, no reasonThe first delta is 61 bytes and the next valid delta is 312 bytes. Every case finishes once and returns retained budget usage to zero. This is an isolated mechanism reproduction, not a claim about the original incident's cause or a real network capture. It directly observes the relay callbacks; the downstream account effect below is established from the current recorder code, rather than from a live account mutation.
The distinction is lost at the accounting boundary. The owned rewrite-budget reservation can raise the local error, and the failed tail already distinguishes its wire code. But the eager callback carries only failure kind/reason. The caller maps an unqualified synthetic failure to 502, which becomes a transient account outcome. Before streaming, the request-decode handler already maps this error type to 413.
Proposed narrow policy:
terminalSource=syntheticproves a local cause.translation_buffer_limitwire code. This is an explicit policy choice: rotating accounts should not be the recovery action for the proxy's own per-request limit. A generic error string/code, malformed upstream payload, arbitrary inspection/rewrite exception, or upstream body rejection does not qualify.connect_neutralor write a host failure: this is not pre-connection reachability evidence. Do not record 200, which would incorrectly clear prior health. The caller/neutral path only releases a matching owned probe lease and preserves failure streaks, cooldowns, affinity and active-account selection.The regression matrix would pair the real local-budget case with reader rejection and unclassified inspection/rewrite controls, then cover a later cleanup error, real terminal before delivery failure, client cancellation, and stale/foreign probe ownership. The small budget above is a test control; production limits would not change.
Would this specific local-limit exception and provenance boundary be an acceptable basis for reopening the enhancement? I have not changed routing policy or reopened the issue. Broader post-200 account-neutral behavior is not part of this proposal.