Repository navigation
[Bug] Intermittent pooled Codex upstream response.failed 502 surge after v2.80 install (Windows; transport/egress isolated) #6740
Description
Client or integration
Codex CLI
Area
Proxy and routing
Summary
On Windows 11, the OpenCodex 2.80.0 canonical ChatGPT-sign-in Responses proxy intermittently returned HTTP 502 upstream_server_error while other requests completed normally. The problem spans gpt-6.1-sol low/medium/high effort and Fast/Standard requests, with some other-model impact. The 502 burst subsequently subsided without a proxy restart, downgrade, or permanent egress change, but Fast-mode response latency remained much slower than the previous-day baseline. We need to distinguish a true upstream terminal response.failed from a transport/session or account-pool problem.
Expected: canonical ChatGPT-backed requests should consistently yield upstream terminal success/error events, with accurate actionable diagnostics; any failure should not be mislabeled as VPN failure or rate limiting.
Reproduction
- Use Windows 11 with OpenCodex
2.80.0, the canonicalopenaiChatGPT-login forward provider and a single pooled OAuth account. GPT-6.1 Sol with normal selectable reasoning and Fast on/off (not hard-pinned). - Start the already-configured local OpenCodex proxy and issue ordinary, legitimate Codex CLI
/v1/responsesrequests, including ongoing medium/large cached-context turns. There is no deterministic per-request trigger; the fault appeared intermittently across independently created and ongoing conversations. - Observe alternating 200 and 502 statuses. The HTTP 502 is generated after the WebSocket request has already been sent and upstream frames have arrived.
- Confirm that basic TLS connectivity remains healthy. A short same-account A/B test of two VPN regions (without changing model/tier/effort) did not eliminate the errors.
- For latency, compare against native Codex direct ChatGPT login using the same installed CLI and the same user configuration, overriding only provider/base URL per invocation (read-only ephemeral tasks); the direct path was faster in two short paired tests. The two paths' actual upstream account identity and affinity have not been proven equivalent, so this is a symptom rather than a conclusive regression attribution.
Version
2.80.0. Package installed around 09:48 local time (UTC+08:00) on 2026-10-08. Latency had already deteriorated before installation. Official 2.79.0 was installed in a separate test prefix without touching production; source comparison of 1,795 old and 1,798 new files found 22 modified and 3 added, with the principal Responses/WebSocket/forward-error source modules byte-identical. Therefore no proof 2.80.0 alone caused the fault. The isolated 2.79.0 proxy booted/health-checked successfully, but could not authenticate live production model requests without transferring credentials (not done).
Operating system
Windows 11 (desktop). Service runtime Bun 1.4.0, Node.js used by CLI launcher. Local proxy on loopback; production network route can change between two outbound regions.
Provider and model
Canonical openai provider, ChatGPT sign-in account pool (single active account), gpt-6.1-sol. Also occasional other-model failures. Both Fast (priority requested) and Standard are selectable; no forced model binding.
Logs or error output
2026-10-07 10:00–11:59 UTC+08:00: 1,526 requests / 1 HTTP 502 (same model and pool route)
2026-10-08 10:00–11:59 UTC+08:00 (partial): 895 requests / 130 HTTP 502
2026-10-08 11:32: 43 HTTP 502 / 217 rolling-15-min GPT-6.1 requests
status=502, errorCode=upstream_server_error
failureStage=protocol-prelude, failureCause=upstream-fault
terminalStatus=failed, terminalSource=synthetic
codexWsStage.sent=true, upstreamFrames=5, relayedEvents=3
codexWsStage.firstResponseMs≈1.4s, closeCode=null
attempts[0].sendCount=1, retry/recoveryKinds=[]
upstreamError="An error occurred while processing your request..."
After 12:06 local: 502 incidence dropped substantially without service restart.
Ongoing Fast issue: 314 recent Fast requests all had priority requested, but
313 successes echoed service_tier=default, tier confirmation=assumed,
responseTierAuthoritative=false, 0 authoritative priority confirmations.
Successful Fast first-output P50 ~9.54s vs previous-day representative
Standard medium P50 ~4.43s (non-equivalent loads; not billing proof).
Different from upstream WebSocket close-code 1011 issue #5887: here upstream sent and relayed frames, but reported a terminal failure. Account IDs, credentials, error correlation IDs and prompts have been omitted.
Screenshots and supporting files
There are no public attachments containing private requests. Aggregated request counts and latency samples are stated above. Further strictly redacted traces can be supplied if requested.
Redacted configuration
{
"providers": {
"openai": {
"adapter": "openai-responses",
"authMode": "forward",
"codexAccountMode": "pool",
"requestPacing": {"enabled": true, "models": {"gpt-6.1-sol": {"minIntervalMs": 125}}},
"retryOnReset": {"enabled": true, "replacements": 1}
}
},
"accountPoolStrategy": "fill-first",
"websockets": false
}websockets=false here is the local inbound client transport and is not evidence that the canonical upstream Codex WebSocket is disabled. All credentials, account identifiers, routing IPs, request contents and customer data have been removed.
Questions: (1) How to distinguish actual upstream response.failed from session failure using safe diagnostics? (2) Is a safe refresh of the single-account pooled session possible without pinning/changing credentials or replaying a potentially submitted request? (3) Is there authoritative tier-acceptance evidence for ChatGPT Fast? (4) Which observed timings should be compared between native direct and OpenCodex after account affinity is controlled?
Checks
- I searched existing issues and documentation.
- I removed secrets, tokens, account details, request credentials, and personal data.
Activity
github-actions commentedon Oct 8, 2026
Issue reopened
The report now contains the information required by the automated check. Thanks for updating it.
kirin-bot139 commentedon Oct 8, 2026
Additional bounded A/B evidence (2026-10-08, Windows, v2.80.0; sensitive material omitted):
We tested the same installed Codex CLI, same GPT-6.1 Sol model, medium reasoning, priority/Fast request, same Chinese prompt, same local egress, read-only ephemeral sessions, comparing:
We also made an initial simpler comparison with user-config disabled, but that was confounded by different MCP/startup configuration and should NOT be treated as definitive.
With the full user configuration retained on both sides:
item.completederror-type items and one agent message; no terminal failure. This could reflect common startup/tool noise, not a uniquely OpenCodex failure.Limitations: only two fully-configured pairs; the upstream ChatGPT account identity/affinity was not independently proven equivalent, and cloud queue load varies. This establishes a performance symptom, not proven causal proxy overhead or a version regression.
Please advise an authoritative way to compare native vs forwarded ChatGPT-login latency and Fast-tier acceptance using the same OAuth account, and which diagnostic spans should be captured for the upstream queue, WS prelude, response generation, and relay without storing prompts/credentials. We have avoided changing production routes, model bindings, OAuth state, and retry policy.
kirin-bot139 commentedon Oct 8, 2026
Follow-up (2026-10-08 ~13:50 local UTC+08), redacted and independently verified:
The live single-account
poolroute (OpenCodex 2.80.0, GPT-6.1 Sol Fast/priority, Windows) regressed again after temporary recovery. The last 15 minutes contained ~285 requests, 276 HTTP 200 and 9 HTTP 502, with first-output P50 ~19.1s, full-duration P50 ~24.5s, and upstream WS first-response P50 ~1.97s. A 5-minute subset had 8/86 HTTP 502 and first-output P50 ~20.4s. Windows network probes remained healthy; the proxy process/port did not restart, and neither egress route nor provider model/pool settings were changed.The recurrent 502s carry:
Failures occur across low/medium/high reasoning effort; all recent requests carried Fast. These are terminal upstream errors after inbound upstream WS frames, not proxy TLS/connect failures.
Important isolation caveat: A separate 2.80.0 loopback test service with isolated OPENCODEX_HOME/CODEX_HOME and
codexAccountMode=directattempted to forward the native Codex CLI caller credential, but returned twoheaders-only / upstream-fault502s after ~31s each. The standalone native Codex invocation succeeded. This isolated direct-mode service was promptly stopped; those failures are not a valid latency comparison of direct vs proxy nor proof of a v2.80 regression, because direct-mode configuration/auth header handling has not been independently validated. Production pool settings and accounts were not touched.Please investigate the pooled canonical upstream terminal
response.failedpattern, Fast-tier/queue latency, and, separately, whether caller-auth direct forwarding should work with the documented canonical provider. No tokens, addresses, request bodies, account IDs, or customer data included.kirin-bot139 commentedon Oct 8, 2026
Correction / counterexample to earlier proxy-overhead hypothesis (2026-10-08 ~13:58 UTC+08:00). A new concurrent, matched setup used the same Codex CLI, full user config, gpt-6.1-sol medium, Fast requested, same read-only ephemeral prompt:
During this upstream slowdown, native direct was slightly slower than the proxy, despite its earlier advantage in other pairs. Therefore do not infer a fixed OpenCodex overhead from the earlier A/B samples. Upstream session/account affinity and model-load variance remain un-controlled variables; both routes exhibit long pre-generation latency.
Independent production cohort: medium effort, Fast, 100k–200k input tokens had first-output P50 of 6.063s (13:10–13:25, n=97) versus 21.231s (13:45–13:55, n=57) with virtually identical median input (~173k). WebSocket first-response P50 changed only 1.467→1.878s, while generation-start P50 changed 5.770→20.011s. The recurrent 502s are separate terminal
upstream_server_errorevents after 5–6 real upstream WS frames.The earlier direct-mode isolated OpenCodex service failed with two
headers-only502s after ~31 seconds; that isolated configuration was shut down and its temporary state removed, and must not be used as a latency A/B.No production proxy/network/model/Fast/pool/parallel configuration was changed in any of these experiments. This update supersedes any stronger attribution to OpenCodex forwarding latency.
kirin-bot139 commentedon Oct 8, 2026
Controlled Windows v2.79.0 production rollback trial completed on 2026-10-08 (UTC+08:00). This supersedes any attribution of the latency cliff to v2.80.0 alone.
This rollback A/B establishes a genuine temporal correlation around install, but falsifies a simple "v2.80.0 code alone caused it; restore v2.79.0 fixes it" hypothesis. It strongly motivates investigation of upstream account/session affinity, model queue and transient capacity errors. No credentials, account IDs, IP addresses, prompts or customer data included. Please avoid non-idempotent automatic retries.
kirin-bot139 commentedon Oct 8, 2026
Mitigation evidence from production, 2026-10-08 ~15:06–15:09 UTC+08 (Windows OpenCodex 2.80.0, same service PID, ChatGPT subscription-backed GPT-6.1 Sol / Fast, network egress unchanged):
openairoute. 81/81 HTTP 200, 0 HTTP 502 across ~3 minutes; first-output P50 ~4.92s, full-duration P50 ~11.12s, WebSocket first response ~2.02s. Fast was requested on 100% of those requests. Service stayed on v2.80.0, same process.Please investigate why a single otherwise valid Pro pooled Codex account produced terminal
response.failed / upstream_server_errorat a high rate while the native-main route worked contemporaneously. Is there a supported, safe health/quarantine or account-eligibility signal for such upstream failures, without automatically replaying a potentially submitted request? Please do not suggest removing credentials or binding the selected model. Account IDs, emails, prompts, IPs and secrets are intentionally omitted.kirin-bot139 commentedon Oct 9, 2026
Additional reproducible production observations, 2026-10-09 09:38–09:45 UTC+08, retaining one selected paid pooled account and the same office egress; no credential/route/model modifications:
72pooled requests,56 HTTP200 / 16 HTTP502; first-output P50 18.25s, upstream WS first-frame/response P50 ~1.95s. Last 30 minutes:500requests,417 HTTP200 / 83 HTTP502, first-output P50 17.85s.failureCause=upstream-fault,failureStage=protocol-prelude,errorCode=upstream_server_error, a synthetic terminal response.failed, and upstream WebSocket frames present (usually 5–6), rather than a failure to dial the node.provider.openai.upstreamWebsocketis unset (default WebSocket). Source supports operator opt-outupstreamWebsocket=falseto use canonical HTTP/SSE for new requests. No HTTP/SSE A/B was performed and no production transport setting was changed, because the proposed execution attempt was blocked by the current execution environment safety guard. Requests remain WS; accounts, models, context, and upstream authentication are untouched.Could maintainers recommend a documented, account-safe, idempotence-aware way to compare HTTP/SSE vs WS for one canonical ChatGPT account and determine whether this specific response.failed surge is transport/session-related or a service-side capacity problem? We explicitly do not want automatic POST replay after frames have been received and do not want to switch the customer's active paid account. This comment contains no account IDs, tokens, prompts, private URLs or user data.
Additional severity evidence — 2026-10-09 10:17 UTC+08 (same selected pooled paid account, same Seoul HY2 IPv6 egress, OpenCodex 2.80.0; no transport/account/route changes): The completed 15-minute production performance snapshot covered GPT-6.1 Sol 226 requests, 130 HTTP 200 and 96 HTTP 502 (42.5% failures). By upstream WS connection cohort: fresh 70/139 failed (50.4%), reused 26/87 failed (29.9%). All 96 classified 502s in these cohorts had the recorded
upstream-fault / protocol-preludesignature; local network monitor remained current withflags=[], and the automatic route remained Tencent Seoul HY2 IPv6. Comparison remains observational (not a causal claim). The customer intentionally selected this paid account and cannot treat routing back to the primary account as a durable fix. Please advise a supported non-replayable-request-safe per-account mitigation or trace collection procedure. No requests have been automatically replayed and no auth, routing, tier, model, or WebSocket production setting was modified by this report.Follow-up cohort, 2026-10-09 10:47 UTC+08 — production account selection was explicitly checked by the documented CLI: the failing xiejiao paid pool account remains active, healthy, unpaused, with no reauthentication requirement; the primary account was not auto-selected. In a rolling 15-minute same-account GPT-6.1 window, 123 requests: 76 HTTP 200, 47 HTTP 502 (38.2%), first-output P50 on successes ~22.0s. Among those requests: new WS 37/68 failed (54.4%), reused WS 10/55 (18.2%). The pattern persists across low and medium reasoning; it is not proof that changing TTL, session identity or concurrency would fix it. There are no new live configuration changes, credential changes, automatic POST replays or forced provider failovers. We specifically need a supported same-account mitigation that does not replay sent requests or change the active account, and clear criteria for differentiating an upstream terminal
response.failedfrom recoverable pre-output connection failures. Note that 2.79 and 2.81 comparisons did not resolve the incident. Redacted aggregate stats only; no user data or secrets.Operational mitigation test — 2026-10-09 11:15 UTC+08 same-account reselect (same selected paid account, no new credentials): Via the documented
ocx account use openai <same-selected-account> --json, the operator performed exactly one same-account re-selection, preserving account identity and active Codex version 2.80.0/service PID/egress/model-choice. The CLI confirmed success; in-flight requests were not replayed or cancelled. Five minutes immediately before had 51 GPT-6.1 turns, 24 HTTP200 and 27 HTTP502 (52.9%). After allowing a 30-second settling gap, the next 53 naturally occurring turns had 33 HTTP200 and 20 HTTP502 (37.7%). The percentage is lower but still unacceptable; time-varying backend load and different sample sizes prevent causal attribution, and this is not a successful fix. Across the same pooled-account route's last 30 minutes, other models also failed: GPT-6 Astra 13/33 HTTP502, GPT-5.6 Luna 5/20 HTTP502, besides GPT-6.1 Sol 78/209. The error is not limited to a specific reasoning tier/model. The same-account reset is not being repeated. Please provide account-safe diagnostic guidance or a shipped fix for upstreamresponse.failed / upstream_server_errorafter the WS already receives frames, preserving a no-replay-after-send contract; a pure 1011/no-upstream-frame mitigation does not explain this signal. No identifying account data, prompts, tokens, request bodies, or customer details provided.Live transport A/B completed — 2026-10-09 11:23–11:25 UTC+08 official HTTP/SSE A/B, same selected pooled paid subscription account and OpenCodex 2.80.0 PID throughout, automatic VPN route unchanged. Via the supported, non-secret provider editor snapshot/apply workflow, only
providers.openai.upstreamWebsocket=falsewas toggled for new requests; no credential extraction or API billing and no automatic replay of committed requests. A fresh readback showed the flag false. Reverted immediately after a short live observation; the subsequent official snapshot compared deep-equal to the exact pre-test public baseline (changedFields=[]), active account unchanged, service not restarted.Strictly separated actual requests: in the preceding ~15-minute WebSocket baseline, 186 pooled-provider requests had 119 × HTTP200 / 67 × HTTP502 (36.0% failures), success first-output P50 ~18.9s. During the short SSE period, 20 genuine no-WS-stage HTTP/SSE requests: 14 × HTTP200, 6 × HTTP503 (30.0% failures), successful first-output P50 ~16.3s. All six SSE failures were actual upstream terminal HTTP503 with
errorCode=server_is_overloaded,failureCause=upstream-declined,failureStage=protocol-prelude,transportPhase=terminal_sse,terminalSource=upstream, one attempt. Separately six previously committed WebSocket exchanges overlapped the transport switch, 5 × 200 / 1 × 502; not included in SSE counts. New WS requests after revert still experienced intermittent 502. Samples/time/model mixes differ, so this is not proof of an SSE improvement or a client WS bug.Key conclusion: Both canonical upstream transports suffer significant errors on the same selected paid account, but the SSE path reports explicit
server_is_overloaded503 instead of synthetic WSupstream_server_error502. This points to upstream request admission/overload as a stronger hypothesis than a pure WebSocket-only transport regression, while account-affinity/service-side specificity remains unproven. Request maintainer guidance on correlation-safe, per-account server diagnostics and whether this SSE 503 code maps to the WS terminalresponse.failedseen here. We reverted; no production transport change remains. No account identifiers, tokens, prompt bodies, IPs, private URLs, or customer data disclosed.Additional controlled-diagnostics exclusions — 2026-10-09 11:35 UTC+08, load and quota exclusion (no production config changes): The selected paid Codex account was still active/Healthy; recent per-account weekly used quota observations were 74% from both response headers and WHAM, so there is no evidence of simple weekly-exhausted admission. An anonymized 45-minute cohort of 391 completed GPT-6.1 responses showed
HTTP502grouped by approximate local active requests at start: 0–1: 12/39 (30.8%); 2–3: 69/189 (36.5%); 4–6: 53/152 (34.9%); ≥7: 4/11 (36.4%). No monotone concurrency/failure pattern; the user’s existing parallel-agent count was not reduced. Across 378 successive request timestamp gaps, none were below the existing 125ms pacing interval, and 502 rates for <1 second vs ≥1 second gaps were 35.6% and 35.7%. Increasing client pacing has no demonstrated benefit in these samples. A separate one-hour audit of 578 naturally occurring pooled requests found 359 HTTP200, 218 upstream 5xx, one other; every request had one attempt andsendCount=1(no proxy-generated replay storm). A few redacted UTC timestamps, WS stage counts and HTTP/SSE error classifications were preserved locally. OpenAI public service status is aggregate healthy, which does not rule out account/model-specific failures. Could the maintainer advise whether the SSEserver_is_overloadedis associated with the same admission path as canonical WebSocketresponse.failed / upstream_server_error, and which privacy-safe upstream correlation header/request ID can establish this without forwarding tokens or prompts? Our existing issue remains unresolved.kirin-bot139 commentedon Oct 10, 2026
Additional production evidence for #6740 (2026-10-10, UTC+08; Windows 11; OpenCodex 2.80.0, Bun 1.4.0; one deliberately selected subscription-backed pooled Pro login; no account identifiers, tokens or raw prompts included).
Same-login and same-model time-window regression surrounding a network egress change (GPT-6.1 Sol; no Fast requests):
Within successful medium-effort 100k–150k input-token observations: US 114 successes first-output P50 5.48s; Korea 149 successes P50 19.79s. For 150k–200k medium: US 82 successes P50 9.62s, Korea 197 P50 17.64s. Successful medium inputs >=200k: US 117 P50 9.82s, Korea 205 P50 21.60s. Failure rows generally lack reported input tokens, so these matched successful-input bins are NOT failure-rate denominators.
Protocol/transport: 502s predominantly failureCause=upstream-fault, failureStage=protocol-prelude, errorCode=upstream_server_error. Failing WebSocket stage records often sent=true, upstreamFrames=5–6, relayedEvents=3–4, closeCode=null; no evidence of close 1009. Even reused WS after switch had 70/444 failures (15.8%) versus 0/226 before; fresh WS 125/437 failures (28.6%) versus 3/166 (1.8%) before. These are time windows, NOT an authenticated simultaneous egress A/B: egress/edge, time-varying backend load and session/account affinity remain confounded.
Network controls: both exits reached the same ChatGPT WS URL with no auth and no model inference (expected 401). Korea exit reached ICN edge in ~0.29–0.70s; US exit reached LAX in ~0.49–0.85s. Seoul IPv6 independently downloaded 8MB at ~34–40Mbps; BWH ~28–106Mbps. A sustained basic-connectivity outage is not supported. Production OpenCodex stayed healthy without restart, while observed external memory grew to ~5–5.6GB and 7–12 turns were active: a separate potential confound. The pooled account was deliberately selected; switching accounts is not acceptable as the test.
Request to maintainers: Please recommend a low-risk per-request or isolated-egress authenticated probe for the same pooled login, and/or a safe upstreamWebsocket=false short-term diagnostic with reliable revert while preserving active threads. The live provider PATCH accepts upstreamWebsocket:false, but the canonical-provider guard does not admit true and a whole-provider reload can clear thread-account maps. For this reason, we have NOT modified production transport, proxy settings, model/account identity or active stream connections. Is there telemetry for terminal response.failed/protocol-prelude that distinguishes region/edge/session/queue without logging sensitive user data?
kirin-bot139 commentedon Oct 10, 2026
Bounded real-traffic egress A/B follow-up to the 2026-10-10 data posted above (UTC+08, production OpenCodex 2.80.0; same deliberately selected subscription-backed pooled login and GPT-6.1 Sol; no synthetic inference, no account/model/version change).
At 10:55:27 the existing top-level auto selector was temporarily moved from Seoul HY2 IPv6 to BWH direct US exit, holding for ~90 seconds, then restored to its original Seoul auto selector at 10:57:17. The route was read back at each transition; production proxy PID persisted, the original BWH fallback was used directly (NOT a BWH-to-Seoul chain), and the selected pooled login remained unchanged.
Interpretation: BWH direct egress did not immediately eliminate the same 502/prelude signature or generation-start slowness. However, only 6 requests completed in the US window, and previously opened WS connections can outlive the selector switch; this cannot estimate a reliable region effect or disprove the earlier association with Seoul. The direct outlet was restored. No account switch, client process restart, automatic replay, provider transport change or permanent VPN pinning took place.
The upstream 502 events therefore need further diagnostics that distinguish real upstream response.failed and account/session/backend admission from local transport. A reversible per-request SSE-vs-WS diagnostic for the existing live pooled credential without losing thread affinity would be particularly valuable.
fix(logs): keep recognized bare upstream error classes in synthesized…
Ingwannu commentedon Oct 10, 2026
The newest counterexample is useful: the short US-egress window did not eliminate the failure, so the region correlation is not yet a causal diagnosis. Please keep the account, live transport and retry policy unchanged while gathering the next evidence; a provider reload is not a harmless per-request switch, and the current reload path explicitly clears OpenAI thread-account mappings.
There is an important diagnostic distinction in the current source (
dev3649d58f5f01a46a888a4c45949d83dbbe5e40bc):sent=truemeansws.send()returned and the turn may have executed upstream.upstreamFramescounts every socket frame, including unparsed/control traffic;relayedEventscounts Responses events forwarded downstream, not specifically a terminal failure. Those counts alone cannot prove that an upstreamresponse.failedframe was received.failureStage=protocol-preludeis the recorder's observable-progress classification. A terminal status can put a request there even when no semantic output was delivered.terminalSource=syntheticsays the proxy constructed its terminal envelope; that can carry an upstream error, but it is not by itself evidence of the original event type or backend queue.server_is_overloadedand a generic WSupstream_server_errormust remain separate observed error shapes. Their common backend admission cause is not established by the present samples.For a small sample of already-occurring failed and successful requests, please retain only UTC time, build/runtime version, actual upstream transport and socket reused/fresh, sent/frame/control/relayed counts, terminal event type as observed before any synthetic tail (or unknown), closed error type/code, terminalSource, close code, attempt/send counts, and send→first-frame / first-Responses-event / first-output / completion timings. Group comparisons by effort, input/cache size and stable account-affinity cohort locally; do not publish account identifiers, request/response IDs, headers, raw payloads or prompts. This should separate a received upstream terminal from local termination without another replay/credential-transfer experiment.
I have not verified a supported affinity-preserving per-request SSE override, nor a safe in-place session refresh for this case. I am not recommending provider PATCH/reload, account switching, automatic resend, transport toggling or a production restart. No live account/provider call or scan was performed for this triage; the issue remains open.
lidge-jun commentedon Oct 12, 2026
Source-level analysis of the recurring row, against
dev7af092c. The 2.80.0 tag has the same path. I had no Windows runner, no real account traffic and no live backend frames, so everything below about what your backend sent is an inference from source.What the row is. On the canonical pooled path, with the default upstream WebSocket and an HTTP client, this exact combination comes from one specific sequence:
status=502,errorCode=upstream_server_error,terminalSource=synthetic,failureStage=protocol-prelude,failureCause=upstream-fault,closeCode=null, frames received and relayed, and one send. The upstream sends a Responses WebSocket{"type":"error", ...}frame after the response has started, and the socket stays open.codex-ws-exchange.tsrelays theerrorframe as an SSEerrorevent and ends the exchange on it. This is not a socket close, so there is no close code.response.*terminal. It writes aresponse.failedtail itself (terminalSource=synthetic) and does not mark the stream aborted (passthrough-delivery.ts,onSyntheticwith reasonupstream_error).upstream-fault.The other outcomes look different. A socket death is logged as
transport-ambiguouswithstreamAborted. A genuine upstreamresponse.failedis logged withterminalSource=upstream.So the 502 is OpenCodex's label for an error frame the upstream sent; it is not a proxy-side transport failure. The quoted text is the upstream's own message. The counters do not name the events before the error, so this is a reading of the source, not an observation of your frames.
What the proxy dropped. The backend wraps WebSocket failures as
{type:"error", status|status_code, error:{...}}. Native Codex turns thatstatusinto an HTTP error of the same status (codex-rs/codex-api/src/endpoint/responses_websocket.rs,map_wrapped_websocket_error_event). Once the response had started, OpenCodex ignored the wrapper status. If your failing frames carrystatus: 503, they would be the same admission refusal that the SSE trial showed as 503server_is_overloaded, but current rows cannot show that. 2.83.0 (#6889) already keeps recognized overload and rate-limit codes. #6974 also keeps the wrapper's 5xx status in the log row and records which terminal frame ended the exchange. The client-facing terminal and the retry behavior do not change. Every 5xx belongs to the same transient class for account health, so routing does not change either.Version and runtime. The upstream WebSocket client did not change between 2.79.0 and 2.81.0;
git log v2.79.0..v2.81.0overcodex-ws*.ts,ws-upstream.tsandsrc/lib/*ws*is empty. The Bun 1.4.0 to 1.4.2 bump (#6713, first shipped in 2.81.0) fixes a Windowsfetchstreaming defect in which the secondreader.read()fails (#6684). That defect affects fetch bodies. It does not fit error frames received on a WebSocket, or an explicit upstream 503 over SSE. Your 2.79 rollback and the 81/81 control on the other login are consistent with this. The leading hypothesis is account-level or backend admission and capacity for that login, which cannot be checked from source.Account health. On 2.80 and 2.81, a terminal 5xx feeds only the consecutive-failure counter, which any interleaved success resets. At a 20–50% failure rate it rarely trips, and there is no ratio-based steering. 2.82.0 (#6860) adds a 60-second window: with at least 20 samples and a transient-failure ratio of 25% or more, the account is marked degraded for new threads. Bound threads and a manual pin stay where they are.
codexPinnedTransientPolicy: "detour-new-threads"lets new threads leave a pinned account that is degraded. No version replays a request after it has been sent.Next evidence (no identifiers needed). For a small sample of failures and successes that occur anyway, please keep only the following. They are available on a build with #6974, or from the existing rows except for the two new fields.
codexWsStage.ocxVersionandbunVersioncodexWsStage.terminalEventandterminalErrorStatus(new),reused,sent,upstreamFrames,controlFrames,relayedEvents,closeCode,firstFrameMs,firstResponseMs,elapsedMsstatus,errorCode,upstreamErrorTypeandupstreamErrorCode(2.83.0+, closed vocabulary),failureStage,failureCause,terminalSource, and the attempt'ssendCountPlease keep request IDs (including
upstreamRequestId), account labels, headers and payloads out of this thread. If an escalation needs request IDs, they belong in a private support channel.How to read the result:
terminalEvent=errorwithterminalErrorStatus=503across the failing rows ties the WebSocket 502s to the same admission refusal as the SSE 503s.terminalEvent=response.failedmeans the backend failed the response itself. A nullterminalEventwith a close code points to the transport.elapsedMson failures shows how long after the send the error came, which separates a fast refusal from a timeout after queueing.fix(logs): keep the Codex WebSocket error wrapper's 5xx status and re…