Repository navigation
readiness: Codex re-auth failure takes healthy multi-provider proxy out of service #5181
Description
Client or integration
Direct HTTP/API client
Area
Service lifecycle
Summary
When one or more Codex OAuth pool accounts require re-authentication, OpenCodex reports /readyz as HTTP 503 even though the proxy process is running and other configured providers/routes remain usable. In a single-replica Kubernetes deployment, the readiness failure removes the only Service endpoint and makes the entire proxy inaccessible.
Expected: Codex account-pool degradation should be reported as a component/provider failure. The proxy should remain ready when its HTTP server is healthy and at least one route/provider can serve traffic; native Codex traffic can fail or be marked degraded independently.
Reproduction
- Run OpenCodex in a container/Kubernetes pod with multiple providers and a Codex account pool.
- Leave one or more Codex accounts expired so they require re-authentication.
- Start the proxy and query
/healthzand/readyz. - Observe
/healthzand the proxy listener become available, while/readyzreturns 503 and Kubernetes marks the pod unready. - With a single replica, the Service has no ready endpoint even though non-Codex provider routes are configured.
Version
v2.59.0 (nordz0r/opencodex image)
Operating system
Debian 12 / Kubernetes (single-node k3s)
Provider and model
Codex account pool; other configured providers include Anthropic, Google Antigravity, OpenAI, and xAI routes.
Logs or error output
🚀 opencodex proxy running on http://localhost:10100
GET /healthz → health check
GET /api/* → management API
[opencodex] catalog sync: ... Codex accounts ... need reauthentication
GET /readyz -> 503
Kubernetes: Startup probe failed: HTTP probe failed with statuscode: 503
Pod: Running, 0/1 Ready, no process crashAfter changing the Kubernetes probe to /healthz, the pod becomes 1/1 Ready; authenticated catalog and non-Codex inference checks succeed while the Codex pool remains partially degraded. This indicates the process is viable and the account-pool state is too broad a readiness gate.
Screenshots and supporting files
Not applicable.
Redacted configuration
{
"deployment": "single replica",
"readinessProbe": "/readyz",
"livenessProbe": "/healthz",
"codexPool": "multiple OAuth accounts, some requiring re-authentication",
"otherRoutes": "configured non-Codex providers"
}Checks
- I searched existing issues and documentation.
- I removed secrets, tokens, account details, request credentials, and personal data.
Activity
fix(server): stop a local catalog-sync warning from un-readying the p…
fix: stop one account failure from taking the whole proxy out of serv…
lidge-jun commentedon Sep 19, 2026
Closed by #5199, squash-merged into
devas09698b9b3a.Your diagnosis was right and the mechanism turned out to be one layer over from where I first looked, so it is worth recording precisely.
/readyzwas driven by the post-startup Codex catalog sync, and a nonemptywarningin that sync's result was terminal for the gate alongsideok !== true. The two fields answer different questions.okis the sync's verdict on whether the essential work succeeded — config injection and the write admission before it.warningnames a degradation the sync itself decided to continue past: no catalog source so Codex keeps its native catalog, combos omitted, a history relabel left to Codex's own writer, or a caught refresh exception after which injection still runs and reports its ownok. None of those stops the process from accepting HTTP or routing to a provider, so none of them should have closed the gate. The fix is to readokalone. A throw, anull, orok !== truestay terminal, because none of those is a classified outcome — the startup path never reached a verdict, so the gate cannot claim one.The correction: the "Codex accounts need reauthentication" line you saw is a bare
console.warnand never reaches the structuredwarningfield, and a pool account's refresh failure is caught per account and swallowed. A single expired pool account was therefore not what closed your gate. What does reach it is a native main credential refresh failure, which sits outside that per-account catch, rejects the whole entitlement gather, and becomes the catch-all "catalog sync skipped" warning that pinned the process un-ready forever. Same blast radius you reported, one layer up.This boundary is not new, only extended. The Claude Code roster reconciliation in
src/cli/claude-agent-startup-sync.tsalready delays the ready transition without being permitted to fail it, on exactly this reasoning.Two related changes landed in the same PR and are worth knowing about, because they make a degraded pool actionable instead of merely non-fatal.
poolAccountDtono longer collapses three reauthentication causes into one boolean and guesses the reason from overlapping flags; the cause observed at the probe is carried through, so a WHAM 401 and a dead refresh grant are now distinguishable by the operator who has to act on one of them. That work is carried from #5191 with the author credited.Your workaround of pointing the readiness probe at
/healthzis no longer necessary. With this ondev,/readyzstays ready while the Codex pool is partially degraded, and native Codex traffic fails or degrades on its own.Evidence is exact-head hosted CI at
3252e665bb2c40211702a5f794f38ace99f080df: 29 successful checks, zero failures —test 1/4through4/4,macos 1/2(13751 tests) andmacos 2/2(13504 tests) all green, plusgates,structure gate,docker smoke,storage policy,api usage, and the keyring and npm-global matrices. No local suite was run in this lane.