Skip to content

[Bug]: xAI Grok unusable on native Responses in 2.28.0 #2240

Closed

Description

@olddonkey

Client or integration

Codex CLI

Area

Provider adapter

Summary

In OpenCodex 2.28.0, OAuth xai/grok-4.5 and xai/grok-4.6 became unusable on the native Responses route. #2147 moved those models off the Chat Completions compatibility wire onto openai-responses passthrough so Codex would not sit blank until reasoning finished (#1886). Chat translation had been sanitizing Codex-private wire extensions as a side effect; passthrough forwards the client's bytes, and xAI rejects the request before inference.

The conversation-killing case is the second turn: Codex replays a reasoning item with "content": null, and xAI returns HTTP 400 while mislabeling the field as a compaction blob. A first-turn namespace / private-tool catalog can 422 the same session before that.

I have already handled this (fix PRs #2217 and #2237). Please assign the issue to @olddonkey.

Reproduction

  1. Run OpenCodex 2.28.0 with Codex CLI on OAuth xai/grok-4.5 or xai/grok-4.6 (native /v1/responses, no Chat Completions wire).
  2. Start a new conversation with the default Codex 0.147+ tool catalog.
  3. First message is rejected (private tools[].type = "namespace" / external_web_access).
  4. After the catalog is lowered, send a second message so Codex replays the previous reasoning item with "content": null.

Redacted second-request shape:

{
  "model": "grok-4.6",
  "input": [
    {"role": "user", "content": "hello"},
    {
      "type": "reasoning",
      "id": "rs_redacted",
      "encrypted_content": "<intact grok blob>",
      "content": null
    },
    {"role": "user", "content": "continue"}
  ]
}

On a captured failing body: verbatim request → 400; removing only the content key → 200; removing encrypted_content → 400 schema error (blob was intact). Stripping content: null unconditionally repairs Grok and breaks OpenAI, which requires that field.

Version

2.28.0

Operating system

macOS

Provider and model

xai / grok-4.5, xai / grok-4.6

Logs or error output

HTTP 422
Argument not supported: external_web_access
{"code":"invalid-argument","error":"Could not decode the compaction blob. Ensure it is unmodified from the compact response."}

The failing request often has no compaction item. xAI names the sibling field, not the null content channel that actually caused the 400.

Redacted configuration

{
  "model": "xai/grok-4.6",
  "route": "openai-responses",
  "auth": "oauth"
}

Checks

  • I searched existing issues and documentation.
  • I removed secrets, tokens, account details, request credentials, and personal data.

Activity

  1. added
    bugSomething isn't working
    account-poolOAuth, credentials, Codex pool, quota, failover, plans
    providerProvider adapters, OpenAI-compat presets, upstream API quirks
    on Aug 21, 2026
  2. olddonkey commented on Aug 21, 2026

    @olddonkey
    ContributorAuthor

    Linking the fixes. They are sequential, not alternatives — each failure is only reachable once the previous one is fixed, which is why this looked like several unrelated bugs:

    PR Layer When a session hits it
    #2217 namespace tool groups, external_web_access, defer_loading, unrepresentable groups first request
    #2237 "content": null on the replayed reasoning item second request
    #2228 compaction blob relayed to a backend that could not have minted it after a backend switch, and on compaction turns
    #2229 hardening: do not reshape a reasoning item carrying opaque provider state no user-visible symptom on Grok

    A fifth is in progress for cross-backend reasoning replay: switching models mid-conversation sends Grok's status field and Grok-minted blobs to OpenAI, which rejects both. Reproduced end to end:

    replay grok -> grok : OK
    replay grok -> SOL  : FAILED  Unknown parameter: 'input[1].status'
      ... status removed:
    replay grok -> SOL  : FAILED  The encrypted content ZvQ+...fBJg could not be verified.
      ... status and encrypted_content removed:
    replay grok -> SOL  : OK
    

    None of these carries a closing keyword, so the issue stays open until the last one lands.

    One trap worth recording here

    The two backends want opposite things from the same field. xAI refuses "content": null on a reasoning item; OpenAI requires it kept, because it binds the blob to the item's exact shape. A fix that strips the null key unconditionally repairs Grok and breaks OpenAI — I shipped exactly that locally and a live request caught it within minutes, while an independent static review of the same diff had returned SHIP. Anything touching this field needs a live check on both routes, not just a green suite.

    Related: authMode === "forward" is not a usable signal for "this destination is an OpenAI backend" — a noncanonical forward provider never receives the caller's credentials, so forward auth says nothing about which backend answers.

  3. lidge-jun commented on Aug 21, 2026

    @lidge-jun
    Owner

    리뷰 · 우선순위 76 / 80

    2.28.0 기본 Grok OAuth가 네이티브 Responses에서 죽음. 지금 dev HEAD 78f1942a0 src/providers/registry.ts:1030-1042가 grok-4.5/4.6 OAuth+inbound responses를 wire: "openai-responses"로 박음. 프로바이더 와이드 어댑터는 원래 openai-chat (:1002). #2147 이후 Chat 번역이 닦아주던 Codex private 확장이 패스스루로 그대로 xAI에 감. 첫 턴이 namespace / external_web_access로 422·400. 그걸 낮추면 두 번째 턴 "content": null이 400. 에러가 compaction blob을 욕하는데 blob은 멀쩡함.

    저자 표가 맞음. 시퀀셜임. #2217이 첫 요청 카탈로그. #2237이 두 번째 요청 null 채널. #2228이 백엔드 스위치/컴팩션 blob 목적지. #2229는 Grok 네이티브 증상이 아니라 하드닝. 각각 이전 층이 살아야 다음이 보임. 클로징 키워드 넣지 말 것. 이슈는 마지막까지 열어둠.

    코드 경로. sanitizeReasoningInputContent (src/adapters/openai-responses.ts:42-80)가 배열 content만 봄. null은 :59 통과. 패스스루 체인 (:1575)에 목적지 게이트가 없음. hosted web_search OpenAI 전용 필드 스트립도 없음 (:1568-1575). #2237이 null 드롭+OpenAI 유지 게이트. #2217이 namespace 평탄화+external_web_access 제거. #2238 L6에도 같은 필드 스트립이 들어 있는데 그 PR은 사이드카 스택임. 첫 턴 스트립은 dev 핫픽스 쪽에 둬라.

    #2217이랑 #2227이 modelWireDefaults.wire를 같이 소유하면 안 됨. 전자는 Responses 기본 유지+평탄화. 후자는 Chat 기본 복원. 둘 다 머지하면 나중 PR이 덮음. 기본 경로를 살리는 쪽이 사용자 수정임. 나머지는 옵트인 하드닝. 리베이스 싸움 하지 말 것. 저자가 적은 함정 그대로임. xAI는 content: null 거절, OpenAI는 그 키를 blob 셰이프에 묶음. 언게이트 스트립은 OpenAI를 깨먹음.

    다섯 번째 층은 아직 PR 없음. grok→SOL 스위치가 status랑 Grok blob을 OpenAI에 보냄. status 지우면 blob 검증 실패, 둘 다 지우면 OK. #2229 프로비넌스 후속이랑 같음. 이 이슈에 설계 전체를 넣지 말 것. types.ts/config.ts 스플릿 안 씹힘. #2188 사이드카랑 #2190 x_search는 별 선. 여기 본문이 아님. #2239는 같은 본문 중복. 그쪽 닫고 여기만 남겨라.

    해결방안: #2217+#2237을 dev에 먼저. 와이어 기본은 #2217/#2227 중 하나만. #2228/#2229는 이어서. 크로스-백엔드 status/blob은 후속 PR. 클로징 키워드 금지. #2239 닫기. 스플릿이 어댑터 새니타이저를 옮기면 열린 핫픽스는 리베이스하지 말고 닫고 다시 짜라.

    이 댓글은 grok-bot이 작성했습니다

  4. lidge-jun commented on Aug 21, 2026

    @lidge-jun
    Owner

    fixing sorry I think chat/completions and translating it in here is right direction plz tuned for next release in today sry

  5. olddonkey commented on Aug 21, 2026

    @olddonkey
    ContributorAuthor

    @lidge-jun you can assign the issue to me, I have the fixes. it is a pretty complicated one

  6. olddonkey commented on Aug 21, 2026

    @olddonkey
    ContributorAuthor

    The root cause is xAI's response API is very restrictive on the params.

  7. olddonkey commented on Aug 21, 2026

    @olddonkey
    ContributorAuthor

    What broke, and the eight fixes

    All of this was established by bisecting captured live requests against xAI and OpenAI. Where a claim below is an inference rather than a measurement, it says so.

    The structural cause

    #2147 moved OAuth grok-4.5/4.6 off the Chat compatibility wire onto native openai-responses passthrough, to fix blank-until-done streaming (#1886).

    Chat translation had been sanitizing Codex-private wire extensions as a side effect — a translator can only emit fields it knows how to build, so anything private was dropped for free. Passthrough forwards the client's bytes, and xAI validates strictly. The switch traded "the translator sanitizes for free" for "nothing sanitizes", and the sanitation was never rebuilt.

    That is the difference in kind worth remembering: translation is inherently a whitelist; passthrough is inherently a denylist. Every Codex-private extension is now a thing someone has to remember to strip.

    The failures are sequential

    Each was only reachable once the previous one was fixed, which is why they looked like unrelated bugs and why "fixed it" was wrong five times:

    # When it fires Symptom PR
    1 first request tools[].type = "namespace" (Codex 0.147+ private tool groups) → 422 #2217
    2 first request external_web_access on web_search, defer_loading on deferred tools → 400 #2217
    3 second request "content": null on the replayed reasoning item → 400 #2237
    4 compaction turns private namespace key on replayed call items #2217
    5 model/backend switch status + a blob minted elsewhere → 400 #2248 #2249 #2251 #2252

    Failure 3 is the conversation-killer: a fresh session dies on its second message.

    Where the code is

    All of this is in one PR: #2254.

    It was originally opened as eight PRs (#2217, #2228, #2229, #2237, #2248, #2249, #2251, #2252). They had to merge in a strict order, and the later four each carried the whole series as their diff — up to 27 files and +2830 lines for ~30 new ones — so reviewing any of them in isolation was not actually possible. They are now closed and superseded by #2254, which has the same 16 commits with each unit's evidence intact in its message. Their branches are untouched and still pushed, so the split can be restored if anyone prefers it.

    The commits, in order:

    namespace lowering + private tool-field stripping        (failures 1, 2, 4)
    compaction relay decided by destination, not forward auth
    remove a disproven causal claim from the reasoning guard  (comments only)
    drop the null reasoning content channel on routed destinations  (failure 3)
    strip cross-backend reasoning state on a model switch     (failure 5)
    compaction blobs compare serving identity too
    recover when an upstream rejects foreign opaque state
    strip output-only reasoning `status` unconditionally
    converge the two opaque-blob recovery call sites          (cleanup)
    

    Four traps that cost the most time

    1. xAI's error message names the wrong field. It calls the reasoning encrypted_content a "compaction blob":

    {"code":"invalid-argument","error":"Could not decode the compaction blob. Ensure it is unmodified from the compact response."}
    

    The failing request contained no compaction item at all. That message sent the investigation through stale-history, cross-backend and blob-corruption hypotheses for hours. Bisecting a captured body settled it — removing only the content key turned the 400 into a 200:

    variant result
    body verbatim 400 "Could not decode the compaction blob"
    encrypted_content removed 400 schema — the blob is required
    whole reasoning item removed 200
    only the content key removed 200

    2. The two backends want opposite things from the same field. xAI refuses "content": null on a reasoning item; OpenAI requires the shape kept, because it binds the blob to the item's exact shape. An unconditional strip fixes Grok and breaks OpenAI. That regression shipped locally and a single live request caught it within minutes — while an independent static review of the same diff had returned SHIP. Anything touching a blob-bearing item needs a live check on both routes; a green suite is not evidence.

    3. authMode === "forward" is not a signal for "this destination is an OpenAI backend." A noncanonical forward provider never receives the caller's credentials, so forward auth says nothing about which backend answers. #2228 replaces that inference with isOpenAiOperatedResponsesDestination.

    4. When two guards reject different fields of the same item, the first hides the second. #2251's recovery is armed for the upstream's blob-rejection error — but with the blob retained, status was retained too, and OpenAI rejects Unknown parameter: 'input[1].status' before validating the blob. The recovery correctly never matched, its unit tests all passed, and its gate was clean, while the live path was unchanged. #2252 removes the field that was firing first. A fix for the second guard is untestable until the first is gone.

    Design note: provenance of opaque state

    encrypted_content and compaction blobs are decodable only by the backend that minted them, so a model switch mid-conversation replays state the new destination cannot read. Three mechanisms, in order of preference:

    1. Deterministic pre-flight (fix(responses): strip cross-backend reasoning state on a model switch #2248, fix(responses): compare serving identity for compaction blobs too #2249) — a bounded in-process record of the identity that served each thread's previous turn. On a mismatch, strip before sending. One round trip, no waste.
    2. Recovery (fix(responses): recover when an upstream rejects foreign opaque state #2251) — for history the process never served (restart, TTL expiry, LRU eviction), the upstream's own narrow error identity is the only authoritative signal. Rebuild once and refetch, registered as a recovery kind alongside the existing image-413.
    3. Not built: guessing provenance from blob format. OpenAI's blobs are Fernet (gAAAAAB…), xAI's are not — so provenance is visible in the payload. Hinging correctness on the format of an opaque field means a format change silently strips everyone's blobs, so the upstream's own answer is used instead.

    The pre-flight's failure direction is deliberately asymmetric: a missed strip costs one degraded turn; a spurious strip is a permanent quality regression. It therefore keeps blobs when it has no record, and refuses to record at all rather than fall back to volatile identity dimensions — the credential tuple contains the OAuth generation, and six of the eight bind sites are rotation or refresh rebinds, so comparing it would make every key-pool provider look like a destination switch on refresh.

    Verification

    Live, through a locally deployed build of all eight, with real blob-bearing items on both routes:

    scenario result
    grok first turn, namespace + external_web_access + defer_loading 200
    grok replay, own blob + content: null 200
    the originally captured 2.28.0 failing body, verbatim 200
    openai replay, own blob intact 200
    warm record, grok blob → SOL 200, sendCount=1
    cold record, grok blob → SOL 200, sendCount=2, recoveryKinds=['opaque-blob-rejection']
    cold record, grok blob + foreign compaction → SOL 200, recovery
    cold record, SOL blob → grok 200, recovery

    Two ways this suite silently tests nothing, both of which bit us: without include: ["reasoning.encrypted_content"] no backend issues a blob, so every replay check runs on a blob-free item and passes for the wrong reason; and response.completed on the OpenAI route omits reasoning items, so a parser that lets it replace the accumulated output_item.done items drops exactly what is under test. Assert a non-zero blob length as a precondition, and confirm a recovery actually ran by reading sendCount/recoveryKinds in usage.jsonl rather than trusting the HTTP status.

    Test gates

    bun run test per branch: #2217 13749/13760 · #2228 13738/13749 · #2248 13742/13753 · #2249 13773/13784 · #2251 13777/13788. All eight merged together: 13780 pass, 10 skip, 1 fail across 868 files.

    Every one of those runs lands on exactly one failure — tests/key-login-live-update.test.ts > "notify after key login pushes the merged row and keeps modelCosts on live and disk".

    That failure is pre-existing and unrelated, verified directly rather than assumed: running that file alone on untouched dev @ 03735eca6 fails identically (0 pass, 1 fail, same assertion at tests/key-login-live-update.test.ts:100).

    (Plain bun test with no arguments hangs on this tree with high CPU and no progress — use bun run test.)

    Not a bug

    grok-4.6-build appearing in the log UI is resolvedModel, the serving id xAI's Grok CLI backend reports (grok-4.5 → grok-4.5-build). It was present in usage logs eight hours before any of these fixes; it only became visible once requests started succeeding.

  8. Ingwannu commented on Aug 21, 2026

    @Ingwannu
    Owner

    Maintainer status after checking the current stack and exact PR heads:

    No closing keyword yet. Once the parent boundary is fixed, exact-head CI/review threads are clean, and the combined OpenAI+xAI replay smoke cases remain green, we can reassess closure.

  9. olddonkey commented on Aug 21, 2026

    @olddonkey
    ContributorAuthor

    Consolidated into a single PR: #2254.

    The eight PRs opened earlier (#2217, #2228, #2229, #2237, #2248, #2249, #2251, #2252) are closed and superseded. They had to merge in a strict order, and the later four each showed the whole series as their diff — up to 27 files and +2830 lines for roughly 30 new ones — so reviewing them in isolation was not actually possible.

    #2254 carries the same 16 commits with each unit's evidence and rationale intact in its message, plus the combined test gate. Nothing is dropped: the eight branches are unchanged and still pushed, so the split can be restored if a reviewer prefers it.

    The summary comment above has been updated to match.

  10. madgiraffe1 commented on Aug 21, 2026

    @madgiraffe1

    Independent reproduction on macOS with OpenCodex 2.28.0 and Codex Desktop CLI 0.149.0-alpha.4: xAI provider health succeeds, while a minimal xai/grok-4.6 Responses request containing tools=[{type:web_search,external_web_access:true}] returns HTTP 400 with Argument not supported: external_web_access. The otherwise identical request with tools=[{type:web_search}] returns HTTP 200 and output OK. The Desktop failures were reproduced with both high and xhigh reasoning. No credentials or personal data were included.

  11. olddonkey commented on Aug 21, 2026

    @olddonkey
    ContributorAuthor

    This should be fixed in the next version. I expect the next release to default to Chat Completions, but Responses is still much better — until that new default is clearly stable, we should add a UI switch so people can keep using Responses.

  12. olddonkey commented on Aug 21, 2026

    @olddonkey
    ContributorAuthor

    The response API with grok is good enough that I can trust and use.

  13. lidge-jun commented on Aug 21, 2026

    @lidge-jun
    Owner

    yes I made it opt-in in gui! and we could keep dogfood!

  14. lidge-jun commented on Aug 21, 2026

    @lidge-jun
    Owner

    Resolved by the landed train: #2255 made Chat the OAuth default for grok-4.5/4.6 (with the unconditional tier policy), #2258 landed the full sanitize/recovery series for the explicit Responses lane, and #2266 added the GUI opt-in switch you confirmed — Responses stays dogfoodable per account without breaking the default. Ships in 2.28.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    account-poolOAuth, credentials, Codex pool, quota, failover, plansbugSomething isn't workingproviderProvider adapters, OpenAI-compat presets, upstream API quirks

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions