Repository navigation
[Bug]: Chat JSON-to-SSE fallback drops tool calls/reasoning and overwrites finish_reason with stop #3770
Description
Resolved in dev
#3803 (ac4a7659fd) preserves tools, reasoning, usage and finish semantics in JSON-to-SSE completion delivery.
Verified against dev 5759d9ea2f1e7281cdc01eb9628f2e0a123fb59c. Original report by @turin-dev. The attribution record was added in #3811.
Client or integration
Direct HTTP/API client
Area
Streaming
Summary
When the Chat endpoint's internal Responses replay returns JSON instead of SSE, stream: true enters a lossy JSON-to-SSE fallback. The fallback serializes only message.content and hard-codes finish_reason: "stop", even though responsesJsonToChatCompletion has already produced tool calls, reasoning content, and the correct finish reason.
The same upstream JSON produces:
stream: false: the expectedmessage.tool_callsandfinish_reason: "tool_calls".stream: true: no tool-call delta at all; just the assistant role and a successful-lookingstopfollowed by[DONE].
A reasoning-plus-text response loses its reasoning_content while retaining normal text. A response incomplete due to max_output_tokens changes from length to stop.
Impact: tool-capable clients cannot execute the requested tool, and clients cannot detect token-limit truncation from the terminal chunk. This is specific to the JSON response fallback, not a claim about the usual SSE-to-SSE or native Chat passthrough paths.
Expected: the synthesized stream preserves the semantic fields and terminal reason of the already-converted Chat completion: indexed delta.tool_calls, delta.reasoning_content, and choices[0].finish_reason.
Reproduction
Executed the unmodified handleChatCompletions and translation modules from commit a349b521bb02f76c7b8820c13633a8076dcec378 with a mocked handleResponses boundary that returns application/json. Routing and unrelated runtime integrations are stubbed so the test needs no provider, credentials, or running proxy.
Save the following as repro-json-chat.ts in the repository root, then run it in a separate process with bun run repro-json-chat.ts (the mocks are process-global):
import { mock } from "bun:test";
let upstreamBody: Record<string, unknown>;
mock.module("./src/adapters/openai-responses", () => ({ FORWARD_HEADERS: [] }));
mock.module("./src/lib/retry-after", () => ({ resolveClientRetryAfter: () => undefined }));
mock.module("./src/lib/token-estimate", () => ({ estimateTokens: () => 1 }));
mock.module("./src/router", () => ({
UnknownRoutingPolicyError: class extends Error {},
NoEligiblePolicyCandidateError: class extends Error {},
routeModel: () => ({
providerName: "fixture", modelId: "fixture", codexAccountMode: "direct",
provider: { adapter: "openai-responses" },
}),
}));
mock.module("./src/routing/request-evidence", () => ({ evidenceFromBody: () => ({}) }));
mock.module("./src/server/adapter-resolve", () => ({ resolveWireProtocolOverride: (_name, _model, provider) => provider }));
mock.module("./src/server/request-decompress", () => ({ readJsonRequestBody: (req) => req.json() }));
mock.module("./src/server/request-log", () => ({
addFinalRequestLog() {}, httpStatusForRequestLogTerminal: () => 200, recordFirstOutput() {},
}));
mock.module("./src/server/relay", () => ({ responseWithDeferredRequestLog: (r) => r }));
mock.module("./src/server/responses", () => ({ handleResponses: async () => Response.json(upstreamBody) }));
mock.module("./src/codex/native-main-admission", () => ({ tryClaimNativeMainProfileForTurn: () => false }));
mock.module("./src/server/chat-native", () => ({ isNativeChatRouteEligible: () => false, handleNativeChatCompletions() { throw Error("unexpected native path"); } }));
mock.module("./src/server/effort-row", () => ({ parseRequestEffortRowId: () => null }));
mock.module("./src/server/fast-row", () => ({ parseSyntheticRowId: () => ({}) }));
mock.module("./src/providers/openai-tiers", () => ({ isCanonicalOpenAiForwardProvider: () => false }));
mock.module("./src/codex/loopback-target", () => ({ CODEX_RESERVE_HELPER_UNSUPPORTED_MESSAGE: "", isCodexReserveHelperUnsupported: () => false }));
const { handleChatCompletions } = await import("./src/server/chat-completions");
const fixtures = {
tool: { status: "completed", output: [{ type: "function_call", call_id: "call_fixture", name: "lookup", arguments: '{"q":"hello"}' }] },
reasoning: { status: "completed", output: [{ type: "reasoning", summary: [{ type: "summary_text", text: "Fixture reasoning." }] }, { type: "message", role: "assistant", content: [{ type: "output_text", text: "Answer." }] }] },
truncated: { status: "incomplete", incomplete_details: { reason: "max_output_tokens" }, output: [{ type: "message", role: "assistant", content: [{ type: "output_text", text: "Partial answer." }] }] },
};
for (const [name, fixture] of Object.entries(fixtures)) {
upstreamBody = { id: "resp_fixture", ...fixture, usage: { input_tokens: 1, output_tokens: 2 } };
for (const stream of [false, true]) {
const req = new Request("http://localhost/v1/chat/completions", {
method: "POST", headers: { "content-type": "application/json" },
body: JSON.stringify({ model: "fixture", messages: [{ role: "user", content: "Hello" }], stream }),
});
const res = await handleChatCompletions(req, {} as any, {} as any);
const text = await res.text();
if (!stream) {
console.log(name, "JSON", res.status, JSON.stringify(JSON.parse(text).choices[0]));
} else {
const frames = text.split("\n\n").filter(s => s.startsWith("data: {")).map(s => JSON.parse(s.slice(6)));
console.log(name, "SSE", res.status, JSON.stringify(frames.map(f => f.choices[0])));
}
}
}This exercises six handler calls: three identical upstream fixtures, each with the client's stream flag false and true. The real request handler returns HTTP 200 for all six. It is a handler-level reproduction with injected upstream responses, not a live-provider/full-server integration test.
Version
Upstream dev commit a349b521bb02f76c7b8820c13633a8076dcec378, checked 2026-09-06.
Operating system
Ubuntu 24.04.3 LTS, Linux x64; Bun 1.4.2.
Provider and model
Synthetic Responses upstream; model identifier fixture. Relevant when the internal Responses replay returns JSON to a streaming Chat request.
Logs or error output
Normalized observed differences (generated IDs omitted):
| Fixture | Client stream=false | Client stream=true |
|---|---|---|
| Completed function call | message.tool_calls[0].function.name = "lookup"; finish tool_calls |
No delta.tool_calls; finish stop |
| Reasoning plus text | reasoning_content = "Fixture reasoning." and content = "Answer." |
Text remains; no reasoning delta |
| Token-limited partial answer | content = "Partial answer."; finish length |
Same partial text; finish stop |
For the tool fixture, the entire streamed choice sequence is:
[
{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null},
{"index":0,"delta":{},"finish_reason":"stop"}
]It ends with [DONE], without ever transmitting the call ID, function name, or arguments.
Screenshots and supporting files
Root cause: src/server/chat-completions.ts, JSON-to-SSE fallback.
- The completed Chat object already exists at line 450.
- Lines 462–465 extract only
message.content. - Lines 466–470 emit role/text chunks only.
- Line 471 replaces the actual finish reason with the constant
"stop".
Suggested fix: serialize the full supported Chat completion message into indexed deltas and reuse the converted choice's finish reason. Keep usage and one terminal [DONE], with the existing stream-budget/lifecycle requirements. Add focused handler tests for tool-only JSON, multiple tool calls, reasoning plus text, and max-output-token truncation.
Distinct from #3767: that report concerns structured refusal content lost inside src/chat/outbound.ts. Here the JSON-to-Chat conversion successfully preserves the tool/reasoning fields, and the subsequent SSE synthesis in the HTTP handler discards them.
Duplicate checks included issue searches for JSON fallback/upstream and finish_reason, plus PR searches for JSON fallback, JSON upstream + tool, and synthesized Chat streams. No matching fix/report was found; the defect remains in the pinned current-dev source.
Redacted configuration
No real configuration is needed. Routing is a synthetic direct Responses fixture in the harness. All identifiers, text, function names, arguments, and usage values are synthetic.
Checks
- I searched existing issues and documentation.
- I removed secrets, tokens, account details, request credentials, and personal data.
Activity
lidge-jun commentedon Sep 6, 2026
리뷰 · 우선순위 72 / 80
이 이슈는
/v1/chat/completions에서 클라이언트가stream: true를 보냈는데, 내부 Responses 재생이 SSE가 아니라 JSON으로 돌아올 때 타는 JSON→SSE 합성 폴백이 이미 만든 Chat completion의 의미를 버리는 버그입니다. 같은 업스트림 JSON이stream: false면message.tool_calls와finish_reason: "tool_calls"가 그대로 나오고,stream: true면 tool-call delta가 한 번도 없이 assistant 역할과finish_reason: "stop"만 나갑니다. reasoning이 섞인 답은reasoning_content가 사라지고 본문만 남고,max_output_tokens로 끊긴 답은length가stop으로 바뀝니다. 도구를 쓰는 클라이언트는 “성공한 빈 stop”처럼 보여서 호출을 실행하지 못하고, 토큰 한도 잘림도 마지막 청크로 감지하지 못합니다. 리포터가 말한 대로 평소 SSE→SSE 번역이나 네이티브 Chat 패스스루 문제가 아니라, HTTP 핸들러의 JSON 폴백 합성만의 구멍입니다.지금
devHEAD는bd1cda99c(package2.44.0,#3768수동·자동 credential 선택 공유까지 들어간 release-244). 리포터 기준 커밋a349b521b는 현재 HEAD의 조상이고, HEAD의src/server/chat-completions.tsL450–L480 근처도 같은 코드입니다. 흐름은 이렇습니다.handleChatCompletions가 내부적으로 Responses를 돌린 뒤 JSON을 받으면responsesJsonToChatCompletion(정의는src/chat/outbound.tsL563 근처)으로 이미 완전한 Chat completion을 만듭니다. 그 함수는 tool call을message.tool_calls로 모으고, reasoning summary/text를reasoning_content에 넣고, finish reason을tool_calls/length/stop으로 고릅니다(대략 L605–L624).stream: false는 그 객체를 그대로 JSON으로 돌려주니 맞습니다. 문제는 바로 아래 스트리밍 합성입니다. L462–L465에서message.content문자열만 꺼내고, L466–L470은 role/text 청크만 만들고, L471은finish_reason을 상수"stop"으로 덮어씁니다. 그래서 변환기가 이미 살려 둔 tool/reasoning/정확한 종료 이유가 HTTP 응답에서 증발합니다.평소 스트리밍 경로인
src/chat/outbound.ts의responsesSseToChatCompletionsSse는 이미delta.tool_calls,delta.reasoning_content, 그리고 올바른finish_reason을 냅니다(대략 L245–L307, L448). 즉 “Chat로 번역하는 능력”은 레포에 있고, JSON 폴백만 그 능력을 다시 쓰지 않는 상태입니다.#3767과도 겹치지 않습니다.#3767은outbound.ts번역 단계에서 refusal 텍스트가 빠지는 문제이고, 여기서는 JSON→Chat 변환은 tool/reasoning을 이미 보존한 뒤 HTTP 핸들러 합성이 버립니다. types/config 분할 캠페인과도 무관해서 close-don't-rebase 대상이 아닙니다. 재현 스크립트도handleChatCompletions경계에handleResponses만 JSON으로 mock한 핸들러 단위라서, 라이브 프로바이더 없이도 고치기 좋은 형태입니다.라인 450 -
responsesJsonToChatCompletion으로 이미 tool_calls·reasoning·finish_reason이 있는 completion이 생기는데, 바로 아래 합성은 이 객체를 “텍스트만 있는 가짜 스트림”으로 깎아 냄라인 465 -
message.content문자열만 추출해서tool_calls/reasoning_content/ content가 null인 tool-only 케이스를 전부 놓침라인 466-470 - role·텍스트 delta만 내보내서 indexed
delta.tool_calls와delta.reasoning_content프레임이 한 번도 안 나감라인 471 - 변환기가 고른 finish reason을 무시하고 상수
"stop"으로 덮어써서tool_calls·length가 클라이언트로 전달되지 않음경로
src/chat/outbound.tsresponsesJsonToChatCompletion- 비스트리밍·변환 쪽은 이미 올바른데, 핸들러 폴백만 그걸 재사용하지 않아 stream true/false 의미가 갈라짐경로
src/chat/outbound.tsresponsesSseToChatCompletionsSse- SSE 경로에는 tool/reasoning emitter가 이미 있으므로, JSON 폴백은 이 시맨틱을 복제하거나 completion→chunk 직렬화 헬퍼를 공유하는 쪽이 안전함메인테이너의 판단이 필요한 지점
outbound.ts의 기존 SSE emitter 시맨틱에 맞춰 공용 헬퍼로 뺄지,chat-completions.ts안에서만 completion.choices[0]을 chunk로 펼칠지reasoning_content를 폴백 스트림에 넣을지(비스트리밍·SSE 경로와 맞추려면 넣는 편이 일관됨). 일부 클라이언트가 모르는 필드를 무시하는지만 확인max_output_tokensincomplete를 핸들러 단위 테스트로 고정할지(리포터 제안 그대로 채택해도 됨)#3767refusal 수정과 같은 PR에 묶을지, JSON 폴백만 따로 짧게 고칠지(범위가 달라서 분리가 더 읽기 쉬움)너의 추천
수락하고 고치세요.
src/server/chat-completions.tsJSON→SSE 폴백에서 이미 만든 completion의message.tool_calls·reasoning_content·choices[0].finish_reason을 indexed delta로 직렬화하고, 하드코딩"stop"을 제거하세요. usage와 마지막[DONE]·기존 stream-budget/lifecycle은 유지한 채, tool-only·다중 tool call·reasoning+text·length truncation 핸들러 테스트를 추가하면 release-244에도 안전하게 들어갈 수 있습니다.#3767과는 별 이슈로 두고, 이 이슈만으로 PR을 받는 걸 추천합니다.이 댓글은 grok-bot이 작성했습니다
Ingwannu commentedon Sep 6, 2026
Confirmed on current
dev(bd1cda99c) with a local synthetic JSON upstream and the actualhandleChatCompletionspath. All four new handler regressions fail before the fix: one tool call, two indexed tool calls, reasoning plus answer text, and an incomplete response'slengthfinish reason. No provider account or production configuration is involved.I am preparing a focused patch to serialize the already-converted completion's supported message fields into Chat deltas and retain its finish reason, usage, and one
[DONE]. The native Chat route and actual SSE-upstream translator will remain unchanged. This is separate from #3767's earlier refusal-conversion loss.Ingwannu commentedon Sep 6, 2026
Implementation is in Draft PR #3779 (head 7362d77). The four previously failing handler regressions now pass; the expanded focused run is 24 pass / 0 fail, including ordinary text, empty output, cancellation and layout guards. TypeScript, privacy scan, and the 425-page docs build also pass. Full-suite and platform CI validation remain pending, so this is not merged or released and the issue stays open. The change is limited to the JSON-to-SSE fallback; #3767 remains separate.
lidge-jun commentedon Sep 6, 2026
Landed via #3803 at
ac4a7659fd833ad89c525497abd7847640c30b53(Chat JSON-to-SSE completion semantics ondev).lidge-jun commentedon Sep 6, 2026
Resolved on dev by #3803, with the dependent refusal coverage in #3805. The actual Chat handler now retains indexed tools, reasoning, usage and finish semantics in buffered JSON-to-SSE fallback, with bounded admission and typed failure handling. Final combined cross-platform verification: https://github.com/lidge-jun/opencodex/actions/runs/34058496989 .