diff --git a/CHANGELOG.md b/CHANGELOG.md index ead9960..beda77a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -29,10 +29,11 @@ unreachable model within minutes, and the pipeline no longer imports the chat. N - **The data-change event moved out of the chat.** `chat.suggest.SuggestionDataChanged` is now `event.DataChanged`; `analytics`, `application` and `archive` import nothing of the chat, and `ChatIsolationTest` checks imports and every `chat_*` table the migrations create. -- **Spring AI 2.0.2, with openai-java held at 4.65.0.** 2.0.2 brings openai-java 4.80.0, and from 4.66.0 on a stopped - turn or one past its deadline leaves the model call running (openai/openai-java#1027; 4.96.0 still does). The hold - moves from Spring AI to openai-java, so Spring AI's own fixes land: Anthropic streaming chunks carry the thinking - text, and the configured OpenAI timeout is honoured again. Renovate proposes openai-java in a pull request of its own. +- **Spring AI 2.0.2, with the openai-java it brings (4.80.0).** Its own fixes land: Anthropic streaming chunks carry + the thinking text, and the configured OpenAI timeout is honoured again. From openai-java 4.66.0 on + (openai/openai-java#1027) closing a stream from another thread no longer ends the HTTP call, so a stopped turn, a + deleted one or one past its deadline left the model generating; closing a model response now cancels its call first, + and no openai-java version is held back any more. - **The frontend image builds on Node 26** (`node:26.13.0-alpine`, from 24.22.1). The Angular CLI accepts `>=26.0.0`; the build output is the same static files, served by the same nginx stage. A root `.nvmrc` names the major, and CI and the release gate now set up that Node with `actions/setup-node` instead of building on @@ -44,8 +45,8 @@ unreachable model within minutes, and the pipeline no longer imports the chat. N `''` for a `grid-template-columns` nothing computed, and the shortlist now reads both as "no grid", which a browser can answer too. TypeScript stays on 6.0, which is what Angular 22 accepts. - **Backend: jsoup 1.24.1 and Spotless 8.11.0.** palantir-java-format 2.103.0 reformats four files and stays at - 2.98.0 for a commit of its own; Gradle 9.9.0 waits for its `gradle:9.9.0-jdk25` image. openai-java 4.101.0 still - fails the four chat-stop tests, so the hold at 4.65.0 stays. + 2.98.0 for a commit of its own. Gradle 9.9.0; the backend image keeps building on `gradle:9.8.2-jdk25` until a + `gradle:9.9.0-jdk25` image is published. ### Fixed diff --git a/backend/CLAUDE.md b/backend/CLAUDE.md index 4d42fbb..02c20e7 100644 --- a/backend/CLAUDE.md +++ b/backend/CLAUDE.md @@ -110,6 +110,8 @@ takes `ChatBudget.take()` against `chat.max_calls_per_day` instead — reasoning `docs/decisions/retrieval.md`. - **An UPDATE cannot reference its own target table from a LATERAL item in its FROM clause.** Compute the pairs in a `WITH` and update `FROM` that. Reasoning in `docs/decisions/pipeline-dedupe-filter.md`. +- **Every OpenAI-compatible chat client gets `CancelOnClose.CUSTOMIZER`, as `ChatModels` passes it, or a stopped + stream leaves the model generating.** — reasoning in `docs/decisions/chat.md` § Stop and regenerate. - **Two constructors on a `@Component` are none, and the error names the wrong thing.** Spring picks neither and reports `No default constructor found` — a message about a constructor that was never meant to exist, rather than about the ambiguity. Every context in the suite fails at once: measured at 194 failures from one added convenience diff --git a/backend/build.gradle.kts b/backend/build.gradle.kts index f895b11..4d00c45 100644 --- a/backend/build.gradle.kts +++ b/backend/build.gradle.kts @@ -123,16 +123,6 @@ dependencies { // The BOM governs the two starters' versions and everything they drag in, so the // Anthropic and OpenAI client SDKs are pinned by one line rather than by three. implementation(platform(libs.spring.ai.bom)) - // With one exception: openai-java is held below what the BOM brings, because from 4.66.0 on a - // stop no longer ends the model call. `strictly`, so a newer one dragged in by anything else - // fails the resolution instead of quietly winning. The reason and the way out sit next to - // `openaiJava` in libs.versions.toml. - constraints { - implementation(libs.openai.java.core) { - version { strictly(libs.versions.openaiJava.get()) } - because("openai-java 4.66.0+ leaves a cancelled stream's connection open (openai/openai-java#1027)") - } - } // Naming two vendors here is deliberate and is the one place the "nothing is wired in" // rule does not reach: a dependency is not a value. No host, key or model name appears // in any committed file, and `base_url` still decides who answers. diff --git a/backend/src/main/java/de/codeministry/leadgen/llm/CancelOnClose.java b/backend/src/main/java/de/codeministry/leadgen/llm/CancelOnClose.java new file mode 100644 index 0000000..57fa7b7 --- /dev/null +++ b/backend/src/main/java/de/codeministry/leadgen/llm/CancelOnClose.java @@ -0,0 +1,85 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * + * Copyright 2026 Marcello Muscara (codeministry) + * + * Licensed under the Apache License, Version 2.0. You may obtain a copy of the + * License at http://www.apache.org/licenses/LICENSE-2.0 + */ +package de.codeministry.leadgen.llm; + +import java.io.IOException; +import okhttp3.Call; +import okhttp3.Interceptor; +import okhttp3.MediaType; +import okhttp3.Response; +import okhttp3.ResponseBody; +import okio.BufferedSource; +import org.springframework.ai.openai.http.okhttp.OpenAiHttpClientBuilderCustomizer; + +/** + * Makes closing a model response cancel its HTTP call before the body is closed, so a stopped + * stream ends the request on the wire. + * + *

Why. A chat turn is stopped, deleted or past its deadline by closing its stream from a + * thread other than the one blocked reading it. From openai-java 4.66.0 on (openai/openai-java#1027) + * that close shuts the OkHttp response before the reader, under the read still in flight: okio + * throws "Unbalanced enter/exit", the socket stays open, and the model keeps generating an answer + * nobody reads, paid for in full. {@link Call#cancel()} is OkHttp's own thread-safe way to end a + * call from outside: it closes the socket, the blocked read fails, and the closes after it find + * nothing left to race. + * + *

Cancelling a call whose body was read to its end changes nothing, so the calls that are not + * streamed, the judge's and the classifier's, close as before. + */ +final class CancelOnClose implements Interceptor { + + /** Hands the interceptor to Spring AI's OkHttp client, for the sync and the async one alike. */ + static final OpenAiHttpClientBuilderCustomizer CUSTOMIZER = builder -> builder.interceptor(new CancelOnClose()); + + @Override + public Response intercept(Chain chain) throws IOException { + Response response = chain.proceed(chain.request()); + ResponseBody body = response.body(); + if (body == null) { + return response; + } + return response.newBuilder().body(new Cancelling(body, chain.call())).build(); + } + + /** The response body as it was, except that closing it cancels the call first. */ + private static final class Cancelling extends ResponseBody { + + private final ResponseBody delegate; + private final Call call; + + Cancelling(ResponseBody delegate, Call call) { + this.delegate = delegate; + this.call = call; + } + + @Override + public MediaType contentType() { + return delegate.contentType(); + } + + @Override + public long contentLength() { + return delegate.contentLength(); + } + + @Override + public BufferedSource source() { + return delegate.source(); + } + + @Override + public void close() { + try { + call.cancel(); + } finally { + delegate.close(); + } + } + } +} diff --git a/backend/src/main/java/de/codeministry/leadgen/llm/ChatModels.java b/backend/src/main/java/de/codeministry/leadgen/llm/ChatModels.java index cfab90d..b758ec9 100644 --- a/backend/src/main/java/de/codeministry/leadgen/llm/ChatModels.java +++ b/backend/src/main/java/de/codeministry/leadgen/llm/ChatModels.java @@ -153,7 +153,7 @@ private ChatModel openAi(String baseUrl, String apiKey, String model, Duration t null, ObservationRegistry.NOOP, null, - List.of())) + List.of(CancelOnClose.CUSTOMIZER))) // Both clients, and the asynchronous one is not optional: left unset, the // builder makes its own from its own empty fields and fails with "at least // one credential source must be specified" — a credential error naming a key @@ -174,7 +174,7 @@ private ChatModel openAi(String baseUrl, String apiKey, String model, Duration t null, ObservationRegistry.NOOP, null, - List.of())) + List.of(CancelOnClose.CUSTOMIZER))) // Same trap as in `EmbeddingModels`: without this, `AbstractOpenAiOptions` // fills in its 60 s DEFAULT_TIMEOUT and the per-call value overrides the // client's. The Anthropic options below carry no such default. diff --git a/docs/decisions/chat.md b/docs/decisions/chat.md index acdc0f4..3a5773e 100644 --- a/docs/decisions/chat.md +++ b/docs/decisions/chat.md @@ -429,6 +429,21 @@ leaves nowhere to put them. The model is asked with tool execution left to the c the turn reads between chunks and fires a signal that cancels the model request waiting for its next one, so a local model stops generating rather than finishing into a closed pipe (`ChatStreamTest`: the stub's connection closed within a second, its last chunk never written). + +**Closing the stream cancels its HTTP call first (`llm/CancelOnClose`).** The stop, a delete and the +deadline all close the model's stream from a thread other than the one blocked reading it. openai-java +4.65.0 closed the reader before the response and got away with it; from 4.66.0 on +(openai/openai-java#1027) it closes the OkHttp response first, under the read still in flight, okio throws +"Unbalanced enter/exit", and the socket stays open while the model writes an answer nobody reads. Measured: +the four stop tests in `ChatStreamTest`, `ChatConversationTest` and `ChatTurnDeadlineTest` red on +4.66.1, 4.80.0, 4.92.0, 4.96.0 and 4.101.0. For a while the fix was holding openai-java at 4.65.0 below +what Spring AI 2.0.2 brings; that pinned a client Spring AI was never built against and needed a release +upstream to end. The interceptor ends it here instead: Spring AI's OkHttp client takes one through +`OpenAiSetup`'s customizer list, `ChatModels` hands it to the sync and the async client, and closing a +response body calls `Call.cancel()` before the close — OkHttp's own thread-safe way to end a call from +outside. A body read to its end has nothing left to cancel, so the judge and the classifier, which call +rather than stream, are untouched. Green on 4.80.0 and 4.101.0 alike. + The partial answer is kept, its citations listed, and the turn stored `STOPPED`; the composer takes the next question. diff --git a/gradle/libs.versions.toml b/gradle/libs.versions.toml index aabbe34..ffe5cda 100644 --- a/gradle/libs.versions.toml +++ b/gradle/libs.versions.toml @@ -23,17 +23,6 @@ flexmark = "0.64.8" # third model module would buy nothing. Pinned rather than tracked for the same reason jsoup # and freemarker are — a judge that suddenly answers differently should show up as a diff. springAi = "2.0.2" -# openai-java, held below the version Spring AI brings (2.0.2 brings 4.80.0) and enforced as a -# `strictly` constraint in backend/build.gradle.kts. Since 4.66.0 (openai/openai-java#1027) closing a -# stream from another thread closes the OkHttp body under the in-flight read, which throws -# "Unbalanced enter/exit" and leaves the connection open, so stopping a chat turn or hitting its -# deadline no longer ends the model call. 4.65.0 is the last release before that; 4.66.1, 4.92.0, 4.96.0 and -# 4.101.0 fail the same four tests in ChatStreamTest, ChatConversationTest and ChatTurnDeadlineTest. -# Held here rather than holding Spring AI, so its other fixes land. Every openai-java method that -# spring-ai-openai 2.0.2 calls exists in 4.65.0 (checked against the bytecode, 499 references). -# Renovate proposes it in a pull request of its own; when that one turns green, the hold can go — -# drop the constraint and this entry together, and the next Spring AI picks its own version again. -openaiJava = "4.65.0" # The GraalVM build tools. Applied for BOTH images and not only the native one: applying # it is what makes the Boot plugin register `processAot`, and Spring AOT is half of the JVM # image's startup win. It also carries the version of the reachability-metadata repository @@ -61,7 +50,6 @@ spring-ai-chat = { module = "org.springframework.ai:spring-ai-client-chat" } spring-ai-anthropic = { module = "org.springframework.ai:spring-ai-anthropic" } spring-ai-openai = { module = "org.springframework.ai:spring-ai-openai" } spring-ai-mcp-server = { module = "org.springframework.ai:spring-ai-starter-mcp-server-webmvc" } -openai-java-core = { module = "com.openai:openai-java-core", version.ref = "openaiJava" } jsoup = { module = "org.jsoup:jsoup", version.ref = "jsoup" } lombok = { module = "org.projectlombok:lombok", version.ref = "lombok" } greenmail-junit5 = { module = "com.icegreen:greenmail-junit5", version.ref = "greenmail" } diff --git a/renovate.json b/renovate.json index 6a49dae..b0dd04a 100644 --- a/renovate.json +++ b/renovate.json @@ -36,17 +36,11 @@ "groupName": "spring boot" }, { - "description": "Spring AI gets its own pull request. Its BOM decides the model SDKs, and openai-java is held below the one it brings, so a Spring AI release is the update most likely to need that hold revisited; on its own, its CI result says so without turning the backend group red.", + "description": "Spring AI gets its own pull request. Its BOM decides the model SDKs, so a Spring AI release can change the client that streams the chat; the stop tests then say so on its own CI run without turning the backend group red.", "matchPackageNames": ["/^org\\.springframework\\.ai/"], "groupName": null, "automerge": false }, - { - "description": "openai-java is held at the last release before the stream-cancel regression (reason next to openaiJava in libs.versions.toml). Its own pull request, so a red one stays out of the backend group and a green one says the hold can go.", - "matchPackageNames": ["com.openai:openai-java-core"], - "groupName": null, - "automerge": false - }, { "description": "native-build-tools carries the GraalVM reachability-metadata repository version with it, so a bump changes which third-party metadata the native image is compiled against. Its own pull request, read before merging.", "matchPackageNames": ["/^org\\.graalvm\\.buildtools/"],