From 43def1a812ad934f72b7c2aacecd7cc1cc69bd20 Mon Sep 17 00:00:00 2001 From: Claude Date: Mon, 21 Sep 2026 13:15:12 +0000 Subject: [PATCH] llama-atmosphere-agent: assert the plain-chat integration test on behaviour, not wording The first model-backed CI run (35600558852) proved the loop -- the tool-call and read->write->read tests passed against Qwen2.5-1.5B -- while plainChatStreamsAnAnswer failed because the model answered "OK" to "reply with exactly ATMOSPHERE_OK". That assertion pinned prose a 1.5B model does not produce deterministically, which the class javadoc already said not to do. The test now asks a question with one unambiguous fact (2 + 2), asserts a non-empty streamed answer containing "4" that arrived in several chunks, and documents why. TODO.md drops the "read the first run" item that run resolved. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01UVwj2UuMPybiK1bHyG9toH --- TODO.md | 7 ++----- .../AtmosphereToolLoopIntegrationTest.java | 13 ++++++++++--- 2 files changed, 12 insertions(+), 8 deletions(-) diff --git a/TODO.md b/TODO.md index 17878234..5adbef0f 100644 --- a/TODO.md +++ b/TODO.md @@ -19,12 +19,9 @@ so everything below is genuinely still open. ### Atmosphere coding agent (`llama-atmosphere-agent/`) — follow-ups -The headless loop is verified (see CLAUDE.md "Local coding agent with Atmosphere"). Still open: +The headless loop is verified, including the model-backed CI job (run 35600558852: tool call +answered, read→write→read loop changed the file). Still open: -- **First model-backed CI run.** `test-java-llama-atmosphere-agent-integration` was added without a - run on GitHub's runners; the three assertions are about the loop (tool invoked, result answered, - file changed), but a 1.5B model on a CPU runner may still need a prompt or budget tweak. Read its - first run before trusting it as a signal. - **Tool rounds are not carried across REPL turns** — only `user`/`assistant` text is replayed, so a second question cannot refer to a tool result of the first. Keep the full Atmosphere `ChatMessage` list (incl. `tool_calls`/`tool` messages) per turn instead. diff --git a/llama-atmosphere-agent/src/test/java/net/ladenthin/llama/atmosphere/AtmosphereToolLoopIntegrationTest.java b/llama-atmosphere-agent/src/test/java/net/ladenthin/llama/atmosphere/AtmosphereToolLoopIntegrationTest.java index 1bfe8b26..198f3bcd 100644 --- a/llama-atmosphere-agent/src/test/java/net/ladenthin/llama/atmosphere/AtmosphereToolLoopIntegrationTest.java +++ b/llama-atmosphere-agent/src/test/java/net/ladenthin/llama/atmosphere/AtmosphereToolLoopIntegrationTest.java @@ -116,16 +116,23 @@ private ConsoleSession session() { return new ConsoleSession(new PrintStream(new ByteArrayOutputStream(), true, StandardCharsets.UTF_8), fs); } + /** + * Plain chat: a non-empty answer that arrives as several streamed chunks. The first CI run (run + * 35600558852) showed why this must not pin wording: asked to "reply with exactly ATMOSPHERE_OK", the + * 1.5B model streamed {@code OK} — a perfectly working loop failing a prose assertion. What the wire + * contract guarantees is that the question reaches the model and its answer streams back, so that + * is what is asserted; the one content check is a fact no instruct model gets wrong. + */ @Test void plainChatStreamsAnAnswer() throws Exception { ConsoleSession session = session(); - runner(List.of()).run("Reply with exactly this word and nothing else: ATMOSPHERE_OK", List.of(), session); + runner(List.of()).run("What is 2 + 2? Answer with one short sentence.", List.of(), session); assertThat(session.await(TURN_TIMEOUT), is(true)); assertThat(session.failure(), is(nullValue())); - assertThat( - "streamed text: " + session.text(), session.text().toUpperCase().contains("ATMOSPHERE_OK"), is(true)); + assertThat("streamed text: " + session.text(), session.text().trim().isEmpty(), is(false)); + assertThat("the answer must contain the number 4: " + session.text(), session.text(), containsString("4")); assertThat( "the answer must arrive as several SSE chunks, not one blob", session.chunks().size(),