diff --git a/docs/blog-drafts/README.adoc b/docs/blog-drafts/README.adoc new file mode 100644 index 0000000000000..efd6e139e471c --- /dev/null +++ b/docs/blog-drafts/README.adoc @@ -0,0 +1,28 @@ += Blog drafts: GenAI Observability + +Prototype articles for the https://camel.apache.org/community/articles/[Camel community articles] page. + +== Two-part series + +[cols="1,3,2",options="header"] +|=== +| Part | File | Companion example + +| Blog 1 +| xref:genai-observability-01-jbang-cli-tui.adoc[JBang / CLI / TUI] +| https://github.com/apache/camel-jbang-examples/tree/main/ai/genai-observability[JBang example] + +| Blog 2 +| xref:genai-observability-02-spring-boot-obs-stack.adoc[Spring Boot + obs stack] +| https://github.com/apache/camel-spring-boot-examples/tree/main/genai-observability[Spring Boot example] +|=== + +== Three-phase narrative + +* *Phase 1 Connect* — LLMs in routes +* *Phase 2 Act* — tools, agents, MCP +* *Phase 3 Operate* — GenAI observability (this series) + +Developed with guidance from Claus Ibsen and with the help of AI agents. + +Open PR: https://github.com/apache/camel/pull/25890 diff --git a/docs/blog-drafts/genai-observability-01-jbang-cli-tui.adoc b/docs/blog-drafts/genai-observability-01-jbang-cli-tui.adoc new file mode 100644 index 0000000000000..ce709811ae990 --- /dev/null +++ b/docs/blog-drafts/genai-observability-01-jbang-cli-tui.adoc @@ -0,0 +1,466 @@ += Observe Your Camel AI Routes with GenAI OpenTelemetry (Blog 1) +:author: Omar Atie +:doctitle: Observe Your Camel AI Routes with GenAI OpenTelemetry +:doctype: article +:toc: +:toclevels: 3 +:icons: font + +== Summary + +Apache Camel 4.23 introduces *GenAI observability*: OpenTelemetry spans and Micrometer metrics +for LLM producers, following the https://opentelemetry.io/docs/specs/semconv/gen-ai/[OpenTelemetry GenAI semantic conventions]. + +This post is **Blog 1** in a two-part series. We start with the fastest path to visible AI telemetry: +Camel JBang, a LangChain4j chat route, Ollama, and the Camel CLI/TUI — no Spring Boot required. + +Special thanks to *Claus Ibsen* for guidance on shaping this material. +This article and its example were developed with the help of AI agents. + +== The three phases of Camel Gen AI + +Before diving in, it helps to see where observability fits: + +[cols="1,2,4",options="header"] +|=== +| Phase | Name | What you do + +| 1 +| *Connect* +| Wire LLMs into routes (`langchain4j-chat`, `openai`, Spring AI) + +| 2 +| *Act* +| Give AI tools via `ai-tool:`, agents, MCP server, A2A + +| 3 +| *Operate* +| Run in production with guardrails, RAG, and *GenAI observability* (this series) +|=== + +*Blog 1* covers Phase 1 + the first observability prototype. +*Blog 2* moves to Spring Boot and a full observability stack (Prometheus, VictoriaTraces, Perses). + +== Who is this for? + +* Integration developers trying Camel AI for the first time +* Platform engineers who need token/latency visibility before production +* Teams evaluating Camel vs. standalone agent frameworks +* Anyone who wants to use `camel ask` and the TUI against live LLM routes + +== What you'll build + +A timer-driven route that calls Ollama every 15 seconds. For each LLM invocation you get: + +* Micrometer timer: `gen_ai.client.operation` +* Micrometer counter: `gen_ai.client.token.usage` (tags `input` / `output`) +* OpenTelemetry child spans with `gen_ai.request.model`, token counts, finish reason +* Exchange headers: `CamelLangChain4jChatRequestModel`, `CamelLangChain4jChatResponseModel` +* TUI *Spans* tab visualization + AI Usage view (*Ctrl+U*) + +== Architecture + +[source,text] +---- +timer:genai (every 15s) + └─ langchain4j-chat (Ollama ChatModel) + ├─ camel-ai-observability-api → start GenAI span + record metrics + ├─ camel-opentelemetry2 → export span attributes + └─ camel-micrometer → gen_ai.client.* metrics + +camel run --observe + └─ camel-observability-services + ├─ /observe/health + ├─ /observe/metrics (Prometheus format) + └─ TUI Spans collector (embedded OTLP for dev) + +camel tui + ├─ Spans tab (shortcut: o) + └─ AI panel + Ctrl+U (CLI ask + route GenAI usage combined) +---- + +== Prerequisites + +* Camel JBang **4.23+** (`camel version`) +* https://ollama.com/[Ollama] installed and running +* Optional: OpenAI API key if you prefer `openai:` instead of Ollama (see <>) + +[source,shell] +---- +# Install or upgrade Camel CLI +curl -sL https://www.apache.org/dyn/closer.lua/camel/distribution/ | sh + +# Local LLM +ollama pull llama3.2 +ollama serve +---- + +== Step 1 — Get the example + +The example lives in the https://github.com/apache/camel-jbang-examples[camel-jbang-examples] repository: + +https://github.com/apache/camel-jbang-examples/tree/main/ai/genai-observability[`ai/genai-observability`] + +Files: + +* `GenAiObservabilityRoute.java` — route + Ollama `ChatModel` bean +* `application.properties` — Ollama URL, model name, observability toggle +* `README.md` — run instructions + +Clone or run directly from GitHub: + +[source,shell] +---- +git clone https://github.com/apache/camel-jbang-examples.git +cd camel-jbang-examples/ai/genai-observability +---- + +== Step 2 — Run with observability enabled + +[source,shell] +---- +cd ai/genai-observability + +camel run GenAiObservabilityRoute.java application.properties \ + --observe \ + --dependency=camel-langchain4j-chat \ + --dependency=camel-ai-observability \ + --dependency=langchain4j-ollama +---- + +What `--observe` does: + +* Adds `camel-observability-services` to the classpath +* Enables health checks, Micrometer metrics, and OpenTelemetry tracing +* Exposes endpoints on the management port (default `9876`) +* Powers the TUI *Spans* tab for live trace inspection + +[TIP] +==== +You can also pass dependencies explicitly without `--observe` and configure observability manually +(see <>), but `--observe` is the recommended developer experience. +==== + +Wait for log lines like: + +[source,text] +---- +LLM reply: Apache Camel is an open source integration framework... +Request model: llama3.2 +Response model: llama3.2 +---- + +== Step 3 — Inspect Prometheus metrics + +[source,shell] +---- +curl -s http://127.0.0.1:9876/observe/metrics | grep gen_ai +---- + +Example output (abbreviated): + +[source,text] +---- +# HELP gen_ai_client_operation GenAI client operation duration +# TYPE gen_ai_client_operation summary +gen_ai_client_operation_count{gen_ai_operation_name="chat",gen_ai_system="langchain4j",...} 3.0 + +# HELP gen_ai_client_token_usage GenAI token usage +# TYPE gen_ai_client_token_usage counter +gen_ai_client_token_usage_total{gen_ai_token_type="input",...} 42.0 +gen_ai_client_token_usage_total{gen_ai_token_type="output",...} 18.0 +---- + +Useful PromQL-style filters when you later connect Prometheus (Blog 2): + +* `gen_ai_client_token_usage_total` +* `rate(gen_ai_client_operation_count[5m])` + +== Step 4 — Explore OpenTelemetry spans in the TUI + +Open a second terminal: + +[source,shell] +---- +camel tui +---- + +. Select the running integration (`genai-observability` or similar) +. Press *o* or navigate to *More → Spans* +. Trigger another timer tick (wait ~15s) and watch a new span appear + +Each LLM call produces a child span. Key attributes (from `GenAiAttributes`): + +[cols="2,3",options="header"] +|=== +| Attribute | Meaning + +| `gen_ai.operation.name` +| e.g. `chat`, `embeddings` + +| `gen_ai.system` +| Provider abstraction (e.g. `langchain4j`) + +| `gen_ai.request.model` +| Model requested + +| `gen_ai.response.model` +| Model that served the response + +| `gen_ai.usage.input_tokens` +| Prompt tokens + +| `gen_ai.usage.output_tokens` +| Completion tokens + +| `gen_ai.response.finish_reasons` +| e.g. `stop` + +| `camel.component` +| e.g. `langchain4j-chat` +|=== + +== Step 5 — AI Usage view (Phase 2 TUI integration) + +With the AI panel open in the TUI: + +. Press *Ctrl+U* to toggle the *AI Usage* view +. You see token consumption from: +** *TUI ask* — questions you type in the embedded AI prompt +** *Route LLM calls* — extracted from exported `gen_ai.*` spans + +This is especially useful when debugging: you can ask the AI about your routes *and* see how +much the routes themselves consume — in one screen. + +== Step 6 — Ask Camel about your integration + +The CLI `camel ask` command inspects live integrations (routes, health, traces): + +[source,shell] +---- +# General +camel ask "List all routes and their IDs" + +# GenAI-focused +camel ask "Which routes call an LLM? What model headers do they set?" + +# Troubleshooting +camel ask "Why is route genai-chat slow?" +---- + +`camel ask` auto-detects providers (OpenAI, Azure, Gemini, Ollama) when configured. + +== Full example route + +[source,java] +---- +import dev.langchain4j.model.chat.ChatModel; +import dev.langchain4j.model.ollama.OllamaChatModel; +import org.apache.camel.builder.RouteBuilder; + +import static java.time.Duration.ofSeconds; + +public class GenAiObservabilityRoute extends RouteBuilder { + + @Override + public void configure() { + ChatModel chatModel = OllamaChatModel.builder() + .baseUrl("{{ollama.baseUrl:http://localhost:11434}}") + .modelName("{{ollama.model:llama3.2}}") + .temperature(0.2) + .timeout(ofSeconds(120)) + .build(); + getContext().getRegistry().bind("chatModel", chatModel); + + from("timer:genai?period={{genai.period:15000}}") + .routeId("genai-chat") + .setBody(constant("In one sentence, what is Apache Camel integration?")) + .to("langchain4j-chat:demo?chatModel=#chatModel") + .log("LLM reply: ${body}") + .log("Request model: ${header.CamelLangChain4jChatRequestModel}") + .log("Response model: ${header.CamelLangChain4jChatResponseModel}"); + } +} +---- + +=== YAML equivalent + +If you prefer YAML DSL, bind the model in Java (or Spring) and use: + +[source,yaml] +---- +- route: + id: genai-chat + from: + uri: timer:genai + parameters: + period: "15000" + steps: + - setBody: + constant: "In one sentence, what is Apache Camel integration?" + - to: + uri: langchain4j-chat:demo + parameters: + chatModel: "#chatModel" + - log: + message: "LLM reply: ${body}" +---- + +[[openai-variant]] +== Example variant — OpenAI instead of Ollama + +GenAI observability also covers `openai:`. Minimal Java route: + +[source,java] +---- +from("timer:openai?period=30000") + .setBody(constant("Summarize Apache Camel in 20 words.")) + .to("openai:chat?apiKey=RAW({{openai.apiKey}})&model=gpt-4o-mini") + .log("Reply: ${body}"); +---- + +Run with: + +[source,shell] +---- +export OPENAI_API_KEY=sk-... +camel run openai-route.yaml --observe \ + --dependency=camel-openai \ + --dependency=camel-ai-observability +---- + +Spans will include the same `gen_ai.*` attributes; the `gen_ai.system` value reflects the OpenAI client. + +== Configuration reference + +=== application.properties (example) + +[source,properties] +---- +ollama.baseUrl=http://localhost:11434 +ollama.model=llama3.2 +genai.period=15000 + +# GenAI observability — enabled by default when backends present +camel.ai.observability.enabled=true +---- + +[[manual-config]] +=== Manual observability (without --observe) + +[source,properties] +---- +camel.management.enabled=true +camel.metrics.enabled=true +camel.metrics.path=/observe/metrics +camel.opentelemetry2.enabled=true +camel.ai.observability.enabled=true +---- + +Add `--dependency=camel:observability-services` when running with `camel run`. + +=== Disable GenAI observability globally + +[source,properties] +---- +camel.ai.observability.enabled=false +---- + +Useful for A/B testing overhead or when only Camel route metrics are needed. + +== What's instrumented today (Phase 1) + +|=== +| Component | Operations observed + +| `langchain4j-chat` +| Chat completions + +| `langchain4j-tools` +| Tool-augmented LLM calls + +| `langchain4j-agent` +| AI Service agent loops + +| `langchain4j-embeddings` +| Embedding generation + +| `openai` +| Chat, embeddings, Responses API, etc. +|=== + +*Not yet covered:* `spring-ai-chat` (use Camel metrics + Spring AI actuator separately for now). + +== Extend the example — Phase 2 preview + +Once observability works, add tools without losing visibility: + +[source,yaml] +---- +# Define a tool route +- route: + id: lookup-order + from: + uri: "ai-tool:lookupOrder?description=Look up an order by ID" + steps: + - log: "Tool called with ${body}" + - setBody: + simple: "Order ${body} status: SHIPPED" + +# Agent route calling the tool (observability spans nest under agent + tool calls) +- route: + from: + uri: direct:agent + steps: + - setBody: + constant: "What is the status of order 12345?" + - to: + uri: langchain4j-agent:assistant + parameters: + agent: "#myAgent" +---- + +Each nested LLM and tool invocation produces its own span — you can trace the full agent loop in the TUI. + +== Troubleshooting + +[cols="2,4",options="header"] +|=== +| Symptom | Fix + +| No `gen_ai` metrics +| Confirm `--observe` or manual observability config; check `camel-ai-observability` on classpath + +| Empty Spans tab +| Wait for at least one LLM call; verify `camel.opentelemetry2.enabled=true` + +| Ollama connection refused +| Run `ollama serve`; check `ollama.baseUrl` + +| Metrics port unreachable +| Default management port is `9876`; look for `Management service available` in logs + +| Token headers missing +| LangChain4j agent/embeddings omit response model when provider doesn't expose it — chat producer sets both +|=== + +== Business value (why this matters) + +* *Cost control* — count tokens per route before scaling LLM usage +* *Latency SLOs* — `gen_ai.client.operation` duration per model +* *Audit trail* — tie AI decisions to Camel route IDs and exchange IDs via trace context +* *Developer speed* — `camel ask` + TUI spans reduce time-to-diagnose for AI routes +* *AI training data* — runnable examples + blog posts become high-quality context for coding assistants + +== Next up — Blog 2 + +https://github.com/apache/camel-spring-boot-examples/tree/main/genai-observability[Spring Boot genai-observability example] + +Prometheus + VictoriaTraces + Perses — the same `gen_ai.*` signals at production scale. + +== Links + +* https://github.com/apache/camel/blob/main/components/camel-ai/camel-ai-observability/src/main/docs/ai-observability.adoc[AI Observability component (4.23+, source doc)] +* https://camel.apache.org/manual/camel-jbang-tui.html[Camel TUI manual] +* https://camel.apache.org/manual/jbang-commands/camel-jbang-ask.html[camel ask command] +* https://camel.apache.org/components/next/ai-summary.html[Camel AI components] +* Example: https://github.com/apache/camel-jbang-examples/tree/main/ai/genai-observability[JBang genai-observability example] diff --git a/docs/blog-drafts/genai-observability-02-spring-boot-obs-stack.adoc b/docs/blog-drafts/genai-observability-02-spring-boot-obs-stack.adoc new file mode 100644 index 0000000000000..dcffc72451c08 --- /dev/null +++ b/docs/blog-drafts/genai-observability-02-spring-boot-obs-stack.adoc @@ -0,0 +1,507 @@ += GenAI Observability with Spring Boot and the Camel Observability Stack (Blog 2) +:author: Omar Atie +:doctitle: GenAI Observability with Spring Boot and the Camel Observability Stack +:doctype: article +:toc: +:toclevels: 3 +:icons: font + +== Summary + +In https://github.com/apache/camel/pull/25890[Blog 1] we prototyped GenAI observability with Camel JBang and the TUI. +This follow-up — *Phase 3 (Operate)* — shows the same `gen_ai.*` telemetry in a *Spring Boot* application +wired to the observability stack Camel uses in test-infra: **Prometheus**, **VictoriaTraces**, and **Perses**. + +Special thanks to *Claus Ibsen* for guidance on this series. +The runnable sample lives in https://github.com/apache/camel-spring-boot-examples/tree/main/genai-observability[`genai-observability`] +in the https://github.com/apache/camel-spring-boot-examples[camel-spring-boot-examples] repository. + +== Who is this for? + +* Spring Boot teams running Camel AI in production +* SREs who need Prometheus alerts on token burn rate +* Architects comparing dev-time TUI observability (Blog 1) vs. ops-time dashboards (this post) +* Contributors adding "how-to" examples for AI coding assistants + +== What you'll build + +[source,text] +---- +┌─────────────────┐ scrape ┌──────────────┐ +│ Spring Boot │ ──────────────► │ Prometheus │ +│ Camel + LLM │ │ :9090 │ +│ :8080 │ └──────┬───────┘ +└────────┬────────┘ │ + │ OTLP traces ▼ + ▼ ┌──────────────┐ +┌─────────────────┐ │ Perses │ +│ VictoriaTraces │ ◄── dashboards │ :8088 │ +│ :9428 │ └──────────────┘ +└─────────────────┘ + +Optional: camel tui ──► Spring Boot via cli-connector +---- + +For every LLM call in the YAML route: + +* Actuator `/actuator/prometheus` exposes `gen_ai_client_*` metrics +* OTLP export sends spans with token attributes to VictoriaTraces +* You query traces by `gen_ai.operation.name="chat"` + +== Sample project + +[source,text] +---- +genai-observability/ +├── pom.xml +├── README.adoc +├── docker-compose.yml # Prometheus + VictoriaTraces + Perses +├── prometheus.yml # scrapes localhost:8080/actuator/prometheus +└── src/main/ + ├── java/.../GenAiObservabilityApplication.java + └── resources/ + ├── application.properties + └── camel/genai-route.camel.yaml +---- + +Clone: + +[source,shell] +---- +git clone https://github.com/apache/camel-spring-boot-examples.git +cd camel-spring-boot-examples/genai-observability +---- + +== Step 1 — Prerequisites + +[source,shell] +---- +java -version # 17+ +mvn -version # 3.9+ +docker --version # for observability stack +ollama pull llama3.2 +ollama serve +---- + +== Step 2 — Start the observability stack + +From the example directory: + +[source,shell] +---- +cd genai-observability +docker compose up -d +---- + +Services (images aligned with `camel-test-infra-observability`): + +[cols="1,1,3",options="header"] +|=== +| Service | Port | Role + +| Prometheus +| 9090 +| Scrapes Spring Boot Actuator Prometheus endpoint + +| VictoriaTraces +| 9428 +| Stores OTLP traces; UI at `/select/vmui` + +| Perses +| 8088 +| Metrics dashboards (optional visualization) +|=== + +`prometheus.yml` targets `host.docker.internal:8080` — on Linux you may need +`extra_hosts: host.docker.internal:host-gateway` in compose (already common pattern). + +== Step 3 — Build and run Spring Boot + +[source,shell] +---- +mvn spring-boot:run +---- + +Wait for: + +[source,text] +---- +Started GenAiObservabilityApplication +LLM reply: Apache Camel is an integration framework... +Models: req=llama3.2 resp=llama3.2 +---- + +== Step 4 — Verify GenAI metrics via Actuator + +[source,shell] +---- +curl -s http://localhost:8080/actuator/prometheus | grep gen_ai +---- + +Filter only token counters: + +[source,shell] +---- +curl -s http://localhost:8080/actuator/prometheus \ + | grep 'gen_ai_client_token_usage' +---- + +Expected Micrometer names: + +* `gen_ai_client_operation` — timer/summary of LLM call duration +* `gen_ai_client_token_usage` — counter with tag `gen_ai_token_type=input|output` + +=== Prometheus queries + +Open http://localhost:9090 and try: + +[source,promql] +---- +# Total output tokens over time +increase(gen_ai_client_token_usage_total{gen_ai_token_type="output"}[1h]) + +# LLM call rate +rate(gen_ai_client_operation_count[5m]) + +# By Camel component +sum by (camel_component) (gen_ai_client_operation_count) +---- + +=== Example alert rule (sketch) + +[source,yaml] +---- +groups: + - name: camel-genai + rules: + - alert: HighGenAITokenBurn + expr: rate(gen_ai_client_token_usage_total[15m]) > 1000 + for: 10m + labels: + severity: warning + annotations: + summary: "High GenAI token usage on {{ $labels.job }}" +---- + +== Step 5 — Explore traces in VictoriaTraces + +Open http://localhost:9428/select/vmui + +Useful trace search filters: + +* `gen_ai.operation.name="chat"` +* `gen_ai.request.model="llama3.2"` +* `camel.component="langchain4j-chat"` + +Each span should carry: + +[cols="2,3",options="header"] +|=== +| Field | Example + +| `gen_ai.usage.input_tokens` +| `28` + +| `gen_ai.usage.output_tokens` +| `15` + +| `gen_ai.response.finish_reasons` +| `["stop"]` + +| `gen_ai.system` +| `langchain4j` +|=== + +[TIP] +==== +Correlate traces with logs by passing the trace ID from MDC if you enable log correlation +in your logging config — see https://camel.apache.org/manual/camel-jbang-tui.html#_opentelemetry_spans[OpenTelemetry docs]. +==== + +== Step 6 — Optional: Camel TUI against Spring Boot + +Uncomment in `pom.xml`: + +[source,xml] +---- + + org.apache.camel.springboot + camel-cli-connector-starter + +---- + +And in `application.properties`: + +[source,properties] +---- +camel.cli.enabled=true +---- + +Then: + +[source,shell] +---- +camel tui +---- + +Select the remote Spring Boot integration — you get the same *Spans* tab and *Ctrl+U* AI Usage +view as Blog 1, but against a production-style runtime. + +== Full configuration walkthrough + +=== Maven dependencies (key excerpts) + +[source,xml] +---- + + org.apache.camel.springboot + camel-spring-boot-starter + + + org.apache.camel.springboot + camel-observability-services-starter + + + org.apache.camel.springboot + camel-langchain4j-chat-starter + + + org.apache.camel + camel-ai-observability + + + dev.langchain4j + langchain4j-ollama-spring-boot-starter + + + org.springframework.boot + spring-boot-starter-actuator + + + io.micrometer + micrometer-registry-prometheus + +---- + +=== application.properties (explained) + +[source,properties] +---- +# --- LangChain4j / Ollama (auto-configures chatLanguageModel bean) --- +langchain4j.ollama.chat-model.base-url=http://localhost:11434 +langchain4j.ollama.chat-model.model-name=llama3.2 +langchain4j.ollama.chat-model.temperature=0.2 +langchain4j.ollama.chat-model.timeout=PT120S + +# --- Camel YAML routes --- +camel.main.routes-include-pattern=camel/* + +# --- GenAI observability --- +camel.ai.observability.enabled=true +camel.opentelemetry2.enabled=true + +# --- Spring Actuator (Prometheus scrape target) --- +management.endpoints.web.exposure.include=health,prometheus,info +management.prometheus.metrics.export.enabled=true + +# --- OTLP export to VictoriaTraces --- +camel.opentelemetry2.export-target=jaeger +otel.exporter.otlp.endpoint=http://localhost:9428/insert/opentelemetry/v1/traces +otel.exporter.otlp.protocol=http/protobuf +---- + +Property notes: + +* `camel.ai.observability.enabled` — master switch for GenAI spans/metrics (default `true`) +* `camel.opentelemetry2.export-target=jaeger` — use external OTLP collector (VictoriaTraces) +* LangChain4j starter creates `#chatLanguageModel` — referenced in YAML route + +== The Camel route (YAML) + +[source,yaml] +---- +- route: + id: genai-chat + from: + uri: timer:genai + parameters: + period: "15000" + steps: + - setBody: + constant: "In one sentence, what is Apache Camel integration?" + - to: + uri: langchain4j-chat:demo + parameters: + chatModel: "#chatLanguageModel" + - log: + message: "LLM reply: ${body}" + - log: + message: "Models: req=${header.CamelLangChain4jChatRequestModel} resp=${header.CamelLangChain4jChatResponseModel}" +---- + +=== Java RouteBuilder alternative + +[source,java] +---- +@Component +public class GenAiRoute extends RouteBuilder { + @Override + public void configure() { + from("timer:genai?period=15000") + .routeId("genai-chat") + .setBody(constant("Explain EIP in one sentence.")) + .to("langchain4j-chat:demo?chatModel=#chatLanguageModel") + .log("Reply: ${body}"); + } +} +---- + +== docker-compose.yml (excerpt) + +[source,yaml] +---- +services: + prometheus: + image: quay.io/prometheus/prometheus:v3.13.2 + ports: ["9090:9090"] + volumes: ["./prometheus.yml:/etc/prometheus/prometheus.yml:ro"] + + victoriatraces: + image: mirror.gcr.io/victoriametrics/victoria-traces:v0.10.0 + ports: ["9428:9428"] + + perses: + image: mirror.gcr.io/persesdev/perses:v0.54.0 + ports: ["8088:8080"] +---- + +These versions match `camel-test-infra-observability` so CI and blog readers use the same stack. + +== JBang vs Spring Boot — when to use which + +[cols="2,3,3",options="header"] +|=== +| Concern | Blog 1 (JBang) | Blog 2 (Spring Boot) + +| Time to first span +| Minutes (`camel run --observe`) +| Minutes + docker compose + +| Metrics endpoint +| `/observe/metrics` on port 9876 +| `/actuator/prometheus` on port 8080 + +| Trace UI (dev) +| TUI Spans tab (built-in) +| VictoriaTraces VMUI + +| Production fit +| Prototyping, CI demos +| Standard Spring ops (Actuator, K8s probes) + +| AI assistant ergonomics +| `camel ask`, MCP on TUI +| `AGENTS.md`, `/llms.txt`, cli-connector +|=== + +Most teams: **prototype in JBang**, **deploy observability pattern in Spring Boot**. + +== Production checklist + +. Set `camel.ai.observability.enabled=true` explicitly in all environments +. Scrape `/actuator/prometheus` (or `/observe/metrics` for Camel Main) +. Export OTLP to your org's collector (Jaeger, Tempo, VictoriaTraces, etc.) +. Alert on `gen_ai_client_token_usage` rate and `gen_ai_client_operation` p99 +. Use route IDs (`genai-chat`) in dashboards to attribute cost per integration +. Document LLM routes in `AGENTS.md` for AI coding assistants (generated by `camel export` since 4.21) +. Review <> if running load tests without LLM overhead + +[[disable]] +== Disabling GenAI observability + +[source,properties] +---- +camel.ai.observability.enabled=false +---- + +OpenAI streaming only sets `stream_options.include_usage=true` when observability is enabled — +disabling avoids the extra API parameter on compatible endpoints. + +== Extend — multi-route example + +Add a second route that enriches before calling the LLM: + +[source,yaml] +---- +- route: + id: enrich-and-chat + from: + uri: timer:enrich + parameters: + period: "30000" + steps: + - setBody: + constant: "orders" + - setHeader: + name: context + constant: "User asked about integration patterns" + - setBody: + simple: "${header.context}: What is the claim check EIP?" + - to: + uri: langchain4j-chat:demo + parameters: + chatModel: "#chatLanguageModel" + - log: + message: "Enriched reply: ${body}" +---- + +Both routes appear separately in Prometheus (`routeId` tags via Camel metrics) and as distinct +trace branches in VictoriaTraces. + +== Three phases recap + +* *Phase 1 Connect* — LLMs in routes → https://github.com/apache/camel/pull/25890[Blog 1] +* *Phase 2 Act* — `ai-tool:`, `langchain4j-agent`, https://camel.apache.org/manual/camel-jbang-mcp.html[MCP server] +* *Phase 3 Operate* — this post: metrics, traces, dashboards, alerts + +== Troubleshooting + +[cols="2,4",options="header"] +|=== +| Issue | Resolution + +| Prometheus empty targets +| Spring Boot must be running on 8080; check `host.docker.internal` on Linux + +| No traces in VictoriaTraces +| Verify `otel.exporter.otlp.endpoint`; check app logs for OTLP export errors + +| `chatLanguageModel` not found +| Confirm `langchain4j-ollama-spring-boot-starter` and Ollama properties + +| No `gen_ai` metrics +| Add `camel-ai-observability` + `camel-observability-services-starter` + +| Build fails on SNAPSHOT +| Use camel-spring-boot-examples tag matching your Camel release (see root README.adoc) +|=== + +== Business outcomes + +* *FinOps* — chargeback LLM usage by route/team using token counters +* *Reliability* — alert when LLM latency degrades independently of HTTP latency +* *Compliance* — trace attributes document which model answered which integration flow +* *Knowledge base* — camel-spring-boot-examples + blog = training material for human and AI readers + +== Series links + +* https://github.com/apache/camel/pull/25890[Blog 1 drafts PR] +* https://github.com/apache/camel-jbang-examples/tree/main/ai/genai-observability[JBang example] +* https://github.com/apache/camel-spring-boot-examples/tree/main/genai-observability[Spring Boot example] + +== Documentation + +* https://github.com/apache/camel/blob/main/components/camel-ai/camel-ai-observability/src/main/docs/ai-observability.adoc[AI Observability (4.23+, source doc)] +* https://camel.apache.org/components/others/observability-services.html[Observability Services] +* https://camel.apache.org/manual/langchain4j-spring-boot-integration.html[LangChain4j Spring Boot Integration] +* https://camel.apache.org/components/next/ai-summary.html[All Camel AI components]