From 0dd5731661490d37546cc2c779497f95bd545c9d Mon Sep 17 00:00:00 2001 From: haoshan98 Date: Wed, 30 Sep 2026 09:04:53 +0000 Subject: [PATCH] chore: roadmap, readme and general docs updates Signed-off-by: haoshan98 --- AGENTS.md | 2 ++ ARCHITECTURE.md | 2 +- CHANGELOG.md | 73 +++++++++++++++++++++++++++++++------------- README.md | 78 ++++++++++++++++++++++++++++++++++------------- ROADMAP.md | 20 ++++++++++-- docs/api/index.md | 71 ++++++++++++++++++++++++++++++++++++------ docs/index.md | 19 +++++++----- mkdocs.yaml | 1 + 8 files changed, 203 insertions(+), 63 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 5f30b65b..f93fd7b0 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -27,6 +27,8 @@ This repository is Rust-first under the `vllm-project` GitHub organization. ├── TERMINOLOGY.md # Normative project vocabulary ├── crates/agentic-server/ # Axum binary, transport handlers, and configuration ├── crates/agentic-server-core/ # Protocol types, execution, tools, and persistence +├── crates/agentic-llm-d/ # Split-execution state backend for the llm-d coordinator +├── crates/agentic-cli-docs/ # Build-time generator for the CLI reference docs ├── crates/agentic-praxis/ # Praxis integration ├── python/agentic_api/ # Python distribution, diagnostics, and launcher ├── tests/python/ # Python package and CLI tests diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 20460346..b2fe201a 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -574,7 +574,7 @@ call inference, run the tool loop, persist. `agentic-server` never reaches past `multi_agent.enabled` selects gateway orchestration; it is not forwarded to the upstream model. Enabled requests require `store: true`; the current validator also -rejects `reasoning.summary` and `reasoning.generate_summary`. These are gateway +rejects `max_tool_calls`, `reasoning.summary`, and `reasoning.generate_summary`. These are gateway validation rules, not a claim that every upstream provider has the same restrictions. New requests without multi-agent configuration use single-agent execution; continuations of a stored agent tree inherit its configuration. diff --git a/CHANGELOG.md b/CHANGELOG.md index 0da1f970..bca9aa66 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,57 @@ All notable changes to Agentic API are documented here. ## [Unreleased] +### Added + +- Added HTTP multi-agent Responses execution (#372, #373, part of #298). A stored request with + `multi_agent.enabled: true` spawns and coordinates subagents with isolated histories, returns attributed + `multi_agent_call`, `multi_agent_call_output`, and `agent_message` items over JSON or SSE, and resumes the agent + tree through `previous_response_id`. `max_concurrent_subagents` defaults to 3. Multi-agent requires `store: true` + and is rejected on WebSocket sessions and with `max_tool_calls` or reasoning summaries. +- Added an opt-in `code_interpreter` gateway tool backed by Eryx (#368, closes #308). It requires the + `embedded-code-interpreter` Cargo feature, `[code_interpreter] enabled = true` or + `AGENTIC_CODE_INTERPRETER_ENABLED=true`, and Linux cgroup v2 delegation. See + [the embedded code interpreter design](docs/design/embedded-code-interpreter.md). +- Added the Conversations API item and resource endpoints: retrieve, update, and delete conversations, and create, + list, retrieve, and delete conversation items with cursor pagination (#351, #370, closes #155). +- Added `GET /v1/responses/{response_id}` retrieval of stored response snapshots without calling the upstream + model (#354). +- Forwarded `/v1/chat/completions` and `/v1/completions` to the upstream unchanged (#369). +- Added execution-level OpenTelemetry traces for Responses and Messages, including WebSocket session spans and + upstream W3C context propagation (#349, phase 2 of #279). +- Preserved `prompt_cache_key` across direct and executor-backed Responses paths (#347). +- Added Tavily as a selectable backend for the gateway-owned `web_search` tool (#327, Phase 3 of #291). Select it + with `AGENTIC_WEB_SEARCH_PROVIDER=tavily` or `[web_search] provider = "tavily"` and supply `TAVILY_API_KEY`; the + endpoint defaults to `https://api.tavily.com` and can be overridden with `AGENTIC_WEB_SEARCH_BASE_URL` or + `[web_search] base_url`. Each query is one `POST /search` with a JSON body and a bearer token; the key is never + placed in the body. `allowed_domains` / `blocked_domains` and the model's `include_domains` / `exclude_domains` are + forwarded to Tavily's native `include_domains` / `exclude_domains` and re-checked client-side, `count` is clamped + to Tavily's maximum of 20, `freshness` maps to `time_range` or to `start_date` / `end_date` widened by one day on + each side because Tavily's bounds are exclusive and sent with `filter_by_published_date` so undated results cannot + bypass the requested recency window, `language` keeps Tavily's documented compound tags (`zh-cn`) and + otherwise reduces to its primary subtag, `safesearch` maps to the boolean `safe_search`, and `country` plus the + You.com-specific + `livecrawl`, `livecrawl_formats`, `crawl_timeout`, and `boost_domains` arguments are ignored. Results fill + `results.web` with `published_date` as `page_age`; `results.news` stays empty because a second news search per + query would double credit usage. Rejected credentials, HTTP 429, and Tavily's 432/433 plan-limit statuses fail the + `web_search_call` without an automatic retry, naming the key variable or the upstream `Retry-After` value and never + echoing the secret. Each Tavily `metadata[]` entry carries `"provider": "tavily"`, Tavily's `request_id` as + `search_uuid`, and its `response_time` as `latency`. Tavily inherits the gateway concurrency limit. + +### Changed + +- `WebSearchProviderKind` gains a `Tavily` variant; `WebSearchProviderKind::ALL` is now a `&'static [Self]` slice + listing all three providers, so adding a provider no longer changes its type; and + `WebSearchHandler::from_config` builds the Tavily provider for it (#327). The shared `null_as_default` and + `read_response_limited` helpers moved from `tool/web_search/mod.rs` to `tool/web_search/provider.rs`; both were and + remain crate-private, so no public API changed. + +### Fixed + +- A `store: false` request that continues a stored response is no longer persisted (#387). +- Rejected incomplete or malformed upstream Messages streams before gateway tool dispatch (#345). +- Disabled Nagle's algorithm on accepted gateway connections to remove ~40 ms streaming stalls (#380). + ## [0.8.0] - 2026-09-19 ### Added @@ -40,23 +91,6 @@ All notable changes to Agentic API are documented here. `crawl_timeout`, and `boost_domains` arguments are ignored. Rejected credentials and HTTP 429 responses fail the `web_search_call` without an automatic retry, naming the key variable or the upstream `Retry-After` value and never echoing the secret. Each Brave `metadata[]` entry carries `"provider": "brave"`. -- Added Tavily as a selectable backend for the gateway-owned `web_search` tool (#327, Phase 3 of #291). Select it - with `AGENTIC_WEB_SEARCH_PROVIDER=tavily` or `[web_search] provider = "tavily"` and supply `TAVILY_API_KEY`; the - endpoint defaults to `https://api.tavily.com` and can be overridden with `AGENTIC_WEB_SEARCH_BASE_URL` or - `[web_search] base_url`. Each query is one `POST /search` with a JSON body and a bearer token; the key is never - placed in the body. `allowed_domains` / `blocked_domains` and the model's `include_domains` / `exclude_domains` are - forwarded to Tavily's native `include_domains` / `exclude_domains` and re-checked client-side, `count` is clamped - to Tavily's maximum of 20, `freshness` maps to `time_range` or to `start_date` / `end_date` widened by one day on - each side because Tavily's bounds are exclusive and sent with `filter_by_published_date` so undated results cannot - bypass the requested recency window, `language` keeps Tavily's documented compound tags (`zh-cn`) and - otherwise reduces to its primary subtag, `safesearch` maps to the boolean `safe_search`, and `country` plus the - You.com-specific - `livecrawl`, `livecrawl_formats`, `crawl_timeout`, and `boost_domains` arguments are ignored. Results fill - `results.web` with `published_date` as `page_age`; `results.news` stays empty because a second news search per - query would double credit usage. Rejected credentials, HTTP 429, and Tavily's 432/433 plan-limit statuses fail the - `web_search_call` without an automatic retry, naming the key variable or the upstream `Retry-After` value and never - echoing the secret. Each Tavily `metadata[]` entry carries `"provider": "tavily"`, Tavily's `request_id` as - `search_uuid`, and its `response_time` as `latency`. Tavily inherits the gateway concurrency limit. - Added `[web_search] max_concurrent_queries` and `AGENTIC_WEB_SEARCH_MAX_CONCURRENT_QUERIES` to cap concurrent provider requests inside one batched search. Brave defaults to `1` for its free-plan rate limit; You.com keeps inheriting `max_concurrent_gateway_calls`. The effective ceiling is the smallest of the gateway limit, this @@ -99,11 +133,6 @@ All notable changes to Agentic API are documented here. uses it. With `provider` unset, You.com behavior, configuration, and model-facing output are unchanged; a generated `config.toml` now records `provider = "you"` and leaves `api_key_env` unset so provider switches select the matching default credential variable. -- `WebSearchProviderKind` gains a `Tavily` variant; `WebSearchProviderKind::ALL` is now a `&'static [Self]` slice - listing all three providers, so adding a provider no longer changes its type; and - `WebSearchHandler::from_config` builds the Tavily provider for it (#327). The shared `null_as_default` and - `read_response_limited` helpers moved from `tool/web_search/mod.rs` to `tool/web_search/provider.rs`; both were and - remain crate-private, so no public API changed. ### Fixed diff --git a/README.md b/README.md index d666b72a..8e54d484 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ **The stateful, agentic API layer for [vLLM](https://github.com/vllm-project/vllm), written in Rust 🦀** -*Run OpenAI-grade agentic workloads (Responses API, server-side tools, Codex) on your own GPUs.* +*Run OpenAI-grade agentic workloads (Responses and Messages APIs, server-side tools, Codex, Claude Code) on your own GPUs.* [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE) [![Rust](https://img.shields.io/badge/Rust-1.85%2B-orange.svg?logo=rust)](Cargo.toml) @@ -27,14 +27,14 @@ vLLM gives you state-of-the-art inference throughput. But real agentic applicati ```mermaid flowchart LR - C(["🧑‍💻 Client
Codex · SDKs · curl"]) -->|"📮 POST /v1/responses
🌐 HTTP  📡 SSE  🔌 WebSocket"| A + C(["🧑‍💻 Client
Codex · Claude Code · SDKs · curl"]) -->|"📮 POST /v1/responses · /v1/messages
🌐 HTTP  📡 SSE  🔌 WebSocket"| A subgraph A ["⚡ Agentic API (Rust 🦀)"] direction TB S["🔄 State hydration
previous_response_id"] - T["🛠️ Server-side tools
web search · functions"] - P["💾 Persistence
SQLite response store"] + T["🛠️ Server-side tools
web search · MCP · code interpreter"] + P["💾 Persistence
SQLite · PostgreSQL"] end - A -->|"🚀 POST /v1/responses
⚙️ stateless  🤝 OpenAI-compatible"| V(["🚀 vLLM core
inference engine"]) + A -->|"🚀 POST /v1/responses · /v1/messages
⚙️ stateless  🤝 OpenAI/Anthropic-compatible"| V(["🚀 vLLM core
inference engine"]) classDef client fill:#FFE8B3,stroke:#F59E0B,stroke-width:2px,color:#7C2D12 classDef inner fill:#E0E7FF,stroke:#6366F1,stroke-width:2px,color:#312E81 @@ -53,12 +53,14 @@ flowchart LR ## ✨ Key Features -- 🔄 **Stateful conversations**: the server manages history via `previous_response_id`. No client-side message tracking, no replaying full transcripts. -- 🛠️ **Server-side tool execution**: an explicit tool-ownership model (gateway / client / provider) decides exactly what runs where. Web search ships today via [You.com](https://you.com), [Brave Search](https://brave.com/search/api/), or [Tavily](https://tavily.com), and the model executes multi-step tool chains automatically. -- 📡 **Every transport**: non-streaming HTTP, server-sent events for token streaming, and full **WebSocket** support for interactive clients. -- 🧰 **Codex-ready**: accepts Codex-shaped Responses traffic out of the box, preserving the tool declarations and response item shapes Codex depends on. -- 🏃 **Background execution**: fire-and-forget requests that keep processing server-side. -- ✅ **Compatibility tested**: validated against the [Open Responses](https://www.openresponses.org/) compatibility suite, with replay-cassette tests for real OpenAI and vLLM traffic. +- 🔄 **Stateful conversations**: the server manages history via `previous_response_id` or the [Conversations API](#-api-surface). No client-side message tracking, no replaying full transcripts. +- 🛠️ **Server-side tool execution**: an explicit tool-ownership model (gateway / client / provider) decides exactly what runs where. Web search ships today via [You.com](https://you.com), [Brave Search](https://brave.com/search/api/), or [Tavily](https://tavily.com), alongside MCP tools and an opt-in [embedded code interpreter](#optional-embedded-code-interpreter); the model executes multi-step tool chains automatically. +- 🤝 **Multi-agent orchestration**: a stored Responses request can spawn and coordinate subagents server-side over HTTP, returning their attributed work in one response ([details](#-multi-agent-responses)). +- 📡 **Every transport**: non-streaming HTTP, server-sent events for token streaming, and full **WebSocket** support for interactive clients, with `stream_id` multiplexing. +- 🧰 **Codex and Claude Code ready**: accepts Codex-shaped Responses traffic and Anthropic Messages traffic, preserving the tool declarations and item shapes each client depends on. +- 🗜️ **Compaction**: automatic and explicit context compaction through `/v1/responses/compact` ([guide](docs/guides/responses-compaction.md)). +- 🔭 **Observability**: opt-in OpenTelemetry traces and metrics over OTLP ([guide](docs/deploying/observability.md)). +- ✅ **Compatibility tested**: validated against the [Open Responses](https://www.openresponses.org/) compatibility suite, with replay-cassette tests for real OpenAI, vLLM, SGLang, and NVIDIA Dynamo traffic. ## 🧭 API Surface @@ -67,13 +69,18 @@ flowchart LR | `POST /v1/responses` | OpenAI-compatible Responses API with state, tools, and streaming | ✅ | | `GET /v1/responses/{response_id}` | Retrieve a locally stored response | ✅ | | `GET /v1/responses` | WebSocket transport for the Responses API | ✅ | -| `POST /v1/conversations` | Conversation management | ✅ | +| `POST /v1/responses/compact` | Compact direct input or a stored response chain | ✅ | +| `/v1/conversations` · `/v1/conversations/{id}/items` | Conversations API: create, retrieve, update, and delete conversations and their items | ✅ | +| `POST /v1/messages` · `POST /v1/messages/count_tokens` | Anthropic Messages API forwarded to the upstream, with a server-side loop for gateway-owned tools | ✅ | | `GET /v1/models` | Model listing proxied from vLLM | ✅ | | `POST /v1/chat/completions` · `POST /v1/completions` | Forwarded to the upstream verbatim, so clients on those endpoints keep working when the gateway is the entry point | ✅ | | `GET /health` · `GET /ready` | Liveness and readiness probes | ✅ | -| Messages API | Anthropic-style stateful messages on shared primitives | 🚧 Planned | +| `GET /swagger-ui` · `GET /openapi.json` | Generated OpenAPI spec and Swagger UI, served when `ENABLE_OPENAPI_DOCS=true` | ✅ | +| Stateful Messages | Messages continuation on the shared persistence primitives | ⏳ Planned | | Interactions API | Higher-level agentic workflow surface | ⏳ Planned | +See the [API reference](docs/api/index.md) for routing, authentication, and WebSocket details. + Responses created with `store: true` retain a terminal snapshot for retrieval, including status, usage, and this turn's output. Retrieval does not call the upstream model. Unknown IDs return a JSON `404`; older records created before snapshot storage (or through history-only APIs) return `409` because their @@ -329,8 +336,8 @@ parsed. Order of precedence is `--max-request-body-size-bytes`, then `AGENTIC_MA file setting. `api_key_env` names the process environment variable containing the web-search credential; it does not contain the -credential itself. When it is unset, the selected provider's conventional variable is read (`YOU_API_KEY` or -`BRAVE_API_KEY`). Newly generated files leave `api_key_env` unset so changing providers also changes the default +credential itself. When it is unset, the selected provider's conventional variable is read (`YOU_API_KEY`, +`BRAVE_API_KEY`, or `TAVILY_API_KEY`). Newly generated files leave `api_key_env` unset so changing providers also changes the default credential variable. `AGENTIC_WEB_SEARCH_PROVIDER`, `AGENTIC_WEB_SEARCH_BASE_URL`, `AGENTIC_WEB_SEARCH_MAX_CONCURRENT_QUERIES`, `AGENTIC_MCP_ALLOWED_HOSTS`, `AGENTIC_MAX_REQUEST_BODY_SIZE_BYTES`, and `AGENTIC_MAX_CONCURRENT_GATEWAY_CALLS` can override their typed file settings; `YOU_API_BASE_URL` is still honored as the endpoint override when the provider is @@ -584,19 +591,44 @@ Every tool call has exactly one execution path, so nothing runs by accident: | Ownership | Who executes it | Examples | | --- | --- | --- | -| **Gateway-owned** | Agentic API executes it server-side and continues the loop | Web search, file search, MCP-backed tools | -| **Client-owned** | Preserved and returned to the client | Codex shell / editor tools, your functions | +| **Gateway-owned** | Agentic API executes it server-side and continues the loop | Web search, MCP-backed tools, code interpreter (opt-in) | +| **Client-owned** | Preserved and returned to the client | Shell tool, Codex editor tools, your functions | | **Provider-owned** | Passed through to vLLM or an upstream provider | Provider-native tools | Unknown or ambiguous tool shapes are **never executed by default**. They are preserved and returned. +## 🤝 Multi-agent Responses + +Set `multi_agent.enabled` on a stored Responses request and the gateway runs the [Responses multi-agent](https://developers.openai.com/api/docs/guides/responses-multi-agent) collaboration loop itself: the root agent spawns subagents, each with its own history and tool registry, and the response returns their attributed `multi_agent_call`, `multi_agent_call_output`, and `agent_message` items. Continue the whole agent tree with `previous_response_id`. + +```bash +curl http://localhost:9000/v1/responses \ + -H "Content-Type: application/json" \ + -d '{ + "model": "Qwen/Qwen3-30B-A3B-FP8", + "store": true, + "multi_agent": {"enabled": true, "max_concurrent_subagents": 3}, + "input": "Have one agent review correctness and another review test coverage, then summarize." + }' +``` + +- `store: true` is required; the gateway does not yet support stateless multi-agent requests. +- `max_concurrent_subagents` limits active descendant turns (default `3`); it is not a total agent count. +- Multi-agent execution is HTTP and SSE only. WebSocket requests with multi-agent enabled are rejected. +- `max_tool_calls` and reasoning summaries are rejected while multi-agent is enabled. + +See [HTTP multi-agent execution](ARCHITECTURE.md#http-multi-agent-execution) for scheduling and persistence details. + ## 🏗️ Repository Layout ``` crates/ -├── agentic-server/ # Axum binary, transport handlers (HTTP/SSE/WS), configuration +├── agentic-server/ # Axum binary, transport handlers (HTTP/SSE/WS), configuration, `agentic` CLI ├── agentic-server-core/ # Protocol types, executor, tool framework, persistence +├── agentic-llm-d/ # Split-execution state backend for the llm-d coordinator +├── agentic-cli-docs/ # Build-time generator for the CLI reference docs └── agentic-praxis/ # Praxis gateway integration +python/agentic_api/ # Python distribution and launcher for the packaged gateway docs/ # MkDocs documentation, ADRs, and design notes ``` @@ -623,10 +655,14 @@ Design and migration decisions are tracked as ADRs in [docs/adr/](docs/adr/), wi - [x] **Responses API hydration**: stateful continuation with `previous_response_id` - [x] **Codex support**: practical Codex sessions through the Responses API -- [x] **Server-side tool execution**: explicit ownership, web search built in -- [ ] **Messages API**: built on the same persistence and execution primitives +- [x] **Server-side tool execution**: explicit ownership, web search, MCP, and code interpreter built in +- [x] **Messages API**: Claude Code support with a server-side gateway tool loop +- [x] **Conversations API**: OpenAI-compatible conversation and item management +- [ ] **Multi-agent Responses**: HTTP execution ships today; WebSocket injection and stateless requests are next +- [ ] **File search**: Files and Vector Stores APIs with a gateway-executed `file_search` tool +- [ ] **Stateful Messages**: Messages continuation on the same persistence and execution primitives - [ ] **Interactions API**: durable, higher-level agentic workflows -- [ ] **Production hardening**: storage backends, observability, cached-prefix continuation +- [ ] **Production hardening**: tenant-scoped state, gateway metrics, cached-prefix continuation ______________________________________________________________________ diff --git a/ROADMAP.md b/ROADMAP.md index 17f322ec..865799d2 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -84,6 +84,7 @@ Initial and expected tool areas include: - File search. - Web search. +- Code interpreter. - Web fetch. - Computer use. - Shell tools. @@ -94,6 +95,10 @@ Initial and expected tool areas include: The third major API goal is Messages. +Status: `/v1/messages` forwards Anthropic Messages traffic to the upstream and +runs a server-side loop for gateway-owned tools, which is enough for Claude +Code. Stateful continuation and persistence for Messages remain the work below. + Messages should build on the same persistence, rehydration, and execution foundations as Responses, while exposing the API surface expected by agentic clients that organize state around message objects. @@ -191,9 +196,18 @@ independent WebSocket lanes, async calls, and steering have distinct semantics. ([#334](https://github.com/vllm-project/agentic-api/issues/334)). - **Multi-agent orchestration:** implement isolated agent contexts and shared scheduling through the existing tracker - ([#298](https://github.com/vllm-project/agentic-api/issues/298)), including HTTP - execution ([#299](https://github.com/vllm-project/agentic-api/issues/299)) and - WebSocket injection ([#300](https://github.com/vllm-project/agentic-api/issues/300)). + ([#298](https://github.com/vllm-project/agentic-api/issues/298)). HTTP execution + for stored requests has shipped + ([#299](https://github.com/vllm-project/agentic-api/issues/299)); WebSocket + injection ([#300](https://github.com/vllm-project/agentic-api/issues/300)) and + stateless requests ([#393](https://github.com/vllm-project/agentic-api/issues/393)) + remain. +- **File search:** add the Files and Vector Stores APIs, indexed retrieval, and a + gateway-executed `file_search` tool with OpenAI-compatible streaming events + ([#34](https://github.com/vllm-project/agentic-api/pull/34)). Resolve `file_id` + references in Responses input against the same file store + ([#356](https://github.com/vllm-project/agentic-api/issues/356)). Until then, + `file_search` declarations are accepted but not offered to the model. - **Computer use:** add typed computer calls, ordered actions, screenshots, and explicit execution-location handling through the existing feature issue ([#171](https://github.com/vllm-project/agentic-api/issues/171)). diff --git a/docs/api/index.md b/docs/api/index.md index 3608ecca..1f4ce65e 100644 --- a/docs/api/index.md +++ b/docs/api/index.md @@ -65,15 +65,28 @@ to every other `/v1/*` route; the configured upstream credential is injected onl ### `POST /v1/responses` HTTP Responses requests use the OpenAI-compatible Responses shape. Requests -with `store=true`, `previous_response_id`, `conversation_id`, compaction input, -or `context_management` run through the executor. Other stateless `store=false` +with `store=true`, `previous_response_id`, `conversation_id`, any non-function +tool, `multi_agent.enabled`, tool-search state, compaction input, or +`context_management` run through the executor. Other stateless `store=false` requests are passed directly to the configured vLLM backend. A `store=false` request that continues a stored response is hydrated from that response but is not stored itself: its id can be neither retrieved nor used as `previous_response_id`. Executor-backed requests accept at most 64 MCP server declarations and 128 -discovered MCP tools. MCP discovery metadata shares the request's 1 MiB -response budget with upstream rounds and gateway tool output. +discovered MCP tools. MCP discovery metadata shares the request's retained +response budget (`[responses] max_retained_bytes`, default 8 MiB) with upstream +rounds and gateway tool output. + +#### Multi-agent + +`multi_agent: {"enabled": true, "max_concurrent_subagents": 3}` makes the +gateway orchestrate subagents for one response and return attributed +`multi_agent_call`, `multi_agent_call_output`, and `agent_message` items. +`max_concurrent_subagents` defaults to 3 and must be positive. Multi-agent +requests require `store: true`, reject `max_tool_calls` and reasoning summaries, +and run over HTTP JSON and SSE only; the WebSocket transport rejects them. +Continue the agent tree with `previous_response_id`. See +[HTTP multi-agent execution](https://github.com/vllm-project/agentic-api/blob/main/ARCHITECTURE.md#http-multi-agent-execution). `prompt_cache_key` is forwarded unchanged on direct and executor-backed HTTP requests, WebSocket requests, gateway tool rounds, automatic compaction, and @@ -89,6 +102,15 @@ summary inference requested by a `compaction_trigger` input item. The configured upstream decides whether a request receives a cache hit. +### `GET /v1/responses/{response_id}` + +Returns the terminal snapshot of a response created with `store: true`, including +status, usage, and that turn's output, without calling the upstream model. +Unknown IDs return `404`. Records created before snapshot storage, or through +history-only APIs, return `409`. Request-scoped MCP credentials are stripped +before storage. When OIDC is disabled and `OPENAI_API_KEY` is nonempty, callers +must send that key as a bearer token. + ### `POST /v1/responses/compact` Compacts direct input or a stored previous-response chain into a canonical @@ -144,11 +166,12 @@ default lane; reconnect to start new lanes after that limit. Each retained check is limited to 32,768 items and 16 MiB of serialized state, with a 32 MiB connection budget covering cached, active-parent and prepared replacement checkpoints. Exceeding a checkpoint budget returns 413 before writing response state. These limits do not -measure total process memory. Upstream SSE lines are limited to 256 KiB, normalized -executor events are limited to 1 MiB, and each request shares a 1 MiB response -budget across MCP discovery, upstream rounds, and normalized gateway tool output. -Every outbound WebSocket event, including `stream_id`, is limited to 1 MiB of -serialized JSON. Local `generate: false` requests validate both lifecycle events +measure total process memory. Upstream SSE lines, the per-request retained response +budget, and outbound events follow the `[responses]` limits in `config.toml` +(`max_upstream_sse_line_bytes`, `max_retained_bytes`, and `max_stream_event_bytes`; +defaults 16 MiB, 8 MiB, and 16 MiB). Every outbound WebSocket event, including +`stream_id`, counts toward `max_stream_event_bytes`. Multi-agent requests are +not supported on this transport. Local `generate: false` requests validate both lifecycle events before storing the response or emitting either event. OIDC identity expiry is checked again when queued work starts, including work in the default lane; expired work receives a tagged `invalid_token` event without reaching inference. @@ -168,3 +191,33 @@ Invalid requests are returned as JSON WebSocket error events: } } ``` + +## Conversations + +| Route | Operations | +| --- | --- | +| `/v1/conversations` | `POST` create, optionally with initial `items` | +| `/v1/conversations/{conversation_id}` | `GET` retrieve, `POST` update metadata, `DELETE` delete | +| `/v1/conversations/{conversation_id}/items` | `POST` add items, `GET` list items with cursor pagination and `order` | +| `/v1/conversations/{conversation_id}/items/{item_id}` | `GET` retrieve, `DELETE` delete | + +Pass `conversation` to `POST /v1/responses` to continue a conversation; Responses +persistence appends each turn's items. Deleting an item detaches it from the +conversation without removing stored response history. Conversation creation and +its initial items commit atomically. These routes currently use the default +tenant rather than deriving ownership from the authenticated principal +([#107](https://github.com/vllm-project/agentic-api/issues/107)). + +## Messages + +### `POST /v1/messages` · `POST /v1/messages/count_tokens` + +Anthropic Messages requests are forwarded to `{LLM_API_BASE}` with Anthropic +headers and body fields preserved. When a request declares a tool the gateway +owns, such as the native `web_search_20250305` server tool, the gateway runs +the tool loop itself: it executes each gateway call, appends the result, and +streams only the client-visible content. Every upstream round must be a +complete, well-formed Messages stream; a truncated or malformed round ends the +response with an Anthropic `api_error` event before any gateway tool runs. +Messages requests are not persisted. See +[Claude Code integration](../design/claude-code-integration.md). diff --git a/docs/index.md b/docs/index.md index f7844b9d..c9f6b576 100644 --- a/docs/index.md +++ b/docs/index.md @@ -23,15 +23,20 @@ vLLM Agentic API provides the stateful APIs needed for real-world agentic applic This project is in early development. Follow along and contribute on [GitHub](https://github.com/vllm-project/agentic-api). -## Responses API +## Agentic APIs -Our first milestone is implementing the [Responses API](https://platform.openai.com/docs/api-reference/responses), bringing stateful, agentic capabilities to vLLM. We validate our implementation against the [Open Responses](https://www.openresponses.org/) compatibility test suite. +Agentic API implements the OpenAI-compatible [Responses API](https://platform.openai.com/docs/api-reference/responses) +and serves the Anthropic Messages API for Claude Code. We validate the Responses implementation against the +[Open Responses](https://www.openresponses.org/) compatibility test suite. -- **Stateful conversations** — The server manages conversation history via `previous_response_id`, eliminating client-side message tracking -- **Built-in tool use** — Web search, file search, and function calling handled within the API, with the model automatically executing multi-step tool chains -- **Streaming** — Server-sent events for real-time token streaming with structured lifecycle events -- **Background execution** — Fire-and-forget requests that continue processing server-side -- **Compatibility tested** — Validated against the open Responses API compatibility test suite +- **Stateful conversations** — The server manages conversation history via `previous_response_id` or the Conversations API, eliminating client-side message tracking +- **Server-side tool execution** — Web search, MCP tools, and an opt-in embedded code interpreter run inside the gateway, with the model automatically executing multi-step tool chains +- **Multi-agent orchestration** — Stored HTTP Responses requests can spawn and coordinate subagents server-side +- **Streaming** — Server-sent events and WebSocket transports with structured lifecycle events +- **Compaction** — Automatic and explicit context compaction for long-running sessions +- **Compatibility tested** — Validated against the open Responses API compatibility test suite and replay recordings of vLLM, SGLang, and NVIDIA Dynamo traffic + +See the [API reference](api/index.md) for the full endpoint list. ## Python Distribution diff --git a/mkdocs.yaml b/mkdocs.yaml index 110f8486..2a9e716d 100644 --- a/mkdocs.yaml +++ b/mkdocs.yaml @@ -128,6 +128,7 @@ nav: - MCP Gateway Integration: design/mcp-gateway-integration.md - Messages Gateway Tool Classification: design/messages-gateway-tool-classification.md - OIDC Bearer Authentication: design/oidc-bearer-authentication.md + - Shell Tool: design/shell-tool.md - Tool Framework: design/tool-framework.md - Deploying: - Container: deploying/container.md