diff --git a/packages/agent-harness/readme.md b/packages/agent-harness/readme.md index ac94bdb..b6cf4b2 100644 --- a/packages/agent-harness/readme.md +++ b/packages/agent-harness/readme.md @@ -45,13 +45,15 @@ constrained modes). the count and add your id, or the suite fails. - [`tests/harnesses.test.ts`](tests/harnesses.test.ts) has a `describe` block per harness asserting metadata (`name`, `displayName`, `defaultPath`, - `promptFlag`, `supportsMaxTurns`) and exact `buildArgs()` output for empty / - partial / full option sets, including mode combinations. Mirror that layout. + `promptFlag`, `supportsMaxTurns`, `supportsStructuredOutput`) and exact + `buildArgs()` output for empty / partial / full option sets, including mode + combinations. Mirror that layout. - If you extend shared behavior (prompt delivery, attachments, limit - patterns), the relevant suites live alongside: + patterns, structured output), the relevant suites live alongside: [`tests/prompt-args.test.ts`](tests/prompt-args.test.ts), [`tests/attachments.test.ts`](tests/attachments.test.ts), - [`tests/detect-usage-limit.test.ts`](tests/detect-usage-limit.test.ts), etc. + [`tests/detect-usage-limit.test.ts`](tests/detect-usage-limit.test.ts), + [`tests/structured-output.test.ts`](tests/structured-output.test.ts), etc. ## Capability flags that matter @@ -62,6 +64,7 @@ See [`AgentHarness`](src/types.ts) for full semantics. Quick decision table: | `name` / `displayName` / `defaultPath` | Always. `defaultPath` is the binary looked up on `PATH` (override-able via resolver env vars). | | [`supportedModes`](src/modes.ts) | List only plan/readonly modes the CLI can **natively enforce** via flags. If you cannot enforce them, leave empty (or omit) — requests fail closed via `assertModeSupported` inside `buildArgs`. Never fake a mode by ignoring it. | | `supportsMaxTurns` | Set `true` only if the CLI accepts a turn-limit flag *and* emits a recognizable diagnostic on exhaustion. Callers skip transcript scanning when this is false/unset, so tool output cannot be mistaken for a turn-limit error. | +| [`supportsStructuredOutput`](src/structured-output.ts) | Set `true` only if the CLI has a documented JSON output mode (verify upstream docs — don't assume). Supporting harnesses emit the flag in `buildArgs` when `options.structuredOutput` is set; runners fail closed with `UnsupportedStructuredOutputError` when it is false/unset. See the per-CLI flag table in [Structured (JSON) output](#structured-json-output). | | `constrainedModeAllowsExternalTools` | Set `true` only if your constrained mode still permits network + MCP tools. No built-in harness sets it today (Codex's read-only sandbox disables network; Claude's plan mode denies non-annotated MCP tools, which aborts headless runs). Callers whose agents need web/MCP access skip constrained modes unless this is true. | | `promptFlag` | Set when the prompt must arrive as a flag value (`kimi --prompt "..."`). Omit for positional prompts (`codex exec "..."`). Prefer argv over stdin — see below. | | `imageInput` / `buildImageArgs` | `"path"` (default): images go into the prompt as markdown paths only. `"native"`: also emit CLI flags via `buildImageArgs(paths)` after the prompt (Codex `-i`). Runners call [`preparePromptWithAttachments`](src/attachments.ts); paths should also appear in `attachmentPaths`. | @@ -71,6 +74,55 @@ first; use [`effectiveSkipPermissions`](src/modes.ts) instead of reading `options.skipPermissions` directly — constrained modes always suppress YOLO / bypass flags. When `mode` is plan/readonly, never emit write-capable flags. +## Structured (JSON) output + +[`src/structured-output.ts`](src/structured-output.ts) lets callers request +machine-readable output per run: pass `structuredOutput: true` in +[`AgentRunOptions`](src/types.ts). This is **opt-in** — the plain-text default +keeps transcript scanning (usage-limit, max-turns, incomplete-implementation, +open-question detectors) working exactly as before. + +- **Capability.** A harness sets `supportsStructuredOutput = true` only when + its CLI documents a JSON output mode; supporting harnesses emit the flag in + `buildArgs` when the option is requested. Runners call + `assertStructuredOutputSupported` before spawning, so requesting JSON from + an unsupported harness throws [`UnsupportedStructuredOutputError`](src/structured-output.ts) + instead of silently returning styled text (same fail-closed contract as + `UnsupportedAgentModeError`). +- **Result.** `AgentRunResult.structured` (`StructuredOutputResult`) carries + the parsed payload: `{ ok: true, value }` where `value` is the single JSON + document (Claude/Cursor/Grok/agy envelopes, Qwen message arrays) or an + array of NDJSON event objects (Codex/Opencode/Kilo/Cline/Kimi/Pi streams). + On failure it is `{ ok: false, error }` — empty output, no JSON found, or a + malformed/truncated payload. Fail closed: partial NDJSON is **not** + returned as a complete result; `stdout`/`stderr` always remain for + diagnostics. The field is absent when the option is unset, so existing + callers are unaffected. +- **Per-CLI flags wired into `buildArgs`** (verified against upstream docs): + + | Harness | Flag | Shape | + | --- | --- | --- | + | claude-code | `--output-format json` | single result envelope | + | cline | `--json` | NDJSON messages | + | codex | `--json` | JSONL events | + | cursor | `--output-format json` | single result object | + | opencode | `--format json` | JSONL events | + | grok | `--output-format json` | single result object | + | goose | `--output-format json` | single result object | + | antigravity (agy, v1.1.8+) | `--output-format json` | single JSON envelope | + | qwen | `--output-format json` | JSON array of messages | + | kilo-code | `--format json` | JSONL events | + | kimi | `--output-format stream-json` | JSONL messages | + | pi | `--mode json` (pairs with `-p`) | JSONL events | + | deepseek (reasonix) | `--output-format json` | single result object | + +- **Detectors & streaming.** Usage-limit and max-turns detectors keep + scanning the raw streams (JSON event lines can still match + provider-diagnostic patterns, e.g. `{"error":{...}}`). `onStdout` / + `onStderr` callbacks and Bun/Node runner behavior are identical in both + modes; parsing happens once after exit. Callers needing live deltas should + consume the streaming callbacks as today. + ## Prompt & input conventions - **Arg, not stdin.** The prompt goes on the command line via diff --git a/packages/agent-harness/src/harnesses/antigravity.ts b/packages/agent-harness/src/harnesses/antigravity.ts index b22d54f..e88685c 100644 --- a/packages/agent-harness/src/harnesses/antigravity.ts +++ b/packages/agent-harness/src/harnesses/antigravity.ts @@ -34,13 +34,20 @@ export class AntigravityHarness implements AgentHarness { /** Executable on PATH after install (`curl -fsSL https://antigravity.google/cli/install.sh | bash`). */ readonly defaultPath = "agy"; readonly promptFlag = "-p"; + /** + * `--output-format json` (print mode, v1.1.8+) emits one JSON envelope with + * status, response text, and usage metadata. + */ + readonly supportsStructuredOutput = true; /** * Build `agy` CLI flags for headless (`-p`) execution. * - * @param options - Supports `skipPermissions` (`--dangerously-skip-permissions`) - * and `model` (`--model`, added upstream in v1.0.5; takes a model slug as - * listed by `agy models`). Max-turns has no `--max-turns` equivalent. + * @param options - Supports `skipPermissions` + * (`--dangerously-skip-permissions`), `model` (`--model`, added upstream + * in v1.0.5; takes a model slug as listed by `agy models`), + * `structuredOutput`, and `workingDir`. Max-turns has no `--max-turns` + * equivalent. * @returns Args excluding the prompt (runner supplies `-p` via {@link promptFlag}). */ buildArgs(options: AgentRunOptions): string[] { @@ -57,6 +64,10 @@ export class AntigravityHarness implements AgentHarness { args.push("--model", options.model); } + if (options.structuredOutput) { + args.push("--output-format", "json"); + } + // Antigravity does not currently support --max-turns. // If it adds support in the future, uncomment: // if (options.maxTurns !== undefined) { diff --git a/packages/agent-harness/src/harnesses/claude-code.ts b/packages/agent-harness/src/harnesses/claude-code.ts index 2ec6d5e..841b607 100644 --- a/packages/agent-harness/src/harnesses/claude-code.ts +++ b/packages/agent-harness/src/harnesses/claude-code.ts @@ -33,11 +33,14 @@ export class ClaudeCodeHarness implements AgentHarness { readonly promptFlag = "-p"; readonly supportedModes = ["plan", "readonly"] as const; readonly supportsMaxTurns = true; + /** Print mode accepts `--output-format json` (single JSON envelope, result in `result`). */ + readonly supportsStructuredOutput = true; /** * Build `claude` CLI flags for non-interactive (`-p`) execution. * - * @param options - Supports `mode`, `skipPermissions`, `model`, and `maxTurns`. + * @param options - Supports `mode`, `skipPermissions`, `model`, `maxTurns`, + * and `structuredOutput`. * @returns Args excluding the prompt (runner supplies `-p` via {@link promptFlag}). */ buildArgs(options: AgentRunOptions): string[] { @@ -63,6 +66,10 @@ export class ClaudeCodeHarness implements AgentHarness { args.push("--max-turns", String(options.maxTurns)); } + if (options.structuredOutput) { + args.push("--output-format", "json"); + } + return args; } } diff --git a/packages/agent-harness/src/harnesses/cline.ts b/packages/agent-harness/src/harnesses/cline.ts index fe0927e..5c1c46a 100644 --- a/packages/agent-harness/src/harnesses/cline.ts +++ b/packages/agent-harness/src/harnesses/cline.ts @@ -18,11 +18,14 @@ export class ClineHarness implements AgentHarness { readonly defaultPath = "cline"; /** No native plan/read-only enforcement documented for headless `cline task`. */ readonly supportedModes = [] as const; + /** `--json` emits newline-delimited JSON message objects instead of styled text. */ + readonly supportsStructuredOutput = true; /** * Build `cline task` subcommand flags for non-interactive execution. * - * @param options - Supports `skipPermissions` (`--yolo`) and `model`. + * @param options - Supports `skipPermissions` (`--yolo`), `model`, and + * `structuredOutput` (`--json`). * @returns Args starting with `task`; prompt is appended as a positional argument. */ buildArgs(options: AgentRunOptions): string[] { @@ -37,6 +40,10 @@ export class ClineHarness implements AgentHarness { args.push("--model", options.model); } + if (options.structuredOutput) { + args.push("--json"); + } + // Cline does not currently support --max-turns. // If it adds support in the future, uncomment the following: // if (options.maxTurns !== undefined) { diff --git a/packages/agent-harness/src/harnesses/codex.ts b/packages/agent-harness/src/harnesses/codex.ts index 79e755e..4fec1e2 100644 --- a/packages/agent-harness/src/harnesses/codex.ts +++ b/packages/agent-harness/src/harnesses/codex.ts @@ -94,6 +94,8 @@ export class CodexHarness implements AgentHarness { readonly supportedModes = ["plan", "readonly"] as const; /** Codex `exec` accepts images via `-i` after the prompt. */ readonly imageInput = "native" as const; + /** `--json` prints stdout as newline-delimited JSON events (thread/turn/item). */ + readonly supportsStructuredOutput = true; /** * Build trailing `-i` flags for native image attachment. @@ -115,7 +117,8 @@ export class CodexHarness implements AgentHarness { /** * Build `codex exec` flags for non-interactive execution. * - * @param options - Supports `mode`, `skipPermissions` (sandbox + approval), and `model`. + * @param options - Supports `mode`, `skipPermissions` (sandbox + approval), + * `model`, and `structuredOutput` (`--json`). * @returns Args starting with `exec`; prompt is appended as a positional argument. */ buildArgs(options: AgentRunOptions): string[] { @@ -143,6 +146,10 @@ export class CodexHarness implements AgentHarness { args.push("--model", options.model); } + if (options.structuredOutput) { + args.push("--json"); + } + // Codex does not currently support --max-turns in exec mode. // If it adds support in the future, uncomment the following: // if (options.maxTurns !== undefined) { diff --git a/packages/agent-harness/src/harnesses/cursor.ts b/packages/agent-harness/src/harnesses/cursor.ts index 16fc303..6d250d4 100644 --- a/packages/agent-harness/src/harnesses/cursor.ts +++ b/packages/agent-harness/src/harnesses/cursor.ts @@ -31,12 +31,14 @@ export class CursorHarness implements AgentHarness { // unambiguous name — other CLIs (e.g. Grok) also install as `agent`. readonly defaultPath = "cursor-agent"; readonly supportedModes = ["plan", "readonly"] as const; + /** `--output-format json` emits one result object on completion (requires `-p`). */ + readonly supportsStructuredOutput = true; /** * Build Cursor `agent` CLI flags for non-interactive (`-p`) execution. * * @param options - Supports `mode`, `skipPermissions` (`--force`, `--trust`, - * `--approve-mcps`), and `model`. + * `--approve-mcps`), `model`, and `structuredOutput`. * @returns Args excluding the prompt (runner supplies it positionally). */ buildArgs(options: AgentRunOptions): string[] { @@ -58,6 +60,10 @@ export class CursorHarness implements AgentHarness { args.push("--model", options.model); } + if (options.structuredOutput) { + args.push("--output-format", "json"); + } + // Cursor does not currently support --max-turns. // If it adds support in the future, uncomment the following: // if (options.maxTurns !== undefined) { diff --git a/packages/agent-harness/src/harnesses/deepseek.ts b/packages/agent-harness/src/harnesses/deepseek.ts index daebc79..41292e0 100644 --- a/packages/agent-harness/src/harnesses/deepseek.ts +++ b/packages/agent-harness/src/harnesses/deepseek.ts @@ -25,12 +25,14 @@ export class DeepSeekHarness implements AgentHarness { readonly defaultPath = "reasonix"; /** No native plan/read-only enforcement documented for headless `reasonix run`. */ readonly supportedModes = [] as const; + /** `--output-format json` emits one final result object (verified in docs/CLI.md). */ + readonly supportsStructuredOutput = true; /** * Build `reasonix run` flags for non-interactive execution. * - * @param options - Supports `model`. `skipPermissions` and `maxTurns` are not - * exposed as CLI flags for `reasonix run`. + * @param options - Supports `model` and `structuredOutput`. `skipPermissions` + * and `maxTurns` are not exposed as CLI flags for `reasonix run`. * @returns Args starting with `run`; prompt is appended as a positional argument. */ buildArgs(options: AgentRunOptions): string[] { @@ -44,6 +46,10 @@ export class DeepSeekHarness implements AgentHarness { args.push("--model", options.model); } + if (options.structuredOutput) { + args.push("--output-format", "json"); + } + // Reasonix turn limits are configured via max_steps in reasonix.toml / // ~/.reasonix/config.toml, not a CLI --max-turns flag. // If it adds support in the future, uncomment the following: diff --git a/packages/agent-harness/src/harnesses/goose.ts b/packages/agent-harness/src/harnesses/goose.ts index 23db1de..fc4fda0 100644 --- a/packages/agent-harness/src/harnesses/goose.ts +++ b/packages/agent-harness/src/harnesses/goose.ts @@ -22,11 +22,14 @@ export class GooseHarness implements AgentHarness { readonly promptFlag = "-t"; /** No native plan/read-only enforcement documented for headless `goose run`. */ readonly supportedModes = [] as const; + /** `--output-format json` emits structured session/results output after completion. */ + readonly supportsStructuredOutput = true; /** * Build `goose run` flags for non-interactive (`-t`) execution. * - * @param options - Supports `skipPermissions` (`--no-session`) and `model`. + * @param options - Supports `skipPermissions` (`--no-session`), `model`, and + * `structuredOutput`. * @returns Args starting with `run`; prompt is supplied via {@link promptFlag}. */ buildArgs(options: AgentRunOptions): string[] { @@ -44,6 +47,10 @@ export class GooseHarness implements AgentHarness { args.push("--model", options.model); } + if (options.structuredOutput) { + args.push("--output-format", "json"); + } + // Goose does not currently support --max-turns via CLI flag. // If it adds support in the future, uncomment the following: // if (options.maxTurns !== undefined) { diff --git a/packages/agent-harness/src/harnesses/grok.ts b/packages/agent-harness/src/harnesses/grok.ts index b11d68e..acdf605 100644 --- a/packages/agent-harness/src/harnesses/grok.ts +++ b/packages/agent-harness/src/harnesses/grok.ts @@ -35,12 +35,14 @@ export class GrokHarness implements AgentHarness { readonly defaultPath = "grok"; readonly promptFlag = "-p"; readonly supportedModes = ["plan", "readonly"] as const; + /** `--output-format json` emits one JSON object (text, stopReason, session, usage). */ + readonly supportsStructuredOutput = true; /** * Build `grok` CLI flags for headless (`-p`) execution. * - * @param options - Supports `mode`, `skipPermissions` (`--always-approve`), `model` (`-m`), - * and `workingDir` (`--cwd`). + * @param options - Supports `mode`, `skipPermissions` (`--always-approve`), + * `model` (`-m`), `structuredOutput`, and `workingDir` (`--cwd`). * @returns Args excluding the prompt (runner supplies `-p` via {@link promptFlag}). */ buildArgs(options: AgentRunOptions): string[] { @@ -62,6 +64,10 @@ export class GrokHarness implements AgentHarness { args.push("--cwd", options.workingDir); } + if (options.structuredOutput) { + args.push("--output-format", "json"); + } + // Grok Build does not currently support --max-turns. // If it adds support in the future, uncomment the following: // if (options.maxTurns !== undefined) { diff --git a/packages/agent-harness/src/harnesses/kilo-code.ts b/packages/agent-harness/src/harnesses/kilo-code.ts index db15cba..bc7e32d 100644 --- a/packages/agent-harness/src/harnesses/kilo-code.ts +++ b/packages/agent-harness/src/harnesses/kilo-code.ts @@ -18,11 +18,14 @@ export class KiloCodeHarness implements AgentHarness { readonly defaultPath = "kilo"; /** No native plan/read-only enforcement documented for headless `kilo run`. */ readonly supportedModes = [] as const; + /** `--format json` streams raw JSON events (one object per line) to stdout. */ + readonly supportsStructuredOutput = true; /** * Build `kilo run` subcommand flags for non-interactive execution. * - * @param options - Supports `skipPermissions` (`--auto`) and `model`. + * @param options - Supports `skipPermissions` (`--auto`), `model`, and + * `structuredOutput`. * @returns Args starting with `run`; prompt is appended as a positional argument. */ buildArgs(options: AgentRunOptions): string[] { @@ -37,6 +40,10 @@ export class KiloCodeHarness implements AgentHarness { args.push("--model", options.model); } + if (options.structuredOutput) { + args.push("--format", "json"); + } + // Kilo Code does not currently support --max-turns. // If it adds support in the future, uncomment the following: // if (options.maxTurns !== undefined) { diff --git a/packages/agent-harness/src/harnesses/kimi.ts b/packages/agent-harness/src/harnesses/kimi.ts index 25eb9aa..dbe4d0f 100644 --- a/packages/agent-harness/src/harnesses/kimi.ts +++ b/packages/agent-harness/src/harnesses/kimi.ts @@ -19,11 +19,17 @@ export class KimiHarness implements AgentHarness { readonly promptFlag = "--prompt"; /** No native plan/read-only enforcement documented for headless `kimi --print`. */ readonly supportedModes = [] as const; + /** + * Print mode accepts `--output-format stream-json` (JSONL messages); its + * only structured format. + */ + readonly supportsStructuredOutput = true; /** * Build `kimi --print` flags for non-interactive execution. * - * @param options - Supports `skipPermissions` (`--yolo`) and `model`. + * @param options - Supports `skipPermissions` (`--yolo`), `model`, and + * `structuredOutput`. * @returns Args including `--print`; prompt is supplied via {@link promptFlag}. */ buildArgs(options: AgentRunOptions): string[] { @@ -38,6 +44,10 @@ export class KimiHarness implements AgentHarness { args.push("--model", options.model); } + if (options.structuredOutput) { + args.push("--output-format", "stream-json"); + } + // Kimi CLI does not currently support --max-turns. // If it adds support in the future, uncomment the following: // if (options.maxTurns !== undefined) { diff --git a/packages/agent-harness/src/harnesses/opencode.ts b/packages/agent-harness/src/harnesses/opencode.ts index 20044e5..c038855 100644 --- a/packages/agent-harness/src/harnesses/opencode.ts +++ b/packages/agent-harness/src/harnesses/opencode.ts @@ -34,11 +34,14 @@ export class OpencodeHarness implements AgentHarness { readonly displayName = "Opencode"; readonly defaultPath = "opencode"; readonly supportedModes = ["plan", "readonly"] as const; + /** `--format json` streams raw JSON events (one object per line) to stdout. */ + readonly supportsStructuredOutput = true; /** * Build `opencode run` flags for non-interactive execution. * - * @param options - Supports `mode`, `skipPermissions`, `model`, and `workingDir`. + * @param options - Supports `mode`, `skipPermissions`, `model`, + * `structuredOutput`, and `workingDir`. * @returns Args starting with `run`; prompt is appended as a positional argument. */ buildArgs(options: AgentRunOptions): string[] { @@ -64,6 +67,10 @@ export class OpencodeHarness implements AgentHarness { args.push("--model", options.model); } + if (options.structuredOutput) { + args.push("--format", "json"); + } + // Opencode does not currently support --max-turns. // If it adds support in the future, uncomment the following: // if (options.maxTurns !== undefined) { diff --git a/packages/agent-harness/src/harnesses/pi.ts b/packages/agent-harness/src/harnesses/pi.ts index ad4c5b1..051f5d8 100644 --- a/packages/agent-harness/src/harnesses/pi.ts +++ b/packages/agent-harness/src/harnesses/pi.ts @@ -19,15 +19,19 @@ export class PiHarness implements AgentHarness { readonly promptFlag = "-p"; /** No native plan/read-only enforcement documented for headless `pi`. */ readonly supportedModes = [] as const; + /** `--mode json` emits the session event stream as JSON lines. */ + readonly supportsStructuredOutput = true; /** * Build `pi` CLI flags for non-interactive (`-p`) execution. * * Supports `model` (`--model `; accepts a model ID, `provider/id`, - * or `:`). Pi's CLI does not currently expose turns or - * permission flags. + * or `:`) and `structuredOutput` (`--mode json`, which pairs + * with the `-p` prompt flag: print mode with JSON event output). Pi's CLI + * does not currently expose turns or permission flags. * - * @param options - Accepted for interface compatibility; only `model` is used. + * @param options - Accepted for interface compatibility; only `model` and + * `structuredOutput` are used. * @returns Args excluding the prompt (runner supplies `-p` via {@link promptFlag}). */ buildArgs(options: AgentRunOptions): string[] { @@ -38,6 +42,10 @@ export class PiHarness implements AgentHarness { args.push("--model", options.model); } + if (options.structuredOutput) { + args.push("--mode", "json"); + } + // Pi does not currently expose --max-turns or // --skip-permissions flags on its CLI in documented form. // If support is added in the future, uncomment the following: diff --git a/packages/agent-harness/src/harnesses/qwen.ts b/packages/agent-harness/src/harnesses/qwen.ts index c5e9fe4..9af2de1 100644 --- a/packages/agent-harness/src/harnesses/qwen.ts +++ b/packages/agent-harness/src/harnesses/qwen.ts @@ -18,11 +18,14 @@ export class QwenCodeHarness implements AgentHarness { readonly promptFlag = "-p"; /** No native plan/read-only enforcement documented for headless `qwen`. */ readonly supportedModes = [] as const; + /** `--output-format json` buffers messages and emits them as a JSON array. */ + readonly supportsStructuredOutput = true; /** * Build `qwen` CLI flags for headless (`-p`) execution. * - * @param options - Supports `skipPermissions` (`--yolo`) and `model`. + * @param options - Supports `skipPermissions` (`--yolo`), `model`, and + * `structuredOutput`. * @returns Args excluding the prompt (runner supplies `-p` via {@link promptFlag}). */ buildArgs(options: AgentRunOptions): string[] { @@ -37,6 +40,10 @@ export class QwenCodeHarness implements AgentHarness { args.push("--model", options.model); } + if (options.structuredOutput) { + args.push("--output-format", "json"); + } + // Qwen Code does not currently support --max-turns. // If it adds support in the future, uncomment the following: // if (options.maxTurns !== undefined) { diff --git a/packages/agent-harness/src/index.ts b/packages/agent-harness/src/index.ts index 580c07f..a959352 100644 --- a/packages/agent-harness/src/index.ts +++ b/packages/agent-harness/src/index.ts @@ -17,6 +17,7 @@ export type { AgentRunOptions, AgentRunResult, ResolvedHarness, + StructuredOutputResult, } from "./types.js"; // Run modes (plan / readonly) and capability checks @@ -136,3 +137,10 @@ export { // Open-question detection (agent blocked on user input) export { detectOpenQuestions, type OpenQuestionsResult } from "./detect-open-questions.js"; + +// Structured (JSON) output +export { + UnsupportedStructuredOutputError, + assertStructuredOutputSupported, + parseStructuredOutput, +} from "./structured-output.js"; diff --git a/packages/agent-harness/src/runners/bun.ts b/packages/agent-harness/src/runners/bun.ts index 1da7ef2..fa8703f 100644 --- a/packages/agent-harness/src/runners/bun.ts +++ b/packages/agent-harness/src/runners/bun.ts @@ -16,6 +16,7 @@ import { assertModeSupported } from "../modes.js"; import { buildPromptArgs } from "../prompt-args.js"; import { spawnReapable, reapTree } from "../process-reaper.js"; import { resolveExecutablePathWithRetry } from "../resolver.js"; +import { assertStructuredOutputSupported, parseStructuredOutput } from "../structured-output.js"; import type { AgentHarness, AgentRunOptions, AgentRunResult } from "../types.js"; /** @@ -28,8 +29,11 @@ import type { AgentHarness, AgentRunOptions, AgentRunResult } from "../types.js" * @param executablePath - Resolved path to the agent executable. * @param prompt - Task prompt passed to the agent. * @param options - Run options (turn limits, model, input method, etc.). - * @returns Captured stdout, stderr, and process exit code. + * @returns Captured stdout, stderr, and process exit code, plus the parsed + * structured payload when `options.structuredOutput` was requested. * @throws {UnsupportedAgentModeError} when `options.mode` is not supported. + * @throws {UnsupportedStructuredOutputError} when `options.structuredOutput` + * is requested from a harness whose CLI cannot emit JSON. */ export async function runAgentBun( harness: AgentHarness, @@ -38,6 +42,7 @@ export async function runAgentBun( options: AgentRunOptions = {}, ): Promise { assertModeSupported(harness, options.mode); + assertStructuredOutputSupported(harness, options); const inputMethod = options.inputMethod ?? "arg"; const { prompt: effectivePrompt, imageArgs } = preparePromptWithAttachments( @@ -106,10 +111,14 @@ export async function runAgentBun( throw new UsageLimitError(usageLimit.resetsAt); } - return { + const result: AgentRunResult = { stdout, stderr, exitCode, maxTurnsReached: detectMaxTurnsReached(stdout, stderr, harness.supportsMaxTurns === true), }; + if (options.structuredOutput) { + result.structured = parseStructuredOutput(stdout); + } + return result; } diff --git a/packages/agent-harness/src/runners/node.ts b/packages/agent-harness/src/runners/node.ts index 00796e7..d0c3dea 100644 --- a/packages/agent-harness/src/runners/node.ts +++ b/packages/agent-harness/src/runners/node.ts @@ -14,6 +14,7 @@ import { assertModeSupported } from "../modes.js"; import { buildPromptArgs } from "../prompt-args.js"; import { spawnReapable, reapTree } from "../process-reaper.js"; import { resolveExecutablePathWithRetry } from "../resolver.js"; +import { assertStructuredOutputSupported, parseStructuredOutput } from "../structured-output.js"; import type { AgentHarness, AgentRunOptions, AgentRunResult } from "../types.js"; export interface NodeRunnerOptions extends AgentRunOptions { @@ -45,9 +46,12 @@ function logAgent(message: string, options: NodeRunnerOptions): void { * @param executablePath - Resolved path to the agent executable. * @param prompt - Task prompt passed to the agent. * @param options - Node-specific run options (cwd, timeout, displayRealtime, etc.). - * @returns Captured stdout, stderr, and process exit code. + * @returns Captured stdout, stderr, and process exit code, plus the parsed + * structured payload when `options.structuredOutput` was requested. * @throws {Error} When the executable is not found (`ENOENT`) or the process times out. * @throws {UnsupportedAgentModeError} when `options.mode` is not supported. + * @throws {UnsupportedStructuredOutputError} when `options.structuredOutput` + * is requested from a harness whose CLI cannot emit JSON. */ export async function runAgentNode( harness: AgentHarness, @@ -56,6 +60,7 @@ export async function runAgentNode( options: NodeRunnerOptions = {}, ): Promise { assertModeSupported(harness, options.mode); + assertStructuredOutputSupported(harness, options); // Wait out any in-progress CLI auto-update swap before spawning (see // resolveExecutablePathWithRetry), so a transient `spawn ENOENT` doesn't @@ -192,12 +197,16 @@ export async function runAgentNode( } else if (usageLimit?.limited) { reject(new UsageLimitError(usageLimit.resetsAt)); } else { - resolve({ + const result: AgentRunResult = { stdout, stderr, exitCode: code ?? 1, maxTurnsReached: detectMaxTurnsReached(stdout, stderr, harness.supportsMaxTurns === true), - }); + }; + if (options.structuredOutput) { + result.structured = parseStructuredOutput(stdout); + } + resolve(result); } }); diff --git a/packages/agent-harness/src/structured-output.ts b/packages/agent-harness/src/structured-output.ts new file mode 100644 index 0000000..d6cd7b4 --- /dev/null +++ b/packages/agent-harness/src/structured-output.ts @@ -0,0 +1,167 @@ +/** + * Structured (JSON) output support for agent harnesses. + * + * Several wrapped CLIs can emit machine-readable JSON instead of styled + * transcript text (Claude Code `--output-format json`, Codex `--json`, + * Cline `--json`, Cursor `--output-format json`, ...). Requesting it is + * opt-in per run via {@link AgentRunOptions.structuredOutput}; harnesses + * declare support with the `AgentHarness.supportsStructuredOutput` + * capability flag, mirroring `supportsMaxTurns`. Runners fail closed when + * the option is requested from a harness without the capability instead of + * silently returning plain text. + * + * Parsing is deliberately tolerant of real-world CLI behavior (ANSI styling, + * interleaved log lines, NDJSON event streams) but fails closed on malformed + * or truncated payloads, so downstream automation never mistakes partial + * data for a complete result: {@link parseStructuredOutput} returns + * `{ ok: false, error }` and callers still have the raw `stdout`/`stderr`. + * + * Transcript detectors (usage limit, max turns) keep scanning the raw + * streams unchanged; JSON event lines can still match provider-diagnostic + * patterns (e.g. `{"error":{"code":"1302",...}}`). The plain-text default + * path is untouched. + */ + +import type { AgentHarness, AgentRunOptions, StructuredOutputResult } from "./types.js"; +import { stripAnsi } from "./output-lines.js"; + +/** + * Error thrown when a caller requests structured (JSON) output from a + * harness whose CLI cannot emit it. + */ +export class UnsupportedStructuredOutputError extends Error { + readonly harnessName: string; + + constructor(harness: AgentHarness) { + super( + `${harness.displayName} (${harness.name}) does not support structured (JSON) output. ` + + `Use a harness whose CLI can emit JSON, or omit structuredOutput for plain-text output.`, + ); + this.name = "UnsupportedStructuredOutputError"; + this.harnessName = harness.name; + } +} + +/** + * Fail closed when structured output is requested but the harness cannot + * provide it. No-op when the option is unset (plain-text default). + * + * @throws {UnsupportedStructuredOutputError} when the CLI has no JSON mode + */ +export function assertStructuredOutputSupported( + harness: AgentHarness, + options: AgentRunOptions, +): void { + if (options.structuredOutput === true && harness.supportsStructuredOutput !== true) { + throw new UnsupportedStructuredOutputError(harness); + } +} + +/** + * Recover the structured payload from captured agent stdout. + * + * Strategy, in order: + * + * 1. Parse the whole output as one JSON document (single-line or + * pretty-printed objects/arrays — Claude Code, Cursor, Grok, agy + * envelopes and Qwen message arrays). + * 2. Parse NDJSON event streams line by line (Codex, Opencode, Kilo, Cline, + * Kimi, Pi), ignoring interleaved non-JSON log lines. A stream where some + * JSON-looking lines parse and others do not is treated as truncated and + * fails closed rather than returning a partial payload. + * 3. Parse a `{...}` / `[...]` bracket span, covering pretty-printed + * documents embedded in banner/log noise (candidate open/close pairs, so + * banners like `[INFO] ...` don't break the span). + * + * @param stdout - Captured standard output (JSON mode goes to stdout) + * @returns Parsed payload, or a failure reason; never throws + */ +export function parseStructuredOutput(stdout: string): StructuredOutputResult { + const text = stripAnsi(stdout) + .replace(/^\uFEFF/, "") + .trim(); + + if (!text) { + return { ok: false, error: "agent produced no stdout to parse as structured output" }; + } + + // Whole-output document. + try { + return { ok: true, value: JSON.parse(text) }; + } catch { + // Fall through to line- and span-based recovery. + } + + const lines = text.split(/\r?\n/); + const events: unknown[] = []; + let malformedLine: number | undefined; + let malformedReason: string | undefined; + + for (let index = 0; index < lines.length; index += 1) { + const line = lines[index]?.trim() ?? ""; + // A line plausibly beginning a JSON value. A bare `[` only counts when + // followed by JSON-ish content, so log banners like `[INFO] ...` are not + // misread as malformed arrays. + const looksLikeJson = line.startsWith("{") || /^\[\s*[{"\d[\]-]/.test(line); + if (!line || !looksLikeJson) { + continue; + } + try { + events.push(JSON.parse(line)); + } catch (error) { + malformedLine ??= index + 1; + malformedReason ??= error instanceof Error ? error.message : String(error); + } + } + + if (malformedLine !== undefined) { + if (events.length > 0) { + // Mixed stream where some JSON lines parse and others do not: a + // truncated or corrupt NDJSON stream. Fail closed instead of returning + // a partial payload that downstream automation could mistake for a + // complete result. + return { + ok: false, + error: + `malformed JSON at line ${malformedLine} (${malformedReason}); ` + + `${events.length} complete JSON line(s) parsed before failing — ` + + `output is likely truncated`, + }; + } + // Every JSON-looking line is a fragment (pretty-printed document embedded + // in log noise); the span recovery below may still reconstruct it. + } else if (events.length > 0) { + // NDJSON event stream (Codex / Opencode / Kilo / Cline / Kimi / Pi), + // optionally interleaved with non-JSON log lines. + return { ok: true, value: events }; + } + + // Pretty-printed document embedded in log noise: parse a bracket span. + // Log banners can themselves contain brackets (`[INFO] ...`), so try each + // opening-bracket candidate against each closing-bracket candidate instead + // of assuming the outermost pair bounds the document. + const starts = [text.indexOf("{"), text.indexOf("[")].filter((pos) => pos >= 0); + starts.sort((a, b) => a - b); + const ends = [text.lastIndexOf("}"), text.lastIndexOf("]")].filter((pos) => pos >= 0); + ends.sort((a, b) => b - a); + for (const start of starts) { + for (const end of ends) { + if (end <= start) { + continue; + } + try { + return { ok: true, value: JSON.parse(text.slice(start, end + 1)) }; + } catch { + // Try the next candidate span. + } + } + } + + return { + ok: false, + error: + malformedLine !== undefined + ? `malformed JSON at line ${malformedLine} (${malformedReason})` + : "no JSON found in agent stdout", + }; +} diff --git a/packages/agent-harness/src/types.ts b/packages/agent-harness/src/types.ts index 3dfe144..73f603f 100644 --- a/packages/agent-harness/src/types.ts +++ b/packages/agent-harness/src/types.ts @@ -67,6 +67,21 @@ export interface AgentRunOptions { * be injected into the prompt automatically). */ imagePaths?: readonly string[]; + /** + * Request machine-readable structured (JSON) output from the agent CLI + * instead of styled transcript text. Opt-in: the plain-text default keeps + * transcript-based detectors (usage limit, max turns) working unchanged. + * + * Requires harness support (`AgentHarness.supportsStructuredOutput`); + * runners fail closed with {@link UnsupportedStructuredOutputError} + * otherwise instead of silently falling back to text. When supported, the + * harness emits the CLI's JSON flag (e.g. Claude Code `--output-format + * json`, Codex `--json`, Cline `--json`) and the run result carries the + * parsed payload in {@link AgentRunResult.structured}; raw `stdout` / + * `stderr` are still returned as today, and streaming callbacks keep + * receiving the raw chunks. + */ + structuredOutput?: boolean; /** Called with each stdout chunk as the agent runs (for live output streaming). */ onStdout?: (chunk: string) => void; /** Called with each stderr chunk as the agent runs (for live status updates). */ @@ -79,6 +94,31 @@ export interface AgentRunResult { exitCode: number; /** True when CLI output indicates the agent hit a max-turns limit. */ maxTurnsReached: boolean; + /** + * Parsed structured output when {@link AgentRunOptions.structuredOutput} + * was requested; absent otherwise (existing callers are unaffected). + */ + structured?: StructuredOutputResult; +} + +/** + * Outcome of parsing structured (JSON) agent output requested via + * {@link AgentRunOptions.structuredOutput}. + */ +export interface StructuredOutputResult { + /** True when a JSON payload was recovered from the agent's stdout. */ + ok: boolean; + /** + * Parsed payload. A single JSON document (object or array) is returned + * as-is; NDJSON event streams are returned as an array of parsed objects. + */ + value?: unknown; + /** + * Human-readable reason when `ok` is false (empty output, no JSON found, + * malformed or truncated payload). `stdout`/`stderr` remain available for + * caller-side diagnostics. + */ + error?: string; } /** @@ -103,6 +143,16 @@ export interface AgentHarness { * turn-limit error. */ readonly supportsMaxTurns?: boolean; + /** + * Whether this harness's CLI can emit machine-readable structured (JSON) + * output when {@link AgentRunOptions.structuredOutput} is requested + * (e.g. Claude Code `--output-format json`, Codex `--json`, Cline + * `--json`). Omitted / false means unsupported: runners fail closed with + * {@link UnsupportedStructuredOutputError} rather than silently returning + * plain text. Supporting harnesses emit the appropriate flag in + * {@link buildArgs}. + */ + readonly supportsStructuredOutput?: boolean; /** * Whether this harness's constrained modes still allow unrestricted * network and MCP tool use (web search, web fetch, MCP servers). diff --git a/packages/agent-harness/tests/harnesses.test.ts b/packages/agent-harness/tests/harnesses.test.ts index c080dd7..3862f96 100644 --- a/packages/agent-harness/tests/harnesses.test.ts +++ b/packages/agent-harness/tests/harnesses.test.ts @@ -24,6 +24,7 @@ describe("ClaudeCodeHarness", () => { expect(h.defaultPath).toBe("claude"); expect(h.promptFlag).toBe("-p"); expect(h.supportsMaxTurns).toBe(true); + expect(h.supportsStructuredOutput).toBe(true); }); test("buildArgs empty", () => { @@ -43,6 +44,13 @@ describe("ClaudeCodeHarness", () => { test("buildArgs partial options", () => { expect(h.buildArgs({ model: "sonnet" })).toEqual(["--model", "sonnet"]); }); + + test("buildArgs structured output", () => { + expect(h.buildArgs({ structuredOutput: true })).toEqual(["--output-format", "json"]); + expect( + h.buildArgs({ skipPermissions: true, model: "opus", structuredOutput: true }), + ).toEqual(["--dangerously-skip-permissions", "--model", "opus", "--output-format", "json"]); + }); }); describe("ClineHarness", () => { @@ -53,6 +61,7 @@ describe("ClineHarness", () => { expect(h.displayName).toBe("Cline"); expect(h.defaultPath).toBe("cline"); expect(h.promptFlag).toBeUndefined(); + expect(h.supportsStructuredOutput).toBe(true); }); test("buildArgs empty", () => { @@ -67,6 +76,15 @@ describe("ClineHarness", () => { "gpt-4", ]); }); + + test("buildArgs structured output", () => { + expect(h.buildArgs({ structuredOutput: true })).toEqual(["task", "--json"]); + expect(h.buildArgs({ skipPermissions: true, structuredOutput: true })).toEqual([ + "task", + "--yolo", + "--json", + ]); + }); }); describe("CodexHarness", () => { @@ -78,12 +96,21 @@ describe("CodexHarness", () => { expect(h.defaultPath).toBe("codex"); expect(h.promptFlag).toBeUndefined(); expect(h.supportsMaxTurns).toBeUndefined(); + expect(h.supportsStructuredOutput).toBe(true); }); test("buildArgs empty", () => { expect(h.buildArgs({})).toEqual(["exec", "--skip-git-repo-check"]); }); + test("buildArgs structured output", () => { + expect(h.buildArgs({ structuredOutput: true })).toEqual([ + "exec", + "--skip-git-repo-check", + "--json", + ]); + }); + test("buildArgs with all options", () => { expect(h.buildArgs({ skipPermissions: true, model: "gpt-4o" })).toEqual([ "exec", @@ -114,12 +141,17 @@ describe("CursorHarness", () => { expect(h.displayName).toBe("Cursor"); expect(h.defaultPath).toBe("cursor-agent"); expect(h.promptFlag).toBeUndefined(); + expect(h.supportsStructuredOutput).toBe(true); }); test("buildArgs empty", () => { expect(h.buildArgs({})).toEqual(["-p"]); }); + test("buildArgs structured output", () => { + expect(h.buildArgs({ structuredOutput: true })).toEqual(["-p", "--output-format", "json"]); + }); + test("buildArgs with all options", () => { expect(h.buildArgs({ skipPermissions: true, model: "claude-3" })).toEqual([ "-p", @@ -140,12 +172,17 @@ describe("AntigravityHarness", () => { expect(h.displayName).toBe("Antigravity CLI"); expect(h.defaultPath).toBe("agy"); expect(h.promptFlag).toBe("-p"); + expect(h.supportsStructuredOutput).toBe(true); }); test("buildArgs empty", () => { expect(h.buildArgs({})).toEqual([]); }); + test("buildArgs structured output", () => { + expect(h.buildArgs({ structuredOutput: true })).toEqual(["--output-format", "json"]); + }); + test("buildArgs with skipPermissions", () => { expect(h.buildArgs({ skipPermissions: true })).toEqual(["--dangerously-skip-permissions"]); }); @@ -192,12 +229,17 @@ describe("GooseHarness", () => { expect(h.displayName).toBe("Goose"); expect(h.defaultPath).toBe("goose"); expect(h.promptFlag).toBe("-t"); + expect(h.supportsStructuredOutput).toBe(true); }); test("buildArgs empty", () => { expect(h.buildArgs({})).toEqual(["run"]); }); + test("buildArgs structured output", () => { + expect(h.buildArgs({ structuredOutput: true })).toEqual(["run", "--output-format", "json"]); + }); + test("buildArgs with all options", () => { expect(h.buildArgs({ skipPermissions: true, model: "gpt-4" })).toEqual([ "run", @@ -216,12 +258,17 @@ describe("KiloCodeHarness", () => { expect(h.displayName).toBe("Kilo Code"); expect(h.defaultPath).toBe("kilo"); expect(h.promptFlag).toBeUndefined(); + expect(h.supportsStructuredOutput).toBe(true); }); test("buildArgs empty", () => { expect(h.buildArgs({})).toEqual(["run"]); }); + test("buildArgs structured output", () => { + expect(h.buildArgs({ structuredOutput: true })).toEqual(["run", "--format", "json"]); + }); + test("buildArgs with all options", () => { expect(h.buildArgs({ skipPermissions: true, model: "claude" })).toEqual([ "run", @@ -240,12 +287,21 @@ describe("KimiHarness", () => { expect(h.displayName).toBe("Kimi CLI"); expect(h.defaultPath).toBe("kimi"); expect(h.promptFlag).toBe("--prompt"); + expect(h.supportsStructuredOutput).toBe(true); }); test("buildArgs empty", () => { expect(h.buildArgs({})).toEqual(["--print"]); }); + test("buildArgs structured output (stream-json is Kimi's only structured format)", () => { + expect(h.buildArgs({ structuredOutput: true })).toEqual([ + "--print", + "--output-format", + "stream-json", + ]); + }); + test("buildArgs with all options", () => { expect(h.buildArgs({ skipPermissions: true, model: "kimi-k2" })).toEqual([ "--print", @@ -264,12 +320,24 @@ describe("OpencodeHarness", () => { expect(h.displayName).toBe("Opencode"); expect(h.defaultPath).toBe("opencode"); expect(h.promptFlag).toBeUndefined(); + expect(h.supportsStructuredOutput).toBe(true); }); test("buildArgs empty", () => { expect(h.buildArgs({})).toEqual(["run", "--print-logs", "--log-level", "ERROR"]); }); + test("buildArgs structured output", () => { + expect(h.buildArgs({ structuredOutput: true })).toEqual([ + "run", + "--print-logs", + "--log-level", + "ERROR", + "--format", + "json", + ]); + }); + test("buildArgs with all options", () => { expect(h.buildArgs({ skipPermissions: true, model: "gpt-4", workingDir: "/tmp/wt" })).toEqual([ "run", @@ -304,12 +372,23 @@ describe("PiHarness", () => { expect(h.displayName).toBe("Pi"); expect(h.defaultPath).toBe("pi"); expect(h.promptFlag).toBe("-p"); + expect(h.supportsStructuredOutput).toBe(true); }); test("buildArgs empty", () => { expect(h.buildArgs({})).toEqual([]); }); + test("buildArgs structured output (--mode json pairs with the -p prompt flag)", () => { + expect(h.buildArgs({ structuredOutput: true })).toEqual(["--mode", "json"]); + expect(h.buildArgs({ model: "claude-sonnet-4-6:high", structuredOutput: true })).toEqual([ + "--model", + "claude-sonnet-4-6:high", + "--mode", + "json", + ]); + }); + test("buildArgs with model", () => { expect(h.buildArgs({ model: "claude-sonnet-4-6:high" })).toEqual([ "--model", @@ -326,12 +405,17 @@ describe("QwenCodeHarness", () => { expect(h.displayName).toBe("Qwen Code"); expect(h.defaultPath).toBe("qwen"); expect(h.promptFlag).toBe("-p"); + expect(h.supportsStructuredOutput).toBe(true); }); test("buildArgs empty", () => { expect(h.buildArgs({})).toEqual([]); }); + test("buildArgs structured output", () => { + expect(h.buildArgs({ structuredOutput: true })).toEqual(["--output-format", "json"]); + }); + test("buildArgs with skipPermissions", () => { expect(h.buildArgs({ skipPermissions: true })).toEqual(["--yolo"]); }); @@ -357,12 +441,21 @@ describe("GrokHarness", () => { expect(h.displayName).toBe("Grok Build"); expect(h.defaultPath).toBe("grok"); expect(h.promptFlag).toBe("-p"); + expect(h.supportsStructuredOutput).toBe(true); }); test("buildArgs empty", () => { expect(h.buildArgs({})).toEqual(["--no-auto-update"]); }); + test("buildArgs structured output", () => { + expect(h.buildArgs({ structuredOutput: true })).toEqual([ + "--no-auto-update", + "--output-format", + "json", + ]); + }); + test("buildArgs with all options", () => { expect( h.buildArgs({ @@ -386,12 +479,17 @@ describe("DeepSeekHarness", () => { expect(h.displayName).toBe("Reasonix"); expect(h.defaultPath).toBe("reasonix"); expect(h.promptFlag).toBeUndefined(); + expect(h.supportsStructuredOutput).toBe(true); }); test("buildArgs empty", () => { expect(h.buildArgs({})).toEqual(["run"]); }); + test("buildArgs structured output", () => { + expect(h.buildArgs({ structuredOutput: true })).toEqual(["run", "--output-format", "json"]); + }); + test("buildArgs with model", () => { expect(h.buildArgs({ model: "deepseek-pro" })).toEqual(["run", "--model", "deepseek-pro"]); }); diff --git a/packages/agent-harness/tests/structured-output.test.ts b/packages/agent-harness/tests/structured-output.test.ts new file mode 100644 index 0000000..7f8aab9 --- /dev/null +++ b/packages/agent-harness/tests/structured-output.test.ts @@ -0,0 +1,362 @@ +import { describe, expect, test } from "bun:test"; + +import { listHarnesses } from "../src/registry.js"; +import { runAgentBun } from "../src/runners/bun.js"; +import { runAgentNode } from "../src/runners/node.js"; +import { + UnsupportedStructuredOutputError, + assertStructuredOutputSupported, + parseStructuredOutput, +} from "../src/structured-output.js"; +import type { AgentHarness, StructuredOutputResult } from "../src/types.js"; + +// Realistic CLI output fixtures (shapes verified against upstream docs). +const CLAUDE_ENVELOPE = + '{"type":"result","subtype":"success","is_error":false,"duration_ms":1234,"result":"All tests pass.","session_id":"0b5e1c2a"}'; +const CODEX_EVENTS = [ + '{"type":"thread.started","thread_id":"0199a213-81c0-7800-8aa1-bbab2a035a53"}', + '{"type":"item.completed","item":{"id":"item_3","type":"agent_message","text":"Repo contains docs, sdk, and examples directories."}}', + '{"type":"turn.completed","usage":{"input_tokens":24763,"output_tokens":122}}', +].join("\n"); + +function harnessWithoutCapability(): AgentHarness { + return { + name: "test-text-only", + displayName: "Text Only", + defaultPath: "/bin/sh", + buildArgs: () => [], + }; +} + +function emittingHarness(stdoutScript: string): AgentHarness { + return { + name: "test-json-emitter", + displayName: "JSON Emitter", + defaultPath: "/bin/sh", + supportsStructuredOutput: true, + buildArgs: () => ["-c", stdoutScript], + }; +} + +/** Assert a run carried a structured result and return it for value checks. */ +function expectStructuredPayload(result: { structured?: StructuredOutputResult }): { + ok: boolean; + value?: unknown; + error?: string; +} { + expect(result.structured).toBeDefined(); + return result.structured as StructuredOutputResult; +} + +describe("parseStructuredOutput", () => { + test("single-line JSON object (Claude Code / Cursor / Grok / agy envelope)", () => { + const parsed = parseStructuredOutput(CLAUDE_ENVELOPE); + expect(parsed.ok).toBe(true); + expect(parsed.error).toBeUndefined(); + const value = parsed.value as { type: string; result: string }; + expect(value.type).toBe("result"); + expect(value.result).toBe("All tests pass."); + }); + + test("pretty-printed JSON object", () => { + const parsed = parseStructuredOutput( + `{ + "status": "SUCCESS", + "response": "A git rebase rewrites history.", + "num_turns": 1 +}`, + ); + expect(parsed.ok).toBe(true); + expect((parsed.value as { status: string }).status).toBe("SUCCESS"); + }); + + test("JSON array (Qwen buffered messages)", () => { + const parsed = parseStructuredOutput( + '[{"type":"system","subtype":"session_start"},{"type":"result","is_error":false,"result":"Paris."}]', + ); + expect(parsed.ok).toBe(true); + expect((parsed.value as unknown[]).length).toBe(2); + }); + + test("NDJSON event stream (Codex / Opencode / Kilo / Cline / Kimi / Pi)", () => { + const parsed = parseStructuredOutput(CODEX_EVENTS); + expect(parsed.ok).toBe(true); + const events = parsed.value as Array<{ type: string }>; + expect(events.map((event) => event.type)).toEqual([ + "thread.started", + "item.completed", + "turn.completed", + ]); + }); + + test("NDJSON interleaved with non-JSON log lines", () => { + const output = [ + "[INFO] session starting", + '{"type":"say","text":"hello","ts":1760501486669}', + "reconnecting... 1/3", + '{"type":"say","text":"done","ts":1760501487999}', + ].join("\n"); + const parsed = parseStructuredOutput(output); + expect(parsed.ok).toBe(true); + expect((parsed.value as unknown[]).length).toBe(2); + }); + + test("pretty-printed document embedded in banner noise (span recovery)", () => { + const output = [ + "[INFO] launching headless run", + "{", + ' "conversation_id": "055a398f",', + ' "status": "SUCCESS",', + ' "response": "ok"', + "}", + "[INFO] exiting", + ].join("\n"); + const parsed = parseStructuredOutput(output); + expect(parsed.ok).toBe(true); + expect((parsed.value as { status: string }).status).toBe("SUCCESS"); + }); + + test("single-line JSON prefixed by log text", () => { + const parsed = parseStructuredOutput('run finished: {"ok":true,"exit":0}'); + expect(parsed.ok).toBe(true); + expect((parsed.value as { ok: boolean }).ok).toBe(true); + }); + + test("ANSI-styled JSON is stripped before parsing", () => { + const styled = `\u001b[32m{"type":"say","text":"styled"}\u001b[0m`; + const parsed = parseStructuredOutput(styled); + expect(parsed.ok).toBe(true); + expect((parsed.value as { text: string }).text).toBe("styled"); + }); + + test("empty stdout fails with a reason", () => { + const parsed = parseStructuredOutput(""); + expect(parsed.ok).toBe(false); + expect(parsed.error).toContain("no stdout"); + }); + + test("whitespace-only stdout fails with a reason", () => { + const parsed = parseStructuredOutput("\n \r\n"); + expect(parsed.ok).toBe(false); + expect(parsed.error).toContain("no stdout"); + }); + + test("plain-text output without JSON fails", () => { + const parsed = parseStructuredOutput("compiled 42 files\nall checks passed\n"); + expect(parsed.ok).toBe(false); + expect(parsed.error).toContain("no JSON"); + }); + + test("truncated single object fails closed", () => { + const parsed = parseStructuredOutput('{"type":"result","result":"partial va'); + expect(parsed.ok).toBe(false); + expect(parsed.value).toBeUndefined(); + expect(parsed.error).toBeDefined(); + }); + + test("truncated NDJSON stream fails closed (no partial payloads)", () => { + const parsed = parseStructuredOutput(`${CODEX_EVENTS}\n{"type":"turn.failed","err`); + expect(parsed.ok).toBe(false); + expect(parsed.value).toBeUndefined(); + expect(parsed.error).toContain("truncated"); + }); + + test("corrupt line between valid NDJSON lines fails closed", () => { + const parsed = parseStructuredOutput( + ['{"i":1}', '{"i":2 oops}', '{"i":3}'].join("\n"), + ); + expect(parsed.ok).toBe(false); + expect(parsed.error).toContain("line 2"); + }); +}); + +describe("structured output capability", () => { + test("all built-in harnesses declare support", () => { + for (const harness of listHarnesses()) { + expect(harness.supportsStructuredOutput).toBe(true); + } + }); + + test("fails closed for a harness without the capability", () => { + const harness = harnessWithoutCapability(); + expect(() => assertStructuredOutputSupported(harness, { structuredOutput: true })).toThrow( + UnsupportedStructuredOutputError, + ); + try { + assertStructuredOutputSupported(harness, { structuredOutput: true }); + } catch (error) { + const structuredError = error as UnsupportedStructuredOutputError; + expect(structuredError.harnessName).toBe("test-text-only"); + expect(structuredError.message).toContain("Text Only (test-text-only)"); + expect(structuredError.message).toContain("structuredOutput"); + } + }); + + test("no-op when structured output is not requested (plain-text default)", () => { + const harness = harnessWithoutCapability(); + expect(() => assertStructuredOutputSupported(harness, {})).not.toThrow(); + expect(() => assertStructuredOutputSupported(harness, { structuredOutput: false })).not.toThrow(); + }); +}); + +describe("runner integration", () => { + test( + "Node runner parses a JSON envelope when requested", + async () => { + const result = await runAgentNode( + emittingHarness(`printf '%s' '${CLAUDE_ENVELOPE}'`), + "/bin/sh", + "ignored", + { silent: true, structuredOutput: true }, + ); + expect(result.exitCode).toBe(0); + expect(result.stdout).toBe(CLAUDE_ENVELOPE); + expect(expectStructuredPayload(result).ok).toBe(true); + expect((expectStructuredPayload(result).value as { result: string }).result).toBe( + "All tests pass.", + ); + }, + 10_000, + ); + + test( + "Node runner parses NDJSON with interleaved logs when requested", + async () => { + const result = await runAgentNode( + emittingHarness("printf 'booting\\n{\"i\":1}\\n{\"i\":2}\\n'"), + "/bin/sh", + "ignored", + { silent: true, structuredOutput: true }, + ); + const structured = expectStructuredPayload(result); + expect(structured.ok).toBe(true); + expect(structured.value as unknown[]).toHaveLength(2); + }, + 10_000, + ); + + test( + "Bun runner keeps stderr streaming working alongside structured parsing", + async () => { + const chunks: string[] = []; + const result = await runAgentBun( + emittingHarness("printf 'status line' >&2; printf '%s' '{\"done\":true}'"), + "/bin/sh", + "ignored", + { silent: true, structuredOutput: true, onStderr: (chunk) => chunks.push(chunk) }, + ); + expect(chunks.join("")).toBe("status line"); + expect(result.stderr).toBe("status line"); + expect((expectStructuredPayload(result).value as { done: boolean }).done).toBe(true); + }, + 10_000, + ); + + test( + "Node runner omits structured field by default (backward compatible)", + async () => { + const result = await runAgentNode( + emittingHarness(`printf '%s' '${CLAUDE_ENVELOPE}'`), + "/bin/sh", + "ignored", + { silent: true }, + ); + expect(result.structured).toBeUndefined(); + expect(result.stdout).toBe(CLAUDE_ENVELOPE); + }, + 10_000, + ); + + test( + "Node runner reports malformed output without discarding stdout", + async () => { + const result = await runAgentNode( + emittingHarness("printf '%s' '{\"truncated\":'"), + "/bin/sh", + "ignored", + { silent: true, structuredOutput: true }, + ); + expect(result.exitCode).toBe(0); + expect(result.structured?.ok).toBe(false); + expect(result.structured?.error).toBeDefined(); + expect(result.stdout).toBe('{"truncated":'); + }, + 10_000, + ); + + test( + "Node runner fails closed before spawning when the harness lacks the capability", + async () => { + await expect( + runAgentNode(harnessWithoutCapability(), "/opt/definitely-missing/agent", "ignored", { + silent: true, + structuredOutput: true, + }), + ).rejects.toBeInstanceOf(UnsupportedStructuredOutputError); + }, + 10_000, + ); + + test( + "Bun runner parses a JSON envelope when requested", + async () => { + const result = await runAgentBun( + emittingHarness(`printf '%s' '${CLAUDE_ENVELOPE}'`), + "/bin/sh", + "ignored", + { silent: true, structuredOutput: true }, + ); + expect(result.exitCode).toBe(0); + expect(expectStructuredPayload(result).ok).toBe(true); + expect((expectStructuredPayload(result).value as { result: string }).result).toBe( + "All tests pass.", + ); + }, + 10_000, + ); + + test( + "Bun runner reports truncated output as a failure", + async () => { + const result = await runAgentBun( + // %b interprets the \n escapes inside the fixture argument. + emittingHarness(`printf '%b' '${CODEX_EVENTS}\\n{"type":"turn.failed","err'`), + "/bin/sh", + "ignored", + { silent: true, structuredOutput: true }, + ); + expect(result.structured?.ok).toBe(false); + expect(result.structured?.error).toContain("truncated"); + // Raw streams remain available for diagnostics. + expect(result.stdout).toContain("turn.completed"); + }, + 10_000, + ); + + test( + "Bun runner omits structured field by default", + async () => { + const result = await runAgentBun( + emittingHarness(`printf '%s' '${CLAUDE_ENVELOPE}'`), + "/bin/sh", + "ignored", + { silent: true }, + ); + expect(result.structured).toBeUndefined(); + }, + 10_000, + ); + + test( + "Bun runner fails closed before spawning when the harness lacks the capability", + async () => { + await expect( + runAgentBun(harnessWithoutCapability(), "/opt/definitely-missing/agent", "ignored", { + silent: true, + structuredOutput: true, + }), + ).rejects.toBeInstanceOf(UnsupportedStructuredOutputError); + }, + 10_000, + ); +});