feat: per-turn transcripts (--transcript-out)

[queso]

Closes #39.

Implements Option B from the design comment on the issue: an opaque per-event sink on the runner, with the engine owning the file.

What this adds

--transcript-out <dir> on compare and measure. It runs claude -p --output-format stream-json --verbose and writes each run's NDJSON event stream to <scenario>_<arm>_<n>.stream.jsonl, next to the existing --raw-out JSON. Off by default; stream files are large.

promptdiff compare --scenario ./scenario.json --transcript-out ./transcripts

jq -r 'select(.type == "assistant") | .message.usage
       | [.input_tokens, .cache_read_input_tokens, .cache_creation_input_tokens] | @tsv' \
  transcripts/injected-context_proposed_1.stream.jsonl

Design

RunnerRunOptions gains one optional field, onStreamEvent?: (line: string) => void. The engine opens the file, names it from caseName/label/runNumber, and closes it in a finally. The runner emits lines and never learns what a scenario is, so buildClaudeArgs stays a pure function with no filesystem in its tests, and naming plus the partial-write policy stay in compare.ts next to writeRawResult.

Lines are written as they arrive. Buffering the verbose stream through new Response(proc.stdout).text() to parse it afterwards is the memory cost this flag exists to avoid, and it would lose every line a killed run had already printed.

Failure paths

The bulk of the work, as the design comment predicted. Three sites assumed stdout was one parseable JSON object; all three now work off a terminal result event decoded incrementally while reading:

• the success parse at claude-p.ts:129
• the error_max_turns recovery, so a turn-capped run still scores as a failure instead of aborting the compare
• describeClaudeFailure, so a budget abort still reads as a budget abort under NDJSON rather than a bare exit code

A truncated stream (timeout kill, SIGTERM) has no terminal event, so "last line is the result" is not assumed. It throws claude produced no terminal result event, and the lines already written stay on disk, including an unterminated final fragment.

Open question from the issue

Cache hits short-circuit the runner entirely. Resolved by warning, matching what --raw-out already does, rather than refusing --transcript-out together with --cache: baseline: cache hit — no transcript to write.

One addition beyond the design comment

RunnerCapabilities gains streamEvents. Without it, --transcript-out --runner openai would write empty files for a runner that cannot stream. Now it fails before any paid run, the same way image and sandbox-tool demands already do. That is why the capability literals in the test mocks changed.

Note for reviewers: this makes streamEvents a required field on RunnerCapabilities. The package ships a bin and no main/exports, so there is no library API contract this breaks, but anyone implementing Runner against src/ directly would need the field.

Verification

• 127 tests pass, tsc --noEmit clean.

• New tests: stream-json args only when a sink is attached; every event reaching the sink with per-turn usage intact; a truncated stream keeping its lines and throwing; turn caps and budget aborts decoded out of NDJSON; engine-level capture for both compare arms and measure; the partial file kept when a run throws; the refusal on a non-streaming runner before any run; the cache-hit warning.

• Ran the real CLI end to end against a fake claude binary. Files land, the args carry stream-json --verbose only when capture is on (--no-session-persistence throughout), the openai runner is refused with exit 1, and the README's jq recipe was run against actual captured output:

  14	0	8100
  9	8100	0
  

  Turn 1 pays the cache write, turn 2 reads it. That is what the aggregate could not answer.

🤖 Generated with Claude Code

[github-actions]

Nitpick review — comment

The change extends the compare engine and Claude runner with transcript streaming and result handling, and is well covered by tests. Two real issues remain: a sandbox directory can be orphaned when opening the transcript throws before the cleanup finally block is entered, and a killed run can be scored against a mid-stream result-shaped event rather than the true terminal outcome. Neither is prod-breaking, but both are worth fixing before merge.

2 inline comment(s).

[queso]

One bug, inline below. Everything else checked out: incremental NDJSON parsing handles chunk boundaries and the unterminated fragment, the three stdout-parse sites all route through the decoded terminal event, --verbose only lands when a sink is attached, and the cache key is untouched so capturing and plain runs share entries.

[github-actions]

Nitpick review — comment

The change adds stream-event transcript capture to the comparison engine, with a new TranscriptSink that writes NDJSON transcripts and a capability guard rejecting runners that can't stream. The sink's writeSync call discards the return value, so a short write (e.g., full disk) would silently truncate a transcript line while the run reports success — worth hardening since these files are the evidence trail for failed runs. Otherwise the diff looks sound; non-blocking.

1 inline comment(s).

[github-actions]

Nitpick review — comment

The change substantially extends the comparison engine and Claude runner with solid test coverage overall. Two non-blocking items survive: the stream parser's last-wins handling of multiple result events is untested on the clean-exit success path, so a regression there would silently score a mid-stream event as the outcome; and the NDJSON line accumulator rescans the whole buffer per chunk, which is quadratic for a single very large line. Neither blocks the merge.

1 inline comment(s).

1 previously-acknowledged finding(s) not re-posted (resolved review threads).

[queso]

On the second item from the latest review summary, which had no inline thread:

the NDJSON line accumulator rescans the whole buffer per chunk, which is quadratic for a single very large line

Fixed in 09811bf. consumeStreamJson now starts each chunk's search at the first unscanned character. The invariant that makes this safe: the inner loop exits only when pending holds no newline, so everything before pending.length at the top of a chunk iteration is already proven newline-free. After a line is sliced off, the remainder is unscanned and the inner search correctly restarts at 0.

Test added: a 2,000,000-character event line, which arrives across 11 stdout chunks on this machine (measured), is read whole. A short line follows the giant one, because a stale scan offset would merge two newlines landing in one chunk rather than dropping them, and the trailing line is what exposes that. Being precise about what it proves: it pins the chunk-boundary arithmetic, not the speedup, and it passes against the old implementation too, which was correct but slow.

[github-actions]

Nitpick review — comment

The change adds a comparison engine with LLM runners, transcript capture, and CLI wiring, with solid test coverage of the main paths. Two non-blocking robustness gaps remain: the claude-p runner's non-zero-exit path trusts a possibly mid-stream result event when scoring a turn-cap failure (the success path guards against this, the failure path doesn't), and the transcript sink's short-write retry loop could spin forever if writeSync ever returns 0. Both are narrow edge cases worth tightening, but neither blocks the merge.

2 inline comment(s).

[github-actions]

Nitpick review — comment

The change adds streaming NDJSON handling, transcript sinks, and cache-aware compare logic to the eval harness, with solid accompanying tests. One real inconsistency survives: the runner's success path scores any result-shaped event without the terminal-event check the failure path applies, so a mid-stream result event could be recorded as the run's outcome. Worth fixing before merge; nothing else rises to a blocking concern.

1 inline comment(s).

[github-actions]

Nitpick review — comment

The change adds transcript capture to the compare engine (a byte-offset NDJSON sink with short-write retry and zero-byte guards), plumbs a streamEvents capability flag through the runners, and integrates cache-served baseline arms that skip transcript writes. The logic is sound overall; the surviving items are a fail-fast validation that can over-reject a fully cache-served baseline arm, and two test-hardening gaps around the new sink and the cache no-transcript guarantee. Nothing blocks the merge.

2 inline comment(s).

1 previously-acknowledged finding(s) not re-posted (resolved review threads).

[github-actions]

Nitpick review — comment

The change adds transcript capture, cache-hit short-circuits, and a streaming line parser to the eval CLI, with the new logic well-structured and defensively written (short-write loop, zero-byte-write guard, unscanned-offset line splitting). One non-blocking note survives: the per-chunk string concatenation in the Claude runner's line buffer is quadratic for very long NDJSON lines, which is worth addressing since large tool-result payloads are the exact path this code handles. Nothing blocking; the change is safe to merge with that follow-up.

1 inline comment(s).

[github-actions]

Nitpick review — approve

The change extends the eval CLI's compare engine, runners, and CLI entrypoint with substantial new test coverage, and the documented transcript-capture behavior is consistent with the sanitization patterns already used elsewhere in the codebase. No concrete defects survive review; the diff is clean and safe to merge.

0 inline comment(s).

Verdict was approve, posted as a comment — this repo does not allow GitHub Actions to approve PRs.