Skip to content

feat: turn caps (--max-turns) and raw per-run results (--raw-out) - #35

Merged
queso merged 1 commit into
mainfrom
feat/max-turns-raw-out
Aug 29, 2026
Merged

queso merged 1 commit into
mainfrom
feat/max-turns-raw-out

Conversation

@queso

@queso queso commented Aug 29, 2026

Copy link
Copy Markdown
Owner

Fixes #33, fixes #34. Built for the AST repo-map spike, where the primary metric is exploration tokens and a turn-capped run must score as a localization failure rather than a cheap success.

Turn caps (#33)

  • maxTurns scenario field (per-case override allowed) + --max-turns on run, compare, and measure, passed through to claude-p as --max-turns. Completion runners are single-turn and ignore it.
  • A run that stops at the cap — result subtype: "error_max_turns", on any exit code — is returned as a measured outcome flagged exhaustedTurns instead of thrown as a crash, so one capped run no longer aborts a whole comparison.
  • The engine scores a capped run as a failed run without grading: partial output that happens to satisfy a grader must not count as a pass, and a judge grader would bill a model call for a result that cannot stand either way. Summary line reads hit the N-turn cap before finishing.
  • The effective cap joins the baseline cache key. stableStringify drops undefined entries, so scenarios without a cap keep their existing cache keys — no spurious cache busts.

Raw per-run results (#34)

  • --raw-out <dir> on compare and measure writes each run's full runner result JSON (claude-p: usage token categories, modelUsage, subtype) as <scenario>_<arm>_<n>.json (_measure_ for measure), sanitized the same way receipts sanitize scenario names.
  • Files are written the moment each run finishes, so an invocation that dies midway still leaves the completed runs' records.
  • Cache-served baseline arms ran in an earlier invocation and have no raw results; the progress output says so explicitly instead of leaving a silent gap.
  • Opt-in; summaries, receipts, and the ndjson report are unchanged.

Verification

  • bun run check green: 113 tests (6 new), typecheck clean.
  • Live smoke test against claude CLI 2.x on macOS: --max-turns is accepted in -p mode (registered in the binary despite being absent from --help), and a capped run returns exit-relevant JSON with subtype: "error_max_turns", result: null — handled by the null-tolerant normalizer.
  • New tests: arg construction with/without the cap; a fake-binary integration test for the non-zero-exit capped path; the crash path still throws; config parsing/validation at both levels plus CLI override precedence; engine test that a capped run fails even when its output would satisfy the grader; raw-out file contents for both compare arms and measure, and that nothing extra is written without the flag.

🤖 Generated with Claude Code

https://claude.ai/code/session_01EqktLpWgVX5orcGAzx7EK1

Two additions for token-level experiments (#33, #34):

maxTurns (scenario field, per-case override, --max-turns on run/compare/
measure) passes --max-turns to the claude-p runner. A run that stops at the
cap (subtype error_max_turns, any exit code) is returned as a measured
outcome flagged exhaustedTurns instead of thrown as a crash, and the engine
scores it as a failed run without grading — partial output must not pass by
accident, and a judge call on a truncated run is money spent on a result
that cannot stand. The cap joins the baseline cache key (undefined caps keep
their pre-existing keys).

--raw-out <dir> on compare and measure writes each run's full runner result
JSON (usage token categories, modelUsage, subtype) as
<scenario>_<arm>_<n>.json the moment the run finishes, so a partial
invocation still leaves its completed records. Summaries and reports keep
digesting to pass/cost; this is the escape hatch for analyses that need the
token categories rather than the cost blend. Cache-served baseline arms ran
in an earlier invocation and write nothing (noted in progress output).

Verified against claude CLI 2.x on macOS: --max-turns is accepted in -p mode
and a capped run returns subtype "error_max_turns" with full result JSON.

Fixes #33
Fixes #34

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EqktLpWgVX5orcGAzx7EK1
@queso
queso merged commit e8d0f2c into main Aug 29, 2026
1 of 2 checks passed
@queso
queso deleted the feat/max-turns-raw-out branch August 29, 2026 21:15
@queso queso mentioned this pull request Aug 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: persist per-run raw usage (token categories) in reports feat: claude-p runner should support a max-turns cap

1 participant