feat: turn caps (--max-turns) and raw per-run results (--raw-out) - #35
Merged
Merged
Conversation
Two additions for token-level experiments (#33, #34): maxTurns (scenario field, per-case override, --max-turns on run/compare/ measure) passes --max-turns to the claude-p runner. A run that stops at the cap (subtype error_max_turns, any exit code) is returned as a measured outcome flagged exhaustedTurns instead of thrown as a crash, and the engine scores it as a failed run without grading — partial output must not pass by accident, and a judge call on a truncated run is money spent on a result that cannot stand. The cap joins the baseline cache key (undefined caps keep their pre-existing keys). --raw-out <dir> on compare and measure writes each run's full runner result JSON (usage token categories, modelUsage, subtype) as <scenario>_<arm>_<n>.json the moment the run finishes, so a partial invocation still leaves its completed records. Summaries and reports keep digesting to pass/cost; this is the escape hatch for analyses that need the token categories rather than the cost blend. Cache-served baseline arms ran in an earlier invocation and write nothing (noted in progress output). Verified against claude CLI 2.x on macOS: --max-turns is accepted in -p mode and a capped run returns subtype "error_max_turns" with full result JSON. Fixes #33 Fixes #34 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqktLpWgVX5orcGAzx7EK1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #33, fixes #34. Built for the AST repo-map spike, where the primary metric is exploration tokens and a turn-capped run must score as a localization failure rather than a cheap success.
Turn caps (#33)
maxTurnsscenario field (per-case override allowed) +--max-turnsonrun,compare, andmeasure, passed through to claude-p as--max-turns. Completion runners are single-turn and ignore it.subtype: "error_max_turns", on any exit code — is returned as a measured outcome flaggedexhaustedTurnsinstead of thrown as a crash, so one capped run no longer aborts a whole comparison.hit the N-turn cap before finishing.stableStringifydrops undefined entries, so scenarios without a cap keep their existing cache keys — no spurious cache busts.Raw per-run results (#34)
--raw-out <dir>oncompareandmeasurewrites each run's full runner result JSON (claude-p:usagetoken categories,modelUsage,subtype) as<scenario>_<arm>_<n>.json(_measure_for measure), sanitized the same way receipts sanitize scenario names.Verification
bun run checkgreen: 113 tests (6 new), typecheck clean.--max-turnsis accepted in-pmode (registered in the binary despite being absent from--help), and a capped run returns exit-relevant JSON withsubtype: "error_max_turns",result: null— handled by the null-tolerant normalizer.🤖 Generated with Claude Code
https://claude.ai/code/session_01EqktLpWgVX5orcGAzx7EK1