Conversation
Reshape the optimization loop into a resumable training epoch. `acumen epoch` benches the current arm on the training split, distils each task's runs into a persistent per-task wiki (observations.md + hypothesis.md, one [version][model] block per epoch from noskill up), creates or improves the skill from that wiki, then benches the new version on the held-out split. - New `wiki.py`: cumulative, resume-safe per-task notes (one agent per task, in parallel); terse-by-design prompt; filtered-source isolation. - Rework `improve.py`: reads the wiki + parent skill + staged train transcripts + filtered package source; creates v1 when no version exists, improves after. - New `epoch.py` + `acumen epoch`; add `acumen wiki`; retire `acumen draft`. - Rename the held-out split `test` -> `valid` (reserving `test` for a future train/valid/test split) across paths, tasks, report, check, review, taskgen, scaffold, prompts, and tests. - bench: free codex sandbox preflight probe so a namespace/sandbox failure is reported up front instead of per paid run; raise the auth-probe budget cap (PREFLIGHT_MAX_USD) so premium models can pass preflight. - Rewrite the README (minimal) and the shipped acumen self-skill for the new loop. Validated end-to-end on a toy package across six models; the skill lifted the weakest model's held-out pass rate without regressing the others. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #13 +/- ##
==========================================
+ Coverage 78.44% 79.02% +0.58%
==========================================
Files 30 34 +4
Lines 5336 6218 +882
==========================================
+ Hits 4186 4914 +728
- Misses 1150 1304 +154
🚀 New features to boost your workflow:
|
…age sum The transcript/trajectory footer for Claude runs summed the session file's per-message usage. Each Claude turn re-sends the whole conversation, so every turn's usage already counts the cached prefix; summing across turns overcounts badly (a real run showed input/cached/output of 22/230109/2999 and a priced cost of $0.53 vs the $0.32 the run was actually billed). result.json was already correct — it records the authoritative ResultMessage.usage, whose priced cost matches Claude's own total_cost_usd. Thread that authoritative usage into the trajectory so the footer matches result.json: build_trajectory / from_claude_transcript / from_claude_records take a `usage` override, and the runner and LiveLog.finalize pass ResultMessage.usage. The per-message sum remains only as a fallback for rendering a bare transcript with no result. Also fix a shadowing bug where the loop's per-message `usage` clobbered the new parameter. Codex is unaffected: it reports usage once per turn.completed as a single total, which the mapper takes directly (never summed). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`acumen fit` drives the epoch loop like training a model: it runs `acumen epoch` back to back, tracks held-out validation mean success, and stops on `--patience` (default 2, no strict improvement over the best-so-far) or the `--max-epochs` hard cap (default 10); `--epochs N` runs exactly N and disables early stopping. - New `training.py`: build the per-epoch training curve from the runs tree (epoch N's train arm = v(N-1)/noskill, valid arm = vN), write `training.csv` (mean success and mean cost per run, overall and per model, train and valid), and the pure early-stop rule (`patience_exhausted`, `epochs_since_best`, `best_version`). Rebuilt from disk each epoch, so `fit` is resumable. - Factor `_run_one_epoch` + `_prepare_pass` out of `_cmd_epoch` so `epoch` and `fit` share one epoch implementation; `fit` prepares the target/rates once. - Each epoch prints a tqdm-style line with train/valid success, best/patience. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
acumen bundled its own skill (guidance for using acumen) under `_skills/`, exposed via the `acumen-install-skills` console script. It had drifted out of date with the epoch/wiki/fit workflow, so drop it: delete `src/acumen/_skills/`, its `[project.scripts]` entry, and the README install note. Unrelated: `acumen ship` — which packages a *target* package's skill into a `<dist>-install-skills` script (`SHIP_INSTALL_TEMPLATE`, ship.py) — is unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
codex exec prints this to stderr whenever its stdin is not a TTY (acumen wires it to /dev/null), then reads immediate EOF. It is harmless but reads as a confusing prompt during a run, so drop it in the stderr drain. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`acumen epoch` and `acumen fit` now show one in-place progress bar per phase (bench train, wiki, an "inferring skill" spinner, bench valid), each filling per finished run with a running mean success rate and cumulative cost, in place of the per-run scrolling logs. - Bar unit is one matrix cell (model x task x rep); success = passed/done. - Bars by default; --verbose restores the full scrolling logs. - Output auto-detects: live \r bars on a TTY with --stream off, else a plain throttled-line fallback so files/CI stay readable. Genuine harness failures (provider-exhausted / sandbox-blocked) still surface to stderr. - bench/wiki/improve standalone commands are unchanged. update_wiki gains an optional on_plan(total) callback so the wiki bar has an accurate denominator (recorded tasks are skipped without a callback). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_cmd_fit called build_training_rows(runs, cfg, tasks) but the function takes two args and ignores tasks, so every fit run crashed while writing the training curve — after a full epoch had already been spent. The unit tests call it with two args, so the suite stayed green. Drop the stray argument at both call sites and add a regression test that drives the post-epoch path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codex logs its own tracing to stderr (rollout writes after a session ends, sandbox-denied commands under approval_policy=never, malformed apply_patch attempts). None signal an acumen fault and all are kept in the run transcript, but each line is unique so StderrFilter never dedupes them — they flood the console and corrupt the \r progress bars. Drop lines matching the Codex tracing shape in _drain_stderr, keeping them in the noise sink so _sandbox_failure still detects real bwrap/landlock errors (which are bare and still print). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The phase bar mixed conventions without saying so: success was a running mean pass-rate while cost was a cumulative sum. Spell both out — "success N% (mean)" and "$X total" — so the numbers are unambiguous. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The improve step reported cost and turns but not how long it took, unlike the phase bars. Time the call and include it — "(2m41s, $5.24, 79 turns)" in bar mode, and "over N turns in 2m41s" in verbose. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The venv every benchmark run reads is scrubbed of any skill the target ships, but the upstream repo is not. An agent that clones or re-installs the target from source smuggles that shipped skill back into the run — observed in a noskill baseline that git-cloned the target — contaminating the very baseline the benchmark is measured against. Deny the fetch at tool-call time on both providers: - prompts: HARNESS_PREAMBLE tells the agent to use only the installed package and not to clone/download/pip-install its source repo or dist. Arm-neutral wording, so noskill and skill prompts stay identical. - scrub.find_source_fetch: pure, testable detector for git clone/fetch, gh repo clone, git+ installs, curl/wget of the repo, source archives, and pip/uv install|download of the package. scrub.make_source_guard wraps it as a Claude PreToolUse hook. - agents/runner/env: new AgentOptions.block_repo/block_pkg, set for benchmark runs from the target (remote only). _claude_hooks adds the guard alongside containment; the generated Codex guard runs the same check (importing acumen.scrub, made import-safe for editable installs). It blocks the fetch, not reads of skill files: the skill arm legitimately reads its own installed skill, and with no clone there is no external skill tree to read. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A perfect validation epoch leaves nothing to gain, but fit kept running more epochs (up to the cap), wasting time and budget. Add a stop on a 100% validation mean success: finish that epoch and don't start another. - training.is_perfect(score): pure predicate mirroring patience_exhausted, true only for an all-pass mean (1.0); nan/None (no runs) is not perfect. - _cmd_fit: check it right after the epoch bar, before the patience test. Unconditional — it fires even under a fixed --epochs N, since a perfect score overrides an explicit count. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Transcripts now render each tool call by kind on the harness-neutral model (Claude Code and Codex alike): commands as shell blocks, file edits as red/green split diffs, writes/reads/web tools legibly, and anything else as a key/value list instead of a raw JSON dump. The report renders skill rationale/feedback markdown. The split-diff renderer and the warm-neutral palette move to new shared modules (acumen.markdown, acumen.htmldiff, acumen.theme) so both pages read as one product without the transcript importing the report. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The "Did it help when it actually loaded?" section rendered a wide table; show it as a dot grid instead — one dot per (model, skill version), colour for the delta success rate vs. baseline among loaded runs (diverging green to red) and area for the load rate, with a single pooled "all models" row. Rewrite the section's paragraph to be shorter and plainer. Also fix a CSS specificity bug in the runs table: `tr.fail td` scored below the `tbody tr:nth-child(even) td` zebra rule, so a failed run on an even row kept the stripe and never highlighted. Match the zebra selector's specificity so the fail tint wins on every row. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Plot success rate across skill versions (no skill, then v1, v2, ...) as a line chart: one train and one valid line per model, coloured by model with the split shown as linestyle, plus a pooled all-models mean line drawn on top so it stays visible through overlap. Each version is measured on both splits; a version with no runs on a split leaves an honest gap. The section sits in the overview with a TOC entry and is omitted for a baseline-only report. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The "did it help when it loaded" delta compared success on the loaded runs against the baseline's whole task mix, so a skill that loaded only on easy tasks looked good from selection alone. Pair each loaded run with the baseline on its own (task, model) and average the per-run differences, so the delta is a real effect that does not depend on which tasks happened to load. Pairing also makes the model-matched pooled scope redundant with the all-models row (a model with no loaded runs is already left out of the baseline), so drop it and the _LOADED_MODELS sentinel. Reword the section's caveat accordingly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Introduce a third, required task split "test": a true holdout the loop never
sees. Training learns from train and the best version is picked on valid, so
test is the skill's first contact with these tasks. It is benched once at the
end of `acumen fit`, on the single best kept version and the baseline, at the
same replicate count, and shown in the report as a standalone section (a
cost-vs-success scatter plus a best-vs-noskill significance table).
- paths: add test to Split/SPLITS; add BENCH_SPLITS so normal epochs and
bench stay train/valid and test runs only when asked.
- tasks/taskgen/prompts/scaffold: test is required per task, generated by
`acumen tasks`, serialized, scaffolded, and checked like the others.
- cli: _run_test_phase benches best + noskill on test after the fit loop,
idempotent via pending().
- report: split-parameterize _cluster_totals/skill_tests/tradeoff_figure and
render the held-out test section when test runs exist.
Breaking: existing tasks.yaml must gain a test: block (and {id}-test.py
reproducers) or regenerate via `acumen tasks`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A fit's purpose is the comparison, so `acumen fit` now writes report.html (and report.csv) to the working directory when it finishes, instead of telling the user to run `acumen report` themselves. Report generation is a summary of what already ran, so a failure there is noted but never fails the fit. Also update the `acumen tasks` next-step hint to recommend `acumen fit` rather than `acumen epoch`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Reshapes acumen's optimization loop from the manual
draft→bench→improvecycle into a single, resumable training epoch.acumen epochruns one epoch end to end:noskillbaseline onvalid).wiki/<task>/observations.md+hypothesis.md(one terse[version][model]block per epoch, fromnoskillup).noskillwiki + filtered source) or improve the latest, reading the wiki + parent skill + staged train transcripts.It resolves state from disk, so a crashed epoch resumes on the right step and a finished one starts the next.
Changes
wiki.py— cumulative, resume-safe per-task knowledge base; parallel agents; brevity-enforced prompt; filtered-source +discover_skills=Falseisolation;.armsmarker for idempotency.improve.py— wiki-driven, create-or-improve, filtered package source; keeps the held-out-split guard hook (renamedfind_valid_access/make_valid_guard).epoch.py+acumen epoch; addacumen wiki; retireacumen draft.test→valid(reservingtestfor a future train/valid/test split) across paths, tasks, report, check, review, taskgen, scaffold, prompts, and tests.codex sandboxpreflight probe so a namespace/sandbox failure is reported up front instead of once per paid run; raise the auth-probe budget cap (PREFLIGHT_MAX_USD) so premium models can pass preflight (pre-existing bug that also blockedacumen benchwith Opus)._skills/data/**) updated to the new loop.Validation
sensorloom) across six models (opus/sonnet/haiku + sol/terra/luna) and six generated tasks: the created skill lifted the weakest model's held-out pass rate (haiku 4/6 → 6/6 onvalid) without regressing the others.🤖 Generated with Claude Code