diff --git a/README.md b/README.md index 9dac23f..b2788b4 100644 --- a/README.md +++ b/README.md @@ -10,263 +10,52 @@ [badge-tests]: https://img.shields.io/github/actions/workflow/status/scverse/acumen/test.yaml?branch=main [badge-docs]: https://app.readthedocs.org/projects/acumen/badge/ -Agentic skills, tool instructions writen in plain text, allow agents to use tools more succesfuly and efficient. -However most python packages do not ship skills with them because developers have no easy way to build and benchmark skills for their tools. -Acumen closes this gap. Point it at a Python package and a few evaluation tasks, and it drafts a skill, benchmarks it and improves it across a train/test split so the gains are generalizable. +An agentic skill is a short, plain-text set of instructions that helps a coding agent use a tool +correctly. Most Python packages ship none — maintainers have no easy way to write one, or to tell +whether it actually helps. acumen closes that loop: point it at a package and a few tasks, and it +learns a skill in **training epochs**, each one benchmarking the current skill, distilling what went +right and wrong into a knowledge wiki, and rewriting the skill — always measured against a no-skill +baseline on a held-out split, so the gains generalize instead of memorizing answers. +## Installation -Many good tools are unusable by coding agents because their maintainers have no way to write -a skill for them — or, having written one, no way to tell whether it helps. acumen closes -that loop: point it at a Python package and a few tasks, and it drafts a skill, benchmarks it -against a no-skill baseline, and improves it across a train/test split so the gains are real -generalization, not memorized answers. - -- **`acumen check`** — rerun the script behind each task's answer, then have an agent judge - whether the prompt actually asks for what that script and answer produce. Both before a - benchmark pass pays to find out. -- **`acumen draft`** — write `skills/v1` from the package's own source. -- **`acumen bench`** — score a skill against a no-skill baseline, in a scrubbed sandbox where - the skill is the only difference between arms. Agent guidance the target ships itself is - removed from the venv first, so the baseline really is skill-free. -- **`acumen improve`** — refine the skill from its train results, then benchmark again. -- **`acumen report`** — aggregate every run into one self-contained `report.html`: success - rate per version, train vs. test. Bars are coloured by model, with a grey bar pooling all - of them; pass `--palette claude-opus-5=#3b7ea1` (repeatable) to recolour any of them. +Python 3.12+ is required. Install the backend you actually run — both are optional, and either +alone is a complete install: -You decide when to stop. Every version is benchmarked on both splits, and only train results -reach the improver — so a widening train/test gap is a visible sign a skill is overfitting -rather than genuinely helping. +| you run | install | also needs | +|---|---|---| +| Claude only | `pip install acumen[claude]` | an Anthropic key or a `claude` login | +| Codex only | `pip install acumen` | the `codex` CLI on `PATH`, plus a Codex login or key | +| both | `pip install acumen[all]` | both of the above | ## Quickstart ```bash -# 1. Scaffold a starter config.yaml and tasks.yaml -acumen init - -# 2. Fill in config.yaml (repo). Write tasks.yaml by hand, or generate it: -acumen tasks # mine the package for real analyses -> tasks.yaml + tasks/ -acumen check # is the ground truth right, and does each prompt ask for it? +acumen init # scaffold config.yaml + tasks.yaml +# edit config.yaml: point `repo` at your package (a GitHub URL or local path) -# 3. Then run the loop: -acumen bench --no-skill # the baseline arm -acumen draft # generate skills/v1 from the package source, or write by hand -acumen bench --skill v1 # benchmark the skill against the baseline -acumen improve # generate skills/v2 from v1's train results, or write by hand -acumen bench --skill v2 -acumen bench # or: every arm at once (baseline + each skills/vN) -acumen report # aggregate every run into report.html +acumen tasks # mine the package for real tasks + ground-truth reproducers +acumen check # verify each task's answer reproduces before you spend -# 4. Once a version proves out, ship it into the package itself: -acumen ship --skill v2 # add a -install-skills console script (PR, or local edit) -``` - -`acumen ship` packages the chosen skill version into the target: the package gains a -`-install-skills` command that installs the skill into the skills directory of whichever -agent the user names — `--agent {claude,codex,agents,claude-science}`, or an explicit `--dest` — -so the package's own users get the guidance with one command, wherever they run their agent. The -same bundle installs verbatim into every framework. - -## Checking the ground truth - -A task is only worth benchmarking if its recorded answer is actually correct. A wrong answer makes -every model fail that task: real money spent, and the failure reads in the report as the model's -fault rather than the task's. `acumen check` catches the two ways that happens. +acumen epoch # one training round: bench train → wiki → create/improve skill → bench held-out +acumen epoch # run again for the next round (v2, v3, …) -**Does the answer still come out of running the code?** Each task keeps a **reproducer** at -`tasks/-.py`, a self-contained script that redoes the analysis in the target venv and -writes its answer to `answer.md` — the same contract a benchmark run has, graded the same way. -`acumen tasks` writes them as it generates the tasks; `acumen check` reruns them: - -```bash -acumen check # every task, both splits -acumen check --task bulk --split train # one cell, while you fix it -acumen check --jobs 8 --timeout 600 # or: --keep to inspect what a script wrote +acumen report # aggregate every run into a self-contained report.html +acumen ship --skill v2 # ship the chosen skill into your package ``` -You get one row per task and split — reproduced, wrong answer, script error, timed out, or no -script at all — then the summary statistics: how much of the task set has a reproducer, how much -of it reproduces, and how many tasks reproduce on both splits. Before running anything it checks -that the package imports in the venv at all, since that one failure would otherwise be reported -once per task. - -**Does the prompt actually ask for what the script and answer produce?** Reproducing an answer -proves the code and the answer agree. It says nothing about the prompt, and a prompt describing -something else fails every agent that reads it correctly. A real example: - -> Find the 3 most deactivated PROGENy pathways in Megakaryocytes … Report only the pathway names -> sorted by score **(ascending)**. - -The script sorted descending, the recorded answer was descending, the reproducer check said `ok` — -and every agent that honoured the prompt produced the reverse order and was graded wrong. So after -the scripts run, one agent reads every split's prompt, recorded answer and reproducer together and -adds a `review` column of `ok` or `mismatch`, with one line naming the contradiction and one naming -the fix. It never edits `tasks.yaml`: which of the three artifacts to repair is your call. - -``` -task split status review detail -scell train ok ok MAPK;Estrogen;TGFb -scell test ok mismatch Trail;JAK-STAT;Estrogen - -1 split the review flagged - scell/test prompt says ascending; script and answer are descending - fix: say descending in the prompt, or reverse the answer -``` - -The review is on by default and picks its model from `meta_model`; it is the one phase that costs -money, so `acumen check --no-review` runs the reproducers alone and spends nothing — what you want -while iterating on a script. `check` takes the same `--auth`, `--stream` and `--log-dir` flags as -the other agentic commands, and `--max-turns`/`--max-usd` bound the reviewer. - -Either phase failing exits non-zero, so `acumen check` works as a gate before a pass. - -A task that needs no code to answer (a licence, a supported species, a documented default) sets -`needs_script: false`; its reproducer column reads `n/a` rather than counting as a gap, and its -prompt and answer are still reviewed. - -The reproducers hold the answers to the held-out test split, so nothing must feed them to an agent -under test. They are safe where they are: `bench`, `draft`, and `improve` confine their agents to -explicit read roots that never include your project directory, and the reviewer reads a staged copy -with no path back to `tasks.yaml`. - -`acumen tasks`, `acumen draft`, and `acumen improve` each accept `--feedback "…"` to steer the -agent with context it can't infer — which functionality to skip when generating tasks, what a -skill should emphasise or fix. The guidance is added to the prompt without overriding the -train/test isolation, and for `draft`/`improve` it is recorded in the version's `meta.json` and -shown in the report. (Don't paste held-out test answers into `improve` feedback — that would -defeat the split.) - -Claude and Codex can run side by side. Put both model families in `models` to compare them -in one matrix; model IDs beginning with `claude` use Claude Code, while `gpt-*`, `o1`, -`o3`, `o4`, and `codex-*` use Codex: - -```yaml -models: - - claude-opus-5 - - claude-sonnet-5 - - claude-haiku-4-5-20251001 - - gpt-5.6-sol - - gpt-5.6-terra - - gpt-5.6-luna -``` - -This spans each provider's quality/cost range; it is not a claim that the tiers are -one-to-one equivalents. - -Neither backend is required. Claude is an optional dependency and Codex is an external CLI, -so install only the one you run — `pip install acumen[claude]`, or plain `acumen` plus the -`codex` CLI on `PATH`. Selecting a model whose backend is missing fails immediately, with the -install command, before acumen prepares a target or spends anything. +`acumen epoch` is the loop. Each round benchmarks the current arm on the training tasks, records +what happened per task in `wiki/`, creates or improves the skill from that wiki, and benchmarks the +new version on the held-out validation tasks. Re-run it to keep going; you decide when to stop. -Claude API runs use `ANTHROPIC_API_KEY`; Codex API runs use `CODEX_API_KEY` (or -`OPENAI_API_KEY`). The meta-agent commands also accept a Codex model through the -`meta_model` config key or `--model`. +The stages also run on their own — `acumen bench`, `acumen wiki`, and `acumen improve`. -Every agentic command — `bench` included — takes `--auth {auto,session,api}` and defaults to -the provider's logged-in subscription, falling back to its API key. Both billing modes report -tokens, so Acumen can calculate the same API-rate estimate for either. Under `session`, that -estimate is what the run *would* have cost at API rates, not money billed — so each run records -its `auth_mode` alongside the figure. - -If the selected subscription runs out of usage or the API account runs out of credit, Acumen -invalidates the pass instead of scoring that as an agent failure: it prints the provider error, -cancels remaining cells for that provider, lets other providers finish all running and queued -cells, and exits non-zero. Replenish the credential and rerun the same command; automatic resume -retries the invalid and cancelled cells. Reports and `improve` refuse invalid quota/credit -evidence. - -`max_turns` and `max_usd` apply to both providers, but they are not equally strict for Codex, -which has no cap of its own — acumen enforces both against its event stream: - -- **`max_turns` bounds the run.** One `codex exec` is a single Codex turn however much work - happens inside it, so turns are counted in completed model actions (a message, a command, a - file change, a tool or search call) and the agent is stopped at the cap. -- **`max_usd` cannot.** Codex reports usage once, when the turn ends, so a breach is only - visible after the money is spent. The run is recorded as a budget failure — the same outcome - Claude gives it — but bound Codex spend with `max_turns`. acumen prints this before the pass. - -**Every cost acumen shows is inferred from tokens.** Each run records its breakdown (fresh -input, cache reads, cache writes, and output) and Acumen prices it with the rate table stored -in `result.json`. That gives Claude and Codex one comparable basis and prevents an old -benchmark from being silently re-priced, so it is what `cost_usd` holds and what every figure, -table, CSV column and console line reports. Where a backend supplies a dollar figure of its own -it is recorded beside it as `provider_cost_usd` (`recorded_cost_usd` in the report's sidecar -CSV), with the gap between the two, but nothing is plotted or tallied from it: Claude's SDK -total covers nested subagents that the run's own usage block does not, so a console reading it -would disagree with the report it summarises. A model no layer prices stays unpriced even when -the provider reported dollars, since one run on a basis the rest of the pass is not on is worse -than a visible gap. - -**Rates are read from the providers' pricing pages, never shipped with the package.** Prices -move, and each run's cost is frozen into its `result.json` and never recomputed, so a table -compiled into a release would store numbers that were already wrong. `bench` resolves rates -before it spends anything and **fails the pass** if the pages cannot be read: cost is a headline -metric, and a benchmark that cannot establish rates has not earned the numbers it would print. -`draft`, `improve`, `tasks`, `check`, and `ship` fetch too but degrade to unpriced instead — their -cost line is progress reporting, not stored evidence. - -Alongside the rates themselves each run records `price_source` (`config` or `fetched`) and -`price_rates_as_of`, so a pass run in August and another in October stay individually -attributable and one report can cover both without restating either. When arms in a report were -priced on different dates, the report says so: the cost gap between them includes the price -change, not only the skill's effect. - -```bash -acumen prices # the rates in use today, and where each came from -acumen prices --refresh # check pinned rates against what the providers publish -``` - -Pin rates with a `prices:` block in `config.yaml` to price a model the providers don't publish, -to price a gateway, or to record negotiated rates — pins outrank a live fetch, since only you -know what you are billed. They are also the only rates that can drift unnoticed, which is what -`--refresh` checks; it prints a diff for you to accept and never rewrites anything, because -picking the wrong tier or context band would silently misprice future runs. A model no layer -prices records its tokens and leaves report cost unavailable — never zero, which would read as -free. - -> One consequence worth knowing: Codex's `max_usd` cap is enforced from these same rates, so an -> unpriced model under Codex has no enforceable budget cap. Bound those runs with `max_turns`, -> or pin the rates. - -`draft`, `improve`, `tasks`, `ship`, and `check`'s review phase each drive an autonomous agent. -Every run writes a live `logs/acumen--.jsonl` (one event per step, flushed as it -goes — so you can watch progress by reading the file) and a rendered `.html` transcript. Add -`--stream` to mirror the conversation to the terminal, or `--log-dir` to change where the logs -land. +`acumen ship` gives your package a `-install-skills` command so its users can install the +skill into whichever agent they use (`--agent {claude,codex,agents,claude-science}`, or `--dest`). ## Getting started -Please refer to the [documentation][], -in particular, the [API documentation][]. - -## Installation - -You need to have Python 3.12 or newer installed on your system. -If you don't have Python installed, we recommend installing [uv][]. - -Install the backend you actually run — both are optional, and either alone is a complete -install: - -| you run | install | also needs | -|---|---|---| -| Claude only | `pip install acumen[claude]` | an Anthropic key or a `claude` login | -| Codex only | `pip install acumen` | the `codex` CLI on `PATH`, plus a Codex login or key | -| both | `pip install acumen[all]` | both of the above | - - - -And to install the acumen skill that ships with the package into your agent's skills directory, -run `acumen-install-skills --agent {claude,codex,agents,claude-science}` (or `--dest ` to -choose the directory yourself): - -```bash -acumen-install-skills --agent claude -``` +Please refer to the [documentation][], in particular the [API documentation][]. ## Release notes @@ -274,7 +63,7 @@ See the [changelog][]. ## Contact -For questions and help requests, you can reach out in the [scverse discourse][]. +For questions and help requests, reach out on the [scverse discourse][]. If you found a bug, please use the [issue tracker][]. ## Citation @@ -288,4 +77,3 @@ If you found a bug, please use the [issue tracker][]. [documentation]: https://acumen.readthedocs.io [changelog]: https://acumen.readthedocs.io/page/changelog.html [api documentation]: https://acumen.readthedocs.io/page/api.html -[pypi]: https://pypi.org/project/acumen diff --git a/pyproject.toml b/pyproject.toml index 3da0ae9..e4be2fc 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -44,7 +44,6 @@ urls.Documentation = "https://acumen.readthedocs.io/" urls.Homepage = "https://github.com/scverse/acumen" urls.Source = "https://github.com/scverse/acumen" scripts.acumen = "acumen.cli:main" -scripts.acumen-install-skills = "acumen._skills.install:main" [dependency-groups] dev = [ diff --git a/src/acumen/__init__.py b/src/acumen/__init__.py index 542ce72..ee500ec 100644 --- a/src/acumen/__init__.py +++ b/src/acumen/__init__.py @@ -29,7 +29,6 @@ summarize_checks, ) from acumen.config import Config, ConfigError, load_config, parse_config -from acumen.draft import DraftError, DraftResult, draft_skill from acumen.env import ( AuthMode, EnvError, @@ -41,15 +40,14 @@ scrubbed_env, session_auth_available, ) +from acumen.epoch import EpochPlan, resolve_epoch from acumen.grade import Grade, Reason, grade_answer, grade_run from acumen.improve import ( ImproveError, ImproveResult, - TrainRun, - collect_train_runs, - find_test_access, + find_valid_access, improve_skill, - make_test_guard, + make_valid_guard, ) from acumen.logs import LiveLog from acumen.paths import RunKey, Split, arm_name, is_complete, parse_run_dir, run_dir, skill_from_arm @@ -104,6 +102,14 @@ harvest_scripts, ) from acumen.tasks import Task, TaskError, TaskSplit, load_tasks, parse_tasks +from acumen.training import ( + EpochRow, + best_version, + build_training_rows, + epochs_since_best, + patience_exhausted, + write_training_csv, +) from acumen.trajectory import Trajectory, render_trajectory from acumen.transcript import ( build_trajectory, @@ -112,8 +118,32 @@ render_codex_transcript, render_transcript, ) +from acumen.wiki import ( + RunRecord, + TaskWikiResult, + WikiError, + collect_arm_runs, + recorded_arms, + update_wiki, +) __all__ = [ + "EpochRow", + "build_training_rows", + "write_training_csv", + "patience_exhausted", + "best_version", + "epochs_since_best", + "EpochPlan", + "resolve_epoch", + "find_valid_access", + "make_valid_guard", + "RunRecord", + "TaskWikiResult", + "WikiError", + "collect_arm_runs", + "recorded_arms", + "update_wiki", "AuthMode", "AgentError", "AgentOptions", @@ -126,8 +156,6 @@ "CheckSummary", "Config", "ConfigError", - "DraftError", - "DraftResult", "EnvError", "Grade", "Harvest", @@ -160,7 +188,6 @@ "TaskGenResult", "TaskSplit", "Trajectory", - "TrainRun", "__version__", "api_auth_available", "arm_metrics", @@ -176,11 +203,8 @@ "build_report", "check_task_split", "check_tasks", - "collect_train_runs", - "draft_skill", "dump_tasks", "find_skill_access", - "find_test_access", "generate_tasks", "grade_answer", "grade_run", @@ -191,7 +215,6 @@ "installer_exists", "is_complete", "make_skill_guard", - "make_test_guard", "latest_version", "load_config", "locate_transcript", diff --git a/src/acumen/_skills/__init__.py b/src/acumen/_skills/__init__.py deleted file mode 100644 index e69de29..0000000 diff --git a/src/acumen/_skills/data/SKILL.md b/src/acumen/_skills/data/SKILL.md deleted file mode 100644 index 0a7fbf3..0000000 --- a/src/acumen/_skills/data/SKILL.md +++ /dev/null @@ -1,127 +0,0 @@ ---- -name: acumen -description: Use for any question or task involving the Python package `acumen` (its CLI or its API) — setting up a benchmark project against a target package, writing or generating benchmark tasks, drafting/improving/hand-editing an agent Skill, running and interpreting skill-vs-baseline benchmark passes, shipping a skill into the target package, diagnosing a run, and choosing what to do next in that loop — open it before answering or running anything, because acumen's defaults, guardrails and correct next step are not guessable from the command names. ---- - -# acumen - -One project directory targets one Python package. Everything is driven by the `acumen` -CLI (a thin shell over an importable API — see `references/python-api.md`). Commands are -run from the project dir and default to `config.yaml`, `tasks.yaml`, `skills/`, `runs/`, -`logs/`. - -```bash -acumen init # scaffold config.yaml + tasks.yaml (placeholders the user must fill) -acumen tasks # optional: mine the package for tasks, run them for ground truth -acumen bench --no-skill # baseline arm (bare `acumen bench` runs every arm there is) -acumen draft # agent reads the package source -> skills/v1/ -acumen bench --skill v1 -acumen improve # agent reads v1's TRAIN runs -> skills/v2/ -acumen bench --skill v2 -acumen report # report.html + report.csv over every arm on disk -acumen ship --skill v2 # wire a `-install-skills` script into the target package -``` - -That is the shape of the loop, not a script to execute unattended, and not a fixed order: -several stages have more than one correct next move, and some stages call for talking to -the user rather than running the next command at all. - -## The loop is collaborative — propose, discuss, then run - -`bench` bills the Anthropic API for every cell and the other commands drive long -autonomous agents, so the user decides each step and when to stop. At every stage: state -what you would do next and why, and let the user choose. Never chain commands -unattended, and never decide on your own that a skill is good enough. - -**Before running anything, get from the user what acumen cannot infer** — the target, the -models, the budgets, whether tasks and results look right, whether to ship. Guessing those -is the most common way to waste a pass. The step after a command is often a question, not -a command. - -| Just happened | What to propose next | -|---|---| -| `acumen init` | Both files are placeholders (`repo: OWNER/REPO`, `REPLACE_ME` answers). Ask the user for what only they can supply — the target repo/path, ref, models, budgets — and fill in `config.yaml` with their answers. Do not guess a target or run the next command. | -| `config.yaml` filled | Offer both ways to get tasks: `acumen tasks` to generate them, or writing `tasks.yaml` by hand. Review generated tasks with the user before benching. | -| config + tasks ready | `acumen bench --no-skill` and `acumen draft` are **both** correct next steps — they are independent, and you need the baseline arm and a skill arm before any comparison means anything. Offer both. | -| a skill version exists | `acumen bench --skill vN`, or bare `acumen bench` to cover the baseline and every version at once (resume means only the unbenched cells cost anything). | -| any `bench` finished | `acumen report` — the per-arm, per-split numbers come from the report; do not judge a version by eyeballing runs or by one arm's pass count. | -| `report` written | Discuss the results with the user and propose next steps: `improve` + re-bench if it has not beaten the baseline or the previous version, `ship` once a version has proven out, or stop. The user decides. | -| user has a skill dir from elsewhere | Copy it into `skills/v1/` (next unused version) inside the acumen project — versions are only ever read from `skills/`; there is no import command and no external path flag. | - -## Route by goal - -| Goal | Command | Depth | -|---|---|---| -| Start a project for package X | `acumen init`, then fill `config.yaml` with the user | `references/setup.md` | -| Get benchmark tasks without writing them | `acumen tasks [--force] [--feedback "…"]` | `references/setup.md` | -| Write tasks by hand | edit `tasks.yaml` | `references/setup.md` | -| Get a first skill | `acumen draft [--feedback "…"]` | `references/authoring.md` | -| Measure whether the skill helps | `acumen bench --no-skill` and `acumen bench --skill vN` | `references/benchmark.md` | -| Make the skill better | `acumen improve [--from vN]` then bench the new version | `references/authoring.md` | -| See results / decide when to stop | `acumen report` | `references/benchmark.md` | -| Hand-edit a skill version | copy `skills/vN` → `skills/v(N+1)`, edit, bench it | `references/authoring.md` | -| Give package users the skill | `acumen ship --skill vN` | `references/ship.md` | -| Script any of this in Python | `from acumen import …` | `references/python-api.md` | -| A run failed / the skill did nothing | inspect `runs/…/result.json`, `logs/*.jsonl` | `references/benchmark.md` | - -## Preconditions - -- Python ≥ 3.12, and **`uv` on PATH** — acumen builds the target's venv with it. -- The target (`repo`: a GitHub URL or a local path) must be pip-installable and declare - `[project].name` in `pyproject.toml`. A local `repo` path is resolved **relative to - `config.yaml`**. -- Credentials: every agentic command takes `--auth {auto,session,api}` and defaults to the - selected provider's logged-in subscription, falling back to its API key. Claude uses - `ANTHROPIC_API_KEY` (or Anthropic provider credentials) / a `claude` login; Codex uses - `CODEX_API_KEY` or `OPENAI_API_KEY` / a `codex login`. `bench` included: cost comes from - token counts, which both billing modes report, so a subscription run prices as accurately - as a metered one — but under `session` `cost_usd` is what the run *would* have cost at API - rates, not metered spend. Each run records its `auth_mode`. - Both backends are optional: `pip install acumen[claude]` for Claude, plain `acumen` plus - the `codex` CLI for Codex. A model whose backend is missing fails preflight with the - install command. -- **Codex caps are not equally strict.** `codex exec` has no cap of its own, so acumen - enforces both from its event stream. `max_turns` bounds the run (counted in completed - model actions, since one `codex exec` is a single Codex turn). `max_usd` cannot: Codex - reports usage only when a turn ends, so an over-budget run is recorded as a `budget` - failure but the spend already happened — bound Codex with `max_turns`. -- **`cost_usd` is derived from token counts**, not from the provider — one arithmetic - path for both. Rates are frozen into each run; see them with `acumen prices`, re-check - them with `acumen prices --refresh`, override or extend via `prices:` in `config.yaml`. -- The target is cloned + installed into a venv cached under `~/.cache/acumen`, keyed by - (repo, ref). Use `--refresh-target` after changing the target's own source. - -## What guessing gets wrong - -1. **Grading is exact string match on `answer.md` after `strip()`, case-sensitive.** - Nothing is normalized. `**TOKEN**` against `TOKEN` fails (recorded as `format_error`, - not `wrong_answer`). Task answers must be one short unambiguous token. -2. **A pass is models × tasks × splits × replicates, times the arms it covers.** The - scaffolded config lists 6 models and `n_replicates: 3`, so *one* task costs 36 agent runs - *per arm* — and bare `acumen bench` runs every arm on disk, so with two skill versions - that is 108. Check with `acumen bench --dry-run` (it plans and exits, spending nothing, - over the same arms the real run would) and agree the size with the user before spending. - Trim with `models:`, `n_replicates: 1`, `--task ID`, `--split`, or by naming one arm. -3. **`max_turns`/`max_usd` in `config.yaml` cap benchmark agents only.** `draft`, - `improve`, `tasks`, and `ship` are **unbounded** unless you pass `--max-turns`/`--max-usd`. -4. **Skill versions are immutable.** `draft` refuses (exit 2) if any `skills/vN` exists; - `improve` always writes the next unused directory. Never edit a benched version in - place — its hash is recorded in every `result.json`. -5. **`SKILL.md` frontmatter `name` must equal `config.skill_name`**, which defaults to the - repo's last path component, slugified and lowercased (`.../My_Pkg` → `my_pkg`). A - mismatch makes `bench`/`improve`/`ship` fail on load. `description` must be non-empty. -6. **`improve` needs benched train evidence for its parent.** Run - `acumen bench --skill vN` before `acumen improve`, or it errors with nothing to read. -7. **Never leak the test split.** `improve` is structurally and hook-blocked from - `runs/*/test/`; don't defeat that by pasting test answers into `--feedback`. A widening - train/test gap in the report is the overfitting signal you are watching for. -8. **Task prompts must not name the target package** — the harness already tells the agent - which package to use and that it is installed. -9. **Resume is automatic**: a valid run is "complete" when its `result.json` exists and is - non-empty, and completed runs are skipped. `--no-resume` re-runs them. Provider usage/credit - exhaustion writes a diagnostic `valid: false` result, cancels only that provider's remaining - cells while other providers finish, and remains pending; replenish the credential and rerun - the same command. Renaming a task `id` orphans its old runs (the id is a path component). -10. **Confirm the skill actually loaded.** `bench` prints `skill loaded in N/M runs` from - per-run `skill_loaded` evidence, and warns when a skill arm never fired the Skill tool - (that arm measures nothing) or when the baseline did. diff --git a/src/acumen/_skills/data/references/authoring.md b/src/acumen/_skills/data/references/authoring.md deleted file mode 100644 index 1817d9a..0000000 --- a/src/acumen/_skills/data/references/authoring.md +++ /dev/null @@ -1,92 +0,0 @@ -# Authoring skill versions: draft, improve, hand-edit - -## The skill directory contract - -``` -skills/v1/ - SKILL.md # required, YAML frontmatter with `name` and `description` - references/*.md # optional - meta.json # acumen bookkeeping — NOT part of the skill -``` - -- `name` must equal `config.skill_name` exactly; `description` must be non-empty. Both are - enforced on load by `bench`, `improve`, and `ship`. -- `meta.json` (`version`, `parent`, `rationale`, `hash`, optional `feedback`) is written by - acumen. It is excluded from the content hash and is never copied to a consuming agent. -- The content hash covers every other file's relative path and bytes, and is recorded in - every `result.json` — so editing a version after benching it silently invalidates the - comparison. **Versions are immutable; always make a new one.** - -## `acumen draft` - -```bash -acumen draft [--force] [--feedback "…"] [--model M] [--max-turns N] [--max-usd X] - [--stream] [--log-dir logs] [--auth auto|session|api] -``` - -One agent that reads the target's **source** (docstrings, examples, docs) and writes -`skills/v1/`. It works in a staging dir; only a skill that loads and validates is promoted, -so a failed draft leaves no broken version behind. Exits 2 if any version already exists — -pass `--force` to add another, or use `improve` to build on the latest. - -## `acumen improve` - -```bash -acumen improve [--from vN] [--feedback "…"] [--model M] [--max-turns N] [--max-usd X] - [--stream] [--log-dir logs] [--auth auto|session|api] -``` - -Reads **how the parent version performed on the train split** — not the package source — -and edits a copy of it into the next version. - -- Defaults to the latest version; `--from vN` picks another. Always writes the next unused - directory. -- Requires `runs/skill_vN/train/**/result.json` to exist: bench the parent first. -- The agent gets a curated copy of train runs (`SUMMARY.md` with failures first, plus each - run's prompt, expected, actual, `script.py`, and transcript). The real `runs/` tree is not - in scope, and a `PreToolUse` hook denies any path resolving under `runs/*/test/`. -- The CLI warns if the new version is byte-identical to its parent (the improver changed - nothing) — that is a signal to give it sharper evidence or `--feedback`. - -## `--feedback` on `tasks` / `draft` / `improve` - -Free text injected into the agent's prompt as *subordinate* guidance — it never overrides -the isolation or anti-overfit rules. Use it for what the agent cannot infer: package -context, what to emphasise, which functionality to skip. For `draft`/`improve` it is -recorded in `meta.json` and shown in the report. **Never paste test-split answers into -`improve --feedback`** — that defeats the split. - -## Writing or editing a version by hand - -Perfectly supported; the agents are a convenience, not a requirement. A skill authored -outside the project is benched the same way: copy the directory in as the next unused -`skills/vN` — `bench` only ever loads versions from the project's `skills/` root. - -```bash -cp -r skills/v2 skills/v3 # then edit skills/v3/, and delete its inherited meta.json -acumen bench --skill v3 -``` - -A hand-made version with no `meta.json` benches fine; it just has no rationale/diff in the -report. What makes a version score well: - -- The `description` decides whether the skill loads at all — name the goals a user would - actually phrase, honestly. An unloaded skill is a wasted arm. -- Keep `SKILL.md` short and push depth into `references/`; every token is paid on every - task, including ones where the skill is irrelevant. -- Spend words on what an agent would get wrong by guessing: non-obvious defaults, required - preprocessing, the function that looks right but isn't, argument shapes/orientation, - where output lands, the right order of steps. -- **Never name a dataset, parameter value, column, or expected answer from the train - tasks.** That is overfitting, and the test split will catch it. -- Verify every claim against the installed package before writing it down. - -## Watching a long agent run - -`tasks`/`draft`/`improve`/`ship` each write `logs/acumen--.jsonl`, -one compact event per agent message, **flushed as it goes** — read that file to follow -progress instead of streaming into your context. Tool results are recorded by status and -size, not inlined. `--stream` mirrors the conversation to the terminal; `--log-dir` moves -the logs. At the end, the run is mapped into acumen's harness-neutral trajectory model and a -rendered `.html` transcript plus a portable `.trajectory.json` land beside the jsonl — the same -model and renderer for every provider, so the reports look alike. diff --git a/src/acumen/_skills/data/references/benchmark.md b/src/acumen/_skills/data/references/benchmark.md deleted file mode 100644 index 56e8a23..0000000 --- a/src/acumen/_skills/data/references/benchmark.md +++ /dev/null @@ -1,162 +0,0 @@ -# Benchmarking and reading results - -## `acumen bench` - -```bash -acumen bench [--no-skill | --skill v1] [--split train|test]... [--task ID]... - [--replicates N] [--max-concurrency N] [--dry-run] [--no-resume] - [--keep-sandboxes] [--refresh-target] - [--auth auto|session|api] - [--config config.yaml] [--tasks tasks.yaml] [--runs runs] [--skills skills] - [--cache ~/.cache/acumen] -``` - -- Neither `--no-skill` nor `--skill` → **every arm the project has**: the baseline plus each - version in `skills/`, benched one arm after another against one prepared target. That is - the whole comparison in one command, and also the largest thing `bench` can spend, so - check it with `--dry-run` first. Name an arm (`--no-skill` / `--skill vN`, mutually - exclusive) to run just that one. -- Arms run sequentially; within an arm, runs go `max_concurrency` at a time. Each arm prints - its own tally, then a combined one. Completed runs are skipped per arm as usual, so adding - `skills/v3` and rerunning `acumen bench` benches only v3. -- A version in `skills/` that fails to load stops the pass at planning, before target prep - and before any spend, rather than being silently dropped from the comparison. -- `--split` and `--task` are repeatable; omitted means all. They apply to every arm. -- `--dry-run` prints the planned matrix and exits **before** target prep and before any - agent runs — free, and the right way to check the size of a pass. It covers the same arms - the real run would, with per-arm counts and a total. -- `--replicates` / `--max-concurrency` override the config for this pass only. -- `--auth auto` (default) prefers a stored account/subscription login, then falls back to - metered API credentials. Use `--auth session` or `--auth api` to require one explicitly. - Each run records the resolved mode; under `session`, `inferred_cost_usd` is the equivalent - API-rate cost computed from tokens, not money charged to the subscription. -- Ctrl-C is safe: completed runs are preserved and the next invocation resumes. -- Provider account/session usage exhaustion or an API account without credit invalidates the - pass rather than counting against the agent. Acumen prints the provider error, cancels the - remaining cells for that provider, lets every other provider finish its running and queued - cells, exits non-zero, and records the triggering cell as `valid: false` for diagnosis. - Reports and `improve` refuse that evidence; after replenishing the credential, rerun the same - command and automatic resume retries invalid and cancelled cells. - -**Arm parity is the whole point.** Both arms get an identical prompt, tools, caps, and -environment; the only difference is that a skill arm copies the skill's content files into -`/.claude/skills//` for Claude or -`/.agents/skills//` for Codex, where project discovery finds it. -`meta.json` is never copied. Each run gets a fresh empty sandbox with the target venv on -PATH, a throwaway `HOME` and `CLAUDE_CONFIG_DIR` — no repo source, no user settings, no -`CLAUDE.md` memories, no visibility of other runs. - -## The run tree - -``` -runs/{arm}/{split}/{model}/{task_id}/rep_{n}/ -``` - -`arm` is `noskill` or `skill_v1`, `skill_v2`, …; `model` is slugified. Each completed leaf -holds five files: - -| File | What | -|---|---| -| `answer.md` | What the agent wrote — the only thing graded | -| `script.py` | The agent's reproduction script (absent if it ran no code) | -| `transcript.jsonl` | The provider's own transcript — SDK-native for Claude, the `codex exec` event stream for Codex | -| `transcript.html` | Rendered transcript, written for both providers | -| `result.json` | The unit of record — written **last** | - -`result.json` presence + non-zero size is what marks a run complete, which is what makes -resume safe. Useful fields: `success`, `reason`, `answer`, `expected`, `model`, `turns`, -`cost_usd`, `input_tokens`, `output_tokens`, `duration_s`, `skill_hash`, `skill_name`, -`skill_loaded`, `pkg_version`, `commit`, `session_id`, `subtype`, `valid`, `error`. - -**`cost_usd` is inferred from tokens, and equals `inferred_cost_usd`.** Each run also records -`provider_cost_usd` (Claude's SDK total, API-equivalent rather than necessarily an invoice -charge under session authentication; Codex reports no dollars), `cost_source`, and the provider -figure's absolute/relative distance from the inferred one when both exist. Inference is the -canonical basis because both providers report tokens and the rates are frozen into the result, -so the figure is reproducible and comparable across providers. Token classes and the rates in -`price_rates` remain available for reproduction, including Claude's separate five-minute and -one-hour cache writes plus the legacy aggregate. A run without rates leaves `cost_usd` and -`inferred_cost_usd` null and `cost_available` false even when the provider supplied a value: -that figure would put one run on a basis no other run in the pass is on. - -**No rates ship with the package.** They are read from the providers' pricing pages, because a -compiled-in table is wrong from whatever date prices next move, and each run's cost is frozen -when written rather than corrected later. `bench` resolves rates before it spends anything and -exits 2 if the pages cannot be read. Each run records `price_source` (`config` or `fetched`) and -`price_rates_as_of` next to the rates themselves, so passes months apart stay individually -attributable and one report can mix them; the report flags arms priced on different dates, whose -cost gap includes the price change rather than only the skill's effect. - -`draft`, `improve`, `tasks`, and `ship` also fetch, but degrade to unpriced rather than failing: -their cost line is progress reporting. Note that Codex's `max_usd` is derived from these rates, -so an unpriced Codex run has no enforceable budget cap — bound it with `max_turns`. - -Inspect today's rates with `acumen prices`. Set `prices:` in `config.yaml` to price a model the -providers don't publish or to pin negotiated rates, which outrank a fetch; `acumen prices ---refresh` reports pins that have drifted from the published price. - -Reports, the run table and the console all show that inferred value, so Claude and Codex stay -on the same pricing basis even when Claude supplies an SDK estimate. The sidecar CSV carries -`cost_usd` (the figure the charts use), `inferred_cost_usd`, and the provider's own figure as -`recorded_cost_usd`. Unpriced runs are dropped from cost charts and comparisons, and the report -opens with a warning naming the models that had no rates. - -`codex exec` has no cap of its own, so acumen enforces both from its event stream, at -different resolutions. `max_turns` really does stop the run — counted in completed model -actions (message, command, file change, tool/search call), because one `codex exec` is a -single Codex turn however much work happens inside it. `max_usd` only marks the outcome: -Codex reports usage once, when the turn ends, so the breach is visible after the spend, not -before. The run is recorded `budget` either way; the CLI warns before the pass. A turn-capped -Codex run is stopped mid-turn and so records no usage — a failure with zero tokens. - -## Reasons - -| `reason` | Meaning | -|---|---| -| `ok` | Exact match — the only success | -| `wrong_answer` | Content differs | -| `format_error` | Content would match but formatting broke the exact match (bold, quotes, "Answer:", code fence, trailing period) | -| `no_answer_file` | The agent never wrote `answer.md` | -| `budget` / `max_turns` | Cap breached — a **failure regardless** of what `answer.md` contains | -| `error` | The agent crashed or produced no result | -| `provider_exhausted` | Provider usage/credit ran out; infrastructure-invalid, never agent evidence | - -A run of `format_error`s means the harness's answer-format instruction is losing to the -task prompt; a run of `no_answer_file` usually means the task is too big for `max_turns`. - -## Diagnosing a disappointing arm - -1. `skill loaded in 0/N runs` on a skill arm → the arm measured nothing. The skill's - `description` is what decides whether it loads; make it name the goals the task prompts - actually phrase. -2. The Skill tool firing in *baseline* runs is also flagged — that baseline is not clean. -3. Open `runs/.../transcript.html` for a failing run, or re-run one cell with - `--task ID --split train --replicates 1 --keep-sandboxes` and inspect the sandbox. -4. Compare `answer` vs `expected` in `result.json` before blaming the skill — a systematic - near-miss is a task-authoring problem, not a skill problem. - -## `acumen report` - -```bash -acumen report [--runs runs] [--tasks tasks.yaml] [--skills skills] [--out report.html] - [--palette claude-opus-5=#3b7ea1]... -``` - -- Writes `report.html` **and a sidecar `report.csv`** next to it, overwriting both; the - report always reflects every run currently on disk across every arm. -- Reads only `result.json` files — never transcripts. Fails if `runs/` holds none. -- **The figures show the TEST split** (the held-out measure). The per-run table below them - lists both splits. -- Sections: Overview (success rate, tokens, cost, time per arm, bars coloured by model), - Per-task breakdown, Runs table (links to each `transcript.html`), and — when `--skills` - resolves — Skill versions with each version's rationale and diff against its parent. -- `--tasks` / `--skills` are optional; missing ones only drop their sections (it says so). -- `--palette MODEL=COLOUR` is repeatable and accepts comma-separated pairs. A key that - matches no model in the data, or an unparseable colour, is rejected up front. - -Run `acumen report` after every bench — it is how a version is judged, and the only view -that compares arms and splits side by side. Then take the results to the user: say what -improved, what did not, and what you would do next (`improve` and re-bench, more tasks, -`ship`, or stop). Train improving while test does not is overfitting — that gap is exactly -what the split exists to expose — but the call on when to stop and what to ship is the -user's. diff --git a/src/acumen/_skills/data/references/python-api.md b/src/acumen/_skills/data/references/python-api.md deleted file mode 100644 index 4f58cda..0000000 --- a/src/acumen/_skills/data/references/python-api.md +++ /dev/null @@ -1,112 +0,0 @@ -# Python API - -Everything is re-exported from the top level: `from acumen import build_report, load_config, …`. -The CLI (`acumen.cli:main`) is a thin shell over it, so anything the CLI does is scriptable. -`main(argv)` returns an exit code and converts the library's error types into `error: …` on -stderr — call the functions directly if you want the exceptions. - -## Loading config and tasks - -```python -from pathlib import Path -from acumen import load_config, load_tasks, parse_config, parse_tasks - -cfg = load_config(Path("config.yaml")) # validates; resolves a local repo relative to the file -tasks = load_tasks(Path("tasks.yaml")) # list[Task]; task.split("train") -> TaskSplit(prompt, answer) -``` - -`Config` is a frozen dataclass — override with `dataclasses.replace(cfg, n_replicates=1)`. -`parse_config` / `parse_tasks` take already-parsed dicts, for building them in memory. - -## Preparing the target - -```python -from acumen import prepare_target -from acumen.env import DEFAULT_CACHE_ROOT - -target = prepare_target(cfg, DEFAULT_CACHE_ROOT, refresh=False) -target.python # venv interpreter with the package installed -target.bin_dir # what goes on an agent's PATH -target.fingerprint # " ", as recorded in result.json -``` - -Needs `uv` on PATH. Cached by (repo, ref) under `~/.cache/acumen`. - -## Planning and running a pass - -```python -import asyncio -from acumen import build_matrix, pending, run_matrix, load_skill, summarize - -skill = load_skill(Path("skills"), "v1", expect_name=cfg.skill_name) # None for the baseline -planned = build_matrix(cfg, tasks, skill="v1", splits=("train",), task_ids=["t1"]) -todo = pending(planned, Path("runs"), resume=True) -outcomes = asyncio.run( - run_matrix( - todo, - target=target, - runs_root=Path("runs"), - max_concurrency=cfg.max_concurrency, - auth_mode="api", - skill=skill, - on_start=lambda p: None, - on_done=lambda o: None, - env_passthrough=cfg.env_passthrough, - ) -) -summarize(outcomes) # {"ok": 5, "wrong_answer": 1, ...} -``` - -`run_matrix` and `run_once` are coroutines. `run_once` records agent crashes as failed runs -rather than raising, so one bad run never kills a pass. `run_once` raises `ValueError` if -`key.arm` and the `skill` argument disagree — the arm is the source of truth. - -`build_matrix` (and `--dry-run`) is pure planning: no agents, no network, no cost. - -## Grading and paths - -```python -from acumen import grade_answer, grade_run, run_dir, parse_run_dir, is_complete, RunKey, arm_name - -grade_answer("**SPI1**", "SPI1") # Grade(success=False, reason='format_error', answer='**SPI1**') -run_dir(Path("runs"), RunKey(arm=arm_name("v1"), split="test", model="claude-opus-5", task_id="t1", rep=1)) -is_complete(d) # a non-empty result.json is what "done" means -``` - -## The meta-agents - -`draft_skill`, `improve_skill`, `generate_tasks`, `ship_skill` are all coroutines taking -keyword-only args (`cfg=`, `target=`, plus their own roots) and returning a result dataclass -(`DraftResult`, `ImproveResult`, `TaskGenResult`, `ShipResult`) carrying the new `Skill`, -`cost_usd`, `turns`, and log paths. `max_turns`/`max_usd` default to `None` = **unbounded**. -Pass a `LiveLog` as `log=` for the JSONL feed: - -```python -from acumen import LiveLog, draft_skill - -log = LiveLog.open(Path("logs"), "draft", stream=False) -with log: - result = asyncio.run(draft_skill(cfg=cfg, target=target, skills_root=Path("skills"), auth_mode="session", log=log)) -``` - -## Aggregating results - -```python -from acumen import load_results, arm_metrics, build_report - -df = load_results(Path("runs")) # one row per result.json, + total_tokens, arm_label -arm_metrics(df[df["split"] == "test"]) # per-arm rate, stderr, tokens, cost, time, n -report = build_report(Path("runs"), Path("report.html"), tasks, skills_root=Path("skills")) -report.n_runs, report.results # the DataFrame behind the HTML -``` - -`build_report` writes `report.html` **and `report.csv`** (same stem) and returns `Report`. -Filter `df` to a split yourself before `arm_metrics` — it does not. - -## Introspection helpers - -`available_versions`, `latest_version`, `next_version`, `skill_hash`, `skill_content` -(for diffing), `installer_exists(src_dir)`, `collect_train_runs(runs_root, arm, tasks)`, -and the two pure guards `find_test_access` / `find_skill_access` (testable without an agent). -`scrubbed_env` / `build_agent_env` / `sandbox` / `install_skill` let you reproduce a run's -exact isolated environment by hand. diff --git a/src/acumen/_skills/data/references/setup.md b/src/acumen/_skills/data/references/setup.md deleted file mode 100644 index 67f9eb8..0000000 --- a/src/acumen/_skills/data/references/setup.md +++ /dev/null @@ -1,109 +0,0 @@ -# Project setup: `config.yaml` and `tasks.yaml` - -`acumen init [--dir DIR] [--force]` writes both files as annotated placeholders. It refuses -to clobber either unless `--force`. Both loaders are **strict: unknown keys are rejected**. - -Nothing else can run until `config.yaml` is filled in: the scaffold ships -`repo: https://github.com/OWNER/REPO` and a `REPLACE_ME` example task. Ask the user for the -values only they know (target repo/path, ref, extras, which models, budgets) instead of -inventing them. - -## config.yaml - -Only `repo` is required; delete a line to take its default. - -| Key | Default | Notes | -|---|---|---| -| `repo` | — | GitHub URL (`https://`, `git@`, `ssh://`, `git://`) or a local path. A local path is resolved relative to `config.yaml` and must exist. | -| `ref` | `main` | Branch/tag/commit; ignored for local paths. | -| `extras` | `[]` | Extras the target publishes in `[project.optional-dependencies]`, e.g. `[test]`. | -| `dependency_groups` | `[]` | PEP 735 groups from the target's `[dependency-groups]`, e.g. `[full]`. | -| `pip_packages` | `[]` | Packages the target declares nowhere, installed alongside it. PEP 508 specifiers allowed (`numpy<2`). | -| `python` | `"3.12"` | Interpreter for the target's venv (quote it — `3.10` unquoted is a float). | -| `env_passthrough` | `[]` | Extra env var names agents may keep. The agent env is a clean allowlist (auth/proxy/TLS only); **everything else from your shell is blanked**. A target needing `OMP_NUM_THREADS`, `R_HOME`, a service key, etc. must name it here. | -| `models` | `[claude-opus-5]` | Benchmark models. Claude (`claude*`) and Codex (`gpt-*`, `o1`/`o3`/`o4`, `codex-*`) models may be mixed. Duplicates rejected. `models[0]` is the default for the four `*_model` keys below. | -| `n_replicates` | `3` | Runs per (model, task, split) cell. | -| `max_concurrency` | `4` | Simultaneous benchmark agents. | -| `max_turns` | `40` | **Benchmark agents only.** | -| `max_usd` | `3.0` | **Benchmark agents only.** | -| `draft_model` / `improve_model` / `tasks_model` / `ship_model` | `models[0]` | Meta-agent models; overridable per command with `--model`. | -| `skill_name` | repo basename, slugified + lowercased | Must equal the `name:` in the skill's frontmatter. | -| `prices` | built-in table | Per-model token rates (USD per million): `{model: {input, output, cached_input?, cache_write?, cache_write_5m?, cache_write_1h?}}`. Overrides or extends acumen's table — needed for a model it doesn't ship a rate for, a gateway, or negotiated rates. See `acumen prices`. | - -**`extras` vs `dependency_groups` — pick the wrong one and the packages are missing.** Extras -are published in package metadata, so they are what `pip install pkg[name]` resolves. PEP 735 -groups live only in the source tree and are invisible to anyone installing from PyPI. Rule of -thumb: if `pip install [name]` from PyPI would not work, `name` is a group. Check the -target's `pyproject.toml` rather than guessing — many scverse packages declare **no** extras -and put their optional stack in a `full` group. Asking for the wrong kind is now a hard error -naming the right key, but only after the checkout, so read the file first. - -Changing `extras`, `dependency_groups` or `pip_packages` changes the cache key, so the venv -rebuilds on the next command — no `--refresh-target` needed. - -The scaffold's `models:` line lists three Claude and three Codex models. With -`n_replicates: 3`, one task expands to 36 runs per arm, so cut the matrix down before -the first real pass. - -## tasks.yaml - -```yaml -tasks: - - id: some_analysis # unique, filesystem-safe: [A-Za-z0-9._-]; used as a path component - train: - prompt: >- - One paragraph: the goal, the input, and exactly what to report. - answer: ONE_TOKEN - test: - prompt: >- - The same analysis on a different input / target. - answer: ANOTHER_TOKEN - # optional per-task overrides — the only other allowed keys: - # max_turns: 60 - # max_usd: 5.0 - # model: claude-sonnet-5 -``` - -Both `train` and `test` are required, each with non-empty `prompt` and `answer`. Both -splits always run in a pass; only train results ever reach `improve`. - -### Writing a task that measures anything - -- **Answers are graded by exact string match after `strip()`, case-sensitive.** Keep the - answer to one token: a name, a category, a count, a number at a stated precision. End the - prompt by stating exactly what to report and in what form. -- **Do not name the target package**, and do not mention a version — the harness preamble - already tells the agent the package is installed and which interpreter to use. -- **State the goal, not the recipe.** No numbered steps, no function/argument names, no - description of the data's shape or columns. What is being measured is whether the agent - can find the "how" itself; a prompt that spells out the calls measures nothing. -- Train and test must be **the same analysis on different inputs**, with different correct - answers — otherwise a skill can pass by memorizing one answer. -- A good task is one the target package solves and the **no-skill baseline gets wrong** — - a task the baseline already passes leaves no room to show a gain. The `--no-skill` arm is - what tells you; it can be run before or after the first skill exists. -- Renaming an `id` orphans every existing run under it. - -## `acumen tasks` — generate tasks.yaml - -```bash -acumen tasks [--out tasks.yaml] [--force] [--feedback "skip the plotting API"] \ - [--model M] [--max-turns N] [--max-usd X] [--stream] [--log-dir logs] -``` - -One autonomous agent that reads the package source **and executes it in the target venv**: -every ground-truth answer comes from a script it actually ran, not from docs. It enumerates -the package's tutorials/vignettes and writes at least one task per tutorial, so expect a -sizeable file. Unbounded in turns and cost by default. - -- Refuses to overwrite an existing `--out` without `--force` (the check runs *before* the - costly target prep). -- **There is no append mode.** The generator never reads your existing tasks file, so - appending would silently duplicate. To combine, generate to a separate path and merge by - hand. -- Existing skills and agent-instruction files (`SKILL.md`, `.agents/`, `.claude/`, `.codex/`, `CLAUDE.md`, - `AGENTS.md`, `.cursor/`, root `skills/`) are stripped from the source copy it reads and - blocked by a hook, so pre-written guidance cannot bias which tasks it picks. -- The output is validated through the strict tasks loader before it is written — it can - never emit a file the rest of the pipeline would reject. -- **Review what it produces.** The answers are only as good as the scripts it ran. diff --git a/src/acumen/_skills/data/references/ship.md b/src/acumen/_skills/data/references/ship.md deleted file mode 100644 index 0a4bd51..0000000 --- a/src/acumen/_skills/data/references/ship.md +++ /dev/null @@ -1,53 +0,0 @@ -# `acumen ship` — put a proven skill inside the package - -```bash -acumen ship --skill v2 [--force] [--model M] [--max-turns N] [--max-usd X] - [--stream] [--log-dir logs] [--auth auto|session|api] -``` - -`--skill VERSION` is **required** — there is no implicit "latest" or "best". Run it only -after the report shows that version earning its place *and* the user has agreed to ship it: -it writes to the target's real checkout and can open a PR. - -## What it produces - -An agent modifies the target checkout so the package gains a `-install-skills` -console script. The package's users then run, e.g.: - -```bash -scanpy-install-skills --agent claude # or codex | agents | claude-science -scanpy-install-skills --dest ./somewhere # explicit destination -scanpy-install-skills --print-path # where the bundled skill lives -``` - -There is **no default framework** — `--agent` or `--dest` is required. The same -`SKILL.md` + `references/` bundle installs verbatim into every framework; there is no -per-framework conversion. Destinations are `/skills/`, with roots -`~/.claude` (`CLAUDE_CONFIG_DIR`), `~/.codex` (`CODEX_HOME`), `~/.agents`, and the active -org's directory for `claude-science`. - -Inside the package the agent creates `/_skills/` with `install.py` (a canonical -template it must copy verbatim) and `data/` holding the skill files, then wires the console -script and — the step that silently fails — the **build-backend packaging** so those -non-`.py` data files actually ship in the wheel. It verifies by building a wheel, installing -it into a fresh (non-editable) venv, and confirming `SKILL.md` lands. - -## Delivery depends on `repo` - -- **GitHub URL** → the agent branches, commits, pushes, and opens a PR with `gh` (it does - not merge). Needs write access; a rejected push is reported, not worked around. -- **Local path** → the change is written straight into your working tree, no branch, no - commit. Review with `git diff`. - -## Non-obvious behaviour - -- **The ship agent is deliberately NOT isolated.** Unlike every other acumen agent it runs - in your real environment: real `HOME`, real network, real git/`gh` credentials, real `uv`, - with `permission_mode="bypassPermissions"`. Only the model credential is constrained (by - `--auth`). Run it when you are ready for it to touch your checkout and remote. -- It refuses to run if the target already declares a `*-install-skills` console script or - has a `_skills/install.py` — pass `--force` to ship anyway. -- It ships exactly one version, copied verbatim; it never authors or edits skill text, and - it writes no tests. -- Unbounded turns/cost by default. Its final summary (PR URL, packaging changes, - build-verify result) is printed and returned. diff --git a/src/acumen/_skills/install.py b/src/acumen/_skills/install.py deleted file mode 100644 index 1ea990a..0000000 --- a/src/acumen/_skills/install.py +++ /dev/null @@ -1,175 +0,0 @@ -"""Install the bundled ``acumen`` skill into an agentic framework's skills directory. - -Console script (wired as ``-install-skills`` in ``pyproject.toml``): copy the skill that -ships inside this package into a chosen framework's skills directory, so an agent can load it. -The same ``SKILL.md`` + ``references/`` bundle is a cross-framework standard, so it installs -verbatim — no per-framework conversion. - -Frameworks (``--agent``): - -- ``claude`` -> ``~/.claude/skills`` (honours ``CLAUDE_CONFIG_DIR``) -- ``codex`` -> ``~/.codex/skills`` (honours ``CODEX_HOME``) -- ``agents`` -> ``~/.agents/skills`` -- ``claude-science`` -> the active org's skills dir, resolved from - ``~/.claude-science/active-org.json`` - -``--dest`` overrides all of them. There is **no default framework**: pass ``--agent`` or -``--dest``. The skill files live in the ``data/`` directory next to this module and are read via -``importlib.resources``, so this works from an installed wheel, not just an editable checkout. -""" - -from __future__ import annotations - -import argparse -import json -import os -import shutil -import sys -from importlib import resources -from pathlib import Path - -#: The skill name; it installs to ``//``. -SKILL_NAME = "acumen" - -#: Framework -> (env var that overrides the config root, default config root). The skills -#: directory is ``/skills``. ``claude-science`` is resolved separately. -_AGENT_ROOTS = { - "codex": ("CODEX_HOME", "~/.codex"), - "claude": ("CLAUDE_CONFIG_DIR", "~/.claude"), - "agents": (None, "~/.agents"), -} - -#: The frameworks ``--agent`` accepts. -AGENTS = (*sorted(_AGENT_ROOTS), "claude-science") - - -def source_dir() -> Path: - """Return the package-owned skill directory (the bundle that gets copied).""" - source = Path(str(resources.files(__package__).joinpath("data"))) - if not (source / "SKILL.md").is_file(): - raise RuntimeError(f"packaged skill data is missing: {source}") - return source - - -def _claude_science_skills_dir() -> Path: - """Resolve the active Claude Science org's skills directory.""" - root = Path("~/.claude-science").expanduser() - active_org_path = root / "active-org.json" - try: - active_org = json.loads(active_org_path.read_text(encoding="utf-8")) - except (OSError, json.JSONDecodeError) as error: - raise ValueError( - f"cannot resolve the Claude Science active organization from {active_org_path}; pass --dest instead" - ) from error - org_uuid = active_org.get("org_uuid") if isinstance(active_org, dict) else None - if not isinstance(org_uuid, str) or not org_uuid or Path(org_uuid).name != org_uuid or org_uuid in {".", ".."}: - raise ValueError(f"invalid Claude Science org_uuid in {active_org_path}; pass --dest instead") - return root / "orgs" / org_uuid / "skills" - - -def _skills_dir(agent: str) -> Path: - """Return the skills directory (parent of the install dir) for a framework.""" - if agent == "claude-science": - return _claude_science_skills_dir() - variable, fallback = _AGENT_ROOTS[agent] - root = os.environ.get(variable) if variable is not None else None - return Path(root or fallback).expanduser() / "skills" - - -def resolve_dest(agent: str | None, dest: Path | None) -> Path: - """Resolve the install destination from ``--agent`` / ``--dest``. - - ``--dest`` wins. With neither, raise ``ValueError`` — there is no default framework. - """ - if dest is not None: - return dest.expanduser() - if agent is None: - raise ValueError("pass --agent {" + ",".join(AGENTS) + "} or --dest to choose where to install") - return _skills_dir(agent) / SKILL_NAME - - -def _snapshot(root: Path) -> dict[str, bytes]: - """Map each file under ``root`` to its bytes, for exact tree comparison.""" - return {str(item.relative_to(root)): item.read_bytes() for item in root.rglob("*") if item.is_file()} - - -def _matches(source: Path, target: Path) -> bool: - """Report whether ``target`` is a byte-for-byte copy of ``source``.""" - return target.is_dir() and _snapshot(source) == _snapshot(target) - - -def main(argv: list[str] | None = None) -> int: - """Install the bundled skill; entry point for the ``-install-skills`` script.""" - parser = argparse.ArgumentParser( - description=f"Install the {SKILL_NAME} skill bundled with this package into an agent's skills directory.", - ) - parser.add_argument( - "--agent", - choices=AGENTS, - default=None, - help="framework to install into (no default — pass this or --dest)", - ) - parser.add_argument( - "--dest", - type=Path, - default=None, - help="exact skills directory to install into (overrides --agent)", - ) - parser.add_argument( - "--force", - action="store_true", - help="overwrite an existing installation that differs from the bundled skill", - ) - action = parser.add_mutually_exclusive_group() - action.add_argument( - "--check", - action="store_true", - help="report whether the installed skill matches the bundled one; do not install", - ) - action.add_argument( - "--print-path", - action="store_true", - help="print the bundled skill's location inside the package and exit", - ) - args = parser.parse_args(argv) - - try: - source = source_dir() - if args.print_path: - print(source) - return 0 - if args.check: - target = resolve_dest(args.agent, args.dest) - if not target.exists(): - print(f"{SKILL_NAME} skill is not installed at {target}", file=sys.stderr) - return 1 - if _matches(source, target): - print(f"{SKILL_NAME} skill at {target} matches the bundled copy") - return 0 - print(f"{SKILL_NAME} skill at {target} differs from the bundled copy", file=sys.stderr) - return 1 - - dest = resolve_dest(args.agent, args.dest) - if dest.exists(): - if _matches(source, dest): - print(f"{SKILL_NAME} skill already up to date at {dest}") - return 0 - if not args.force: - print(f"{dest} already exists and differs; pass --force to overwrite", file=sys.stderr) - return 1 - if dest.is_dir(): - shutil.rmtree(dest) - else: - dest.unlink() - dest.parent.mkdir(parents=True, exist_ok=True) - shutil.copytree(source, dest) - except (OSError, RuntimeError, ValueError) as error: - print(f"error: {error}", file=sys.stderr) - return 1 - - print(f"installed the {SKILL_NAME} skill to {dest}") - return 0 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/src/acumen/agents.py b/src/acumen/agents.py index 2ae4dd8..7ea8271 100644 --- a/src/acumen/agents.py +++ b/src/acumen/agents.py @@ -17,10 +17,12 @@ import asyncio import importlib.util import json +import re import shlex import shutil import signal import sys +import tempfile import time from collections.abc import Callable, Sequence from contextlib import aclosing, suppress @@ -119,6 +121,13 @@ class AgentOptions: #: agent. The shipper sets it False: it is the one agent that runs in the operator's real #: environment on purpose, and it needs the git and ``gh`` credentials that live there. confine: bool = True + #: When set, install a ``PreToolUse`` source guard (:func:`acumen.scrub.make_source_guard` on + #: Claude, its Codex equivalent) that denies fetching the target's own repository or source + #: distribution. ``block_repo`` is the target's remote URL (``None`` for a local target, whose + #: source is not fetchable anyway); ``block_pkg`` is the installed package name. Set only for + #: benchmark runs; meta-agents and ``ship`` leave both ``None``. + block_repo: str | None = None + block_pkg: str | None = None stderr: Callable[[str], None] | None = None #: Prices one provider usage block in USD. Claude enforces ``max_usd`` itself against the #: figure it bills; Codex reports tokens and no dollar amount, so enforcing a budget there @@ -247,21 +256,28 @@ def _claude_hooks(options: AgentOptions) -> dict[str, Any]: alongside containment. """ hooks: dict[str, Any] = {key: list(value) for key, value in (options.claude_hooks or {}).items()} - if not options.confine: - return hooks - - from acumen.guard import containment_hook - - reads, _ = _access_roots(options) - agent_home = options.env.get("HOME") - guard = containment_hook( - reads, - options.deny_paths, - cwd=options.cwd, - home=Path(agent_home) if agent_home else None, - ) - hooks.setdefault("PreToolUse", []) - hooks["PreToolUse"] = [guard, *hooks["PreToolUse"]] + pre = list(hooks.get("PreToolUse", [])) + + if options.block_repo or options.block_pkg: + from acumen.scrub import make_source_guard + + pre.insert(0, make_source_guard(options.block_repo, options.block_pkg)) + + if options.confine: + from acumen.guard import containment_hook + + reads, _ = _access_roots(options) + agent_home = options.env.get("HOME") + guard = containment_hook( + reads, + options.deny_paths, + cwd=options.cwd, + home=Path(agent_home) if agent_home else None, + ) + pre.insert(0, guard) + + if pre: + hooks["PreToolUse"] = pre return hooks @@ -520,7 +536,7 @@ def _codex_command(options: AgentOptions, prompt: str) -> list[str]: "--model", options.model, ] - if options.deny_paths: + if options.deny_paths or options.block_repo or options.block_pkg: command.append("--dangerously-bypass-hook-trust") reads, writes = _access_roots(options) # The Linux command sandbox re-executes the Codex binary through bubblewrap. @@ -531,6 +547,10 @@ def _codex_command(options: AgentOptions, prompt: str) -> list[str]: # evaluates the command path it is given, so allowing /usr/bin does not make # an invocation through the /bin -> /usr/bin symlink readable. reads.extend(path for path in (Path("/bin"), Path("/lib"), Path("/lib64")) if path.exists()) + if options.block_repo or options.block_pkg: + # The source guard imports acumen.scrub; make the package importable inside the sandbox + # regardless of install layout (an editable install lives outside the venv prefix). + reads.append(_acumen_package_root()) reads = list(dict.fromkeys(reads)) command.extend(("-c", 'default_permissions="acumen"')) filesystem = {":root": "deny", ":minimal": "read"} @@ -556,9 +576,26 @@ def _codex_command(options: AgentOptions, prompt: str) -> list[str]: return command -def _codex_guard_source(denied: Sequence[Path]) -> str: - """Return a standalone Codex PreToolUse guard for absolute denied roots.""" +def _acumen_package_root() -> Path: + """Directory containing the importable ``acumen`` package (its parent on ``sys.path``).""" + import acumen + + return Path(acumen.__file__).resolve().parent.parent + + +def _codex_guard_source(denied: Sequence[Path], *, repo: str | None = None, pkg_name: str | None = None) -> str: + """Return a standalone Codex PreToolUse guard. + + Denies two things, either optional: any path resolving under ``denied`` (filesystem + containment), and — via :func:`acumen.scrub.find_source_fetch`, imported at run time because + the guard executes under acumen's own interpreter — any attempt to fetch the target's own + source repository/distribution (``repo``/``pkg_name``). Mirrors the Claude-side + :func:`acumen.scrub.make_source_guard` so both providers enforce the same rule. + """ roots = repr([str(path.resolve()) for path in denied]) + repo_repr = repr(repo) + pkg_repr = repr(pkg_name) + acumen_root_repr = repr(str(_acumen_package_root())) return f"""\ import json import shlex @@ -566,6 +603,8 @@ def _codex_guard_source(denied: Sequence[Path]) -> str: from pathlib import Path ROOTS = [Path(value) for value in {roots}] +REPO = {repo_repr} +PKG = {pkg_repr} def blocked(value, cwd): @@ -606,30 +645,50 @@ def strings(value): yield from strings(item) +def deny(reason): + print(json.dumps({{ + "hookSpecificOutput": {{ + "hookEventName": "PreToolUse", + "permissionDecision": "deny", + "permissionDecisionReason": reason, + }} + }})) + raise SystemExit(0) + + payload = json.load(sys.stdin) cwd = Path(payload.get("cwd") or ".").resolve() -for value in strings(payload.get("tool_input") or {{}}): +tool_input = payload.get("tool_input") or {{}} +for value in strings(tool_input): hit = blocked(value, cwd) if hit is not None: - print(json.dumps({{ - "hookSpecificOutput": {{ - "hookEventName": "PreToolUse", - "permissionDecision": "deny", - "permissionDecisionReason": "acumen blocks access to isolated benchmark data: " + hit, - }} - }})) - raise SystemExit(0) + deny("acumen blocks access to isolated benchmark data: " + hit) + +if REPO or PKG: + if {acumen_root_repr} not in sys.path: + sys.path.insert(0, {acumen_root_repr}) + try: + from acumen.scrub import find_source_fetch + except Exception: + find_source_fetch = None + if find_source_fetch is not None: + hit = find_source_fetch(payload.get("tool_name", ""), tool_input, repo=REPO, pkg_name=PKG) + if hit is not None: + deny( + "acumen benchmarks against the installed package only; fetching the target's " + "source repository or distribution is not permitted: " + str(hit) + ) """ def _install_codex_guard(options: AgentOptions) -> None: """Install a trusted, run-local Codex guard when isolation needs a deny boundary.""" - if not options.deny_paths: + if not options.deny_paths and not (options.block_repo or options.block_pkg): return codex_home = Path(options.env["CODEX_HOME"]) script = codex_home / "acumen_guard.py" script.parent.mkdir(parents=True, exist_ok=True) - script.write_text(_codex_guard_source(options.deny_paths)) + script.write_text(_codex_guard_source(options.deny_paths, repo=options.block_repo, pkg_name=options.block_pkg)) hooks_dir = options.cwd / ".codex" hooks_dir.mkdir(parents=True, exist_ok=True) hooks = { @@ -696,6 +755,54 @@ def _sandbox_failure(lines: Sequence[str]) -> str | None: return None +async def codex_sandbox_probe(env: dict[str, str], *, timeout: float = 30.0) -> str | None: + """Return ``None`` if Codex's command sandbox can start here, else an actionable error string. + + Runs ``codex sandbox -- true`` — no model call, so it is free and deterministic — and reads + its stderr for a namespace/sandbox-init failure. Codex runs every benchmark command inside + this sandbox, so if it cannot start, no Codex run can save an answer; catching it in preflight + turns a paid, per-run ``error_sandbox`` into one free up-front message. + + Conservative on purpose: a probe that cannot run, times out, or exits non-zero for a reason + that is *not* a recognised sandbox failure returns ``None`` (let the run proceed) rather than + block a setup that might work. Only a matched sandbox-init failure is reported. + """ + cli = shutil.which("codex", path=env.get("PATH")) + if cli is None: + return None # a missing CLI is reported by check_agent_cli; nothing to probe here + # Run in a throwaway empty directory with the read-only profile. This tests only the one thing + # that fails on a locked-down host — whether Codex can create its namespace sandbox at all — + # without the workspace profile's git-protection, which mounts a tmpfs over the workspace's + # ``.git`` and errors in a non-repo directory (a false positive unrelated to the real runs, + # which use their own filesystem profile in a temp sandbox dir). + probe_dir = tempfile.mkdtemp(prefix="acumen-codex-probe-") + argv = [cli, "sandbox", "-c", 'default_permissions=":read-only"', "--", "true"] + proc: asyncio.subprocess.Process | None = None + try: + proc = await asyncio.create_subprocess_exec( + *argv, cwd=probe_dir, env=env, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE + ) + _, stderr = await asyncio.wait_for(proc.communicate(), timeout=timeout) + except (OSError, TimeoutError): + if proc is not None: + with suppress(ProcessLookupError): + proc.kill() + return None + finally: + shutil.rmtree(probe_dir, ignore_errors=True) + if proc.returncode == 0: + return None + hit = _sandbox_failure(stderr.decode("utf-8", "replace").splitlines()) + if hit is None: + return None # non-zero for some other reason — not a namespace block + return ( + "codex's command sandbox could not start on this system, so no codex run can execute its " + f"commands or save an answer ({hit}). codex isolates every benchmark command inside an " + "unprivileged namespace sandbox, which this host does not currently permit. Enable " + "unprivileged user namespaces for codex, or run acumen where they are allowed." + ) + + def _is_turn_item(event: dict[str, Any]) -> bool: """Whether ``event`` is one completed model action — the unit Codex turns are counted in.""" if event.get("type") != "item.completed": @@ -918,6 +1025,21 @@ def _codex_terminal( ) +# ``codex exec`` prints this to stderr whenever its stdin is not a TTY (acumen wires it to +# /dev/null), then reads immediate EOF and appends an empty ```` block. It is benign +# but reads as a confusing prompt to anyone watching the run, so drop it before it surfaces. +_CODEX_STDIN_NOTICE = "Reading additional input from stdin" + +# Codex's own ``tracing`` logs, shaped ``Z codex_: ``. +# They report Codex-internal events (rollout writes after a session ends, commands the sandbox +# denies under ``approval_policy="never"``, the model's malformed ``apply_patch`` attempts) that +# are captured in the run transcript anyway and never signal an acumen fault — so they are noise +# on the console, and worse they corrupt the ``\r`` progress bars. Kept in the noise sink for +# sandbox-failure detection, but never echoed. Real bwrap/kernel sandbox errors are bare (no +# ``codex_`` prefix), so they still print and are still detected. +_CODEX_TRACE_RE = re.compile(r"^\d{4}-\d{2}-\d{2}T[\d:.]+Z\s+(?:ERROR|WARN|INFO|DEBUG|TRACE)\s+codex") + + async def _drain_stderr( stream: asyncio.StreamReader, callback: Callable[[str], None] | None, @@ -925,8 +1047,12 @@ async def _drain_stderr( ) -> None: while line := await stream.readline(): text = line.decode(errors="replace").rstrip("\r\n") + if _CODEX_STDIN_NOTICE in text: + continue if sink is not None: sink.append(text) + if _CODEX_TRACE_RE.match(text): + continue if callback is not None: callback(text) else: diff --git a/src/acumen/bench.py b/src/acumen/bench.py index 9e946f4..6d0e6e3 100644 --- a/src/acumen/bench.py +++ b/src/acumen/bench.py @@ -10,12 +10,18 @@ from acumen.agents import AgentProvider, provider_for_model from acumen.config import Config from acumen.env import AuthMode, Target -from acumen.paths import SPLITS, RunKey, Split, arm_name, is_complete, run_dir +from acumen.paths import BENCH_SPLITS, SPLITS, RunKey, Split, arm_name, is_complete, run_dir from acumen.prices import PriceTable from acumen.runner import RunOutcome, run_once from acumen.skills import Skill from acumen.tasks import Task +#: Invalid reasons that stop the rest of a provider's cells for this pass: the credential is +#: empty (``provider_exhausted``) or the network is down (``connection_error``). Both would fail +#: every remaining cell the same way, so the pass raises ``BenchmarkInvalidError`` and the cells +#: stay pending for the next run to resume. ``sandbox_blocked`` is handled separately (pass-scoped). +_PROVIDER_SCOPED_STOP = frozenset({"provider_exhausted", "connection_error"}) + @dataclass(frozen=True) class PlannedRun: @@ -77,13 +83,14 @@ def build_matrix( tasks: Sequence[Task], *, skill: str | None = None, - splits: Iterable[Split] = SPLITS, + splits: Iterable[Split] = BENCH_SPLITS, task_ids: Sequence[str] | None = None, ) -> list[PlannedRun]: """Expand config and tasks into the full list of runs for one arm. - A pass is models x tasks x replicates x splits. Both splits always run; only - train results are ever shown to the improver, and that is enforced downstream. + A pass is models x tasks x replicates x splits. The default benches train and valid; the + held-out ``test`` split is only ever run when asked for explicitly. Only train results are + ever shown to the improver, and that is enforced downstream. Parameters ---------- @@ -275,7 +282,7 @@ async def one(item: PlannedRun) -> RunOutcome | None: if peer is not asyncio.current_task() and not peer.done(): peer.cancel() return outcome - if outcome.reason == "provider_exhausted" and provider not in exhausted: + if outcome.reason in _PROVIDER_SCOPED_STOP and provider not in exhausted: exhausted[provider] = outcome current = asyncio.current_task() # Stop both queued and in-flight siblings for this provider. Tasks for the @@ -314,6 +321,13 @@ async def one(item: PlannedRun) -> RunOutcome | None: return outcomes +#: Budget cap for the auth-probe run. Generous on purpose: the probe is bounded by ``max_turns=5`` +#: and only confirms the credential works, but one turn carries the full ``claude_code`` system +#: prompt, whose input cost at a premium model's rate can exceed a tight cap and fail preflight +#: before the model replies. A trivial "reply ok" never approaches this. +PREFLIGHT_MAX_USD = 2.0 + + async def _preflight_model( model: str, *, @@ -323,7 +337,7 @@ async def _preflight_model( env_passthrough: Sequence[str] | None, ) -> str | None: """Return None if the model can authenticate, else a short error string.""" - from acumen.agents import AgentOptions, run_agent + from acumen.agents import AgentOptions, codex_sandbox_probe, run_agent from acumen.sandbox import sandbox provider = provider_for_model(model) @@ -335,14 +349,27 @@ async def _preflight_model( provider=provider, env_passthrough=env_passthrough, ) as box: + # Codex runs every command inside its own namespace sandbox. A trivial auth probe + # writes no files and runs no command, so it would pass even where that sandbox + # cannot start — and then every real run fails with error_sandbox after paying for + # the tokens. Test the sandbox itself first (free, no model call); report and stop + # here if it cannot start. + if provider == "codex": + sandbox_error = await codex_sandbox_probe(box.env) + if sandbox_error is not None: + return sandbox_error result = await run_agent( "Reply with only the word: ok", options=AgentOptions( cwd=box.root, env=box.env, model=model, + # Bounded by turns, not dollars: the probe only confirms the credential works, + # but one turn carries the full claude_code system prompt, whose input cost at a + # premium model's rate (Opus) exceeds a tight cap — so a dollar cap that low + # fails preflight before the model can reply. max_turns=5 already bounds it. max_turns=5, - max_usd=0.10, + max_usd=PREFLIGHT_MAX_USD, discover_skills=False, confine=False, ), diff --git a/src/acumen/check.py b/src/acumen/check.py index bcab39a..15b2293 100644 --- a/src/acumen/check.py +++ b/src/acumen/check.py @@ -18,7 +18,7 @@ throwaway environment an agent gets. Each one still gets a fresh empty working directory, so its ``answer.md`` cannot be another script's and nothing it writes lands in the project. -The scripts hold the ground truth for the **held-out test split**, so nothing may ever read +The scripts hold the ground truth for the **held-out valid split**, so nothing may ever read them into an agent. That holds today because ``bench``, ``draft`` and ``improve`` confine their agents to explicit read roots (:mod:`acumen.guard`) that never include the project directory. """ diff --git a/src/acumen/cli.py b/src/acumen/cli.py index a7ac24b..9a07153 100644 --- a/src/acumen/cli.py +++ b/src/acumen/cli.py @@ -4,11 +4,15 @@ import argparse import asyncio +import itertools import json +import math import sys import tempfile +import threading import time -from collections.abc import Sequence +from collections.abc import Callable, Sequence +from contextlib import nullcontext from dataclasses import dataclass, replace from datetime import date from pathlib import Path @@ -29,12 +33,12 @@ summarize_checks, ) from acumen.config import Config, ConfigError, load_config -from acumen.draft import DraftError, draft_skill from acumen.env import DEFAULT_CACHE_ROOT, AuthMode, EnvError, prepare_target, resolve_auth_mode +from acumen.epoch import EpochPlan, completed_epochs, resolve_epoch from acumen.grade import INVALID_REASONS from acumen.improve import ImproveError, improve_skill from acumen.logs import LiveLog -from acumen.paths import SPLITS, arm_name +from acumen.paths import BENCH_SPLITS, SPLITS, Split, arm_name from acumen.pricefeed import ( PRICE_SOURCES, PRICE_TIER, @@ -51,9 +55,19 @@ from acumen.runner import RunOutcome, StderrFilter from acumen.scaffold import InitError, is_scaffold_tasks, scaffold from acumen.ship import ShipError, ship_skill -from acumen.skills import Skill, SkillError, available_versions, latest_version, load_skill +from acumen.skills import Skill, SkillError, available_versions, latest_version, load_skill, skill_dir from acumen.taskgen import TaskGenError, generate_tasks from acumen.tasks import Task, TaskError, load_tasks +from acumen.training import ( + EpochRow, + best_version, + build_training_rows, + epochs_since_best, + is_perfect, + patience_exhausted, + write_training_csv, +) +from acumen.wiki import WikiError, collect_arm_runs, update_wiki def _add_bench_args(parser: argparse.ArgumentParser) -> None: @@ -285,6 +299,187 @@ def _fmt_cost(value: float | None) -> str: return f"${value:.2f}" if value is not None else "cost n/a" +def _fmt_rate(value: float | None) -> str: + """Format a 0..1 success rate as a percentage, or ``n/a`` when unknown.""" + if value is None or (isinstance(value, float) and math.isnan(value)): + return "n/a" + return f"{value:.0%}" + + +def _epoch_bar( + done: int, + total: int, + row: EpochRow | None, + *, + best: str | None, + patience: int, + since_best: int, + fixed: bool, +) -> str: + """A tqdm-style one-liner summarising an epoch: bar, version, train/valid success, patience.""" + width = 14 + filled = round(width * done / total) if total else width + bar = "█" * filled + "░" * (width - filled) + version = row.version if row is not None else "?" + train = _fmt_rate(row.train_success) if row is not None else "n/a" + valid = _fmt_rate(row.valid_success) if row is not None else "n/a" + parts = [f"Epoch {done}/{total} |{bar}| {version}", f"train {train}", f"valid {valid}"] + if row is not None and not (isinstance(row.valid_cost, float) and math.isnan(row.valid_cost)): + parts.append(f"${row.valid_cost:.2f}/run") + if not fixed and best is not None: + parts.append(f"(best {best}, patience {min(since_best, patience)}/{patience})") + return " ".join(parts) + + +# Progress rendering for `epoch`/`fit`. Three modes, resolved once per command: +# "bars" — tqdm-style single line rewritten in place with \r (a live terminal) +# "plain" — throttled fresh lines, no \r (output redirected to a file / CI) +# "verbose" — today's per-run scrolling logs, via _Progress (opt-in with --verbose) +_BAR_WIDTH = 14 +_SPINNER_FRAMES = "⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏" + + +def _progress_mode(args: argparse.Namespace) -> str: + """Pick the progress style: explicit --verbose wins, else live bars on a TTY, else plain.""" + if getattr(args, "verbose", False): + return "verbose" + if sys.stdout.isatty() and not getattr(args, "stream", False): + return "bars" + return "plain" + + +def _epoch_header(mode: str, n: int, total: int) -> None: + """Print the marker that starts an epoch — a heavy banner in verbose, a compact line else.""" + if mode == "verbose": + print(f"\n{'═' * 78}\nEPOCH {n}/{total}\n{'═' * 78}", flush=True) + else: + print(f"\nEpoch {n}/{total}", flush=True) + + +def _progress_bar(done: int, total: int, width: int = _BAR_WIDTH) -> str: + filled = width if not total else max(0, min(width, round(width * done / total))) + return "█" * filled + "░" * (width - filled) + + +class _PhaseBar: + r"""A tqdm-style bar for one bench or wiki phase of an epoch. + + Exposes the ``on_start``/``on_done(RunOutcome)`` callbacks ``run_matrix`` expects (bench) and + ``on_wiki_task(TaskWikiResult)`` for the wiki pass. In ``bars`` mode it rewrites one line with + ``\r``; in ``plain`` mode it prints throttled fresh lines so redirected logs stay readable. + (``verbose`` keeps :class:`_Progress` and never builds this, so only those two modes reach here.) + """ + + _THROTTLE_S = 5.0 + + def __init__(self, label: str, total: int, mode: str, *, track_success: bool = True) -> None: + self.label = label + self.total = total + self.mode = mode + self.track_success = track_success + self.done = 0 + self.passed = 0 + self.cost = 0.0 + self._t0 = time.monotonic() + self._last_print = 0.0 + if mode == "bars": + print("\r" + self._line(), end="", flush=True) + + @property + def _elapsed(self) -> float: + return time.monotonic() - self._t0 + + def set_total(self, total: int) -> None: + """Set the denominator once it is known (the wiki pass reports it via ``on_plan``).""" + self.total = total + if self.mode == "bars": + print("\r" + self._line(), end="", flush=True) + + def on_start(self, item) -> None: + """Present so it can be handed to ``run_matrix``; the bar only renders on completion.""" + + def on_done(self, outcome: RunOutcome) -> None: + payload = outcome.payload + priced = payload.get("cost_available", True) and payload.get("cost_usd") is not None + self._surface_error(outcome) + self._tick(success=outcome.success, cost=float(payload["cost_usd"]) if priced else None) + + def on_wiki_task(self, result) -> None: + self._tick(success=None, cost=result.cost_usd) + + def _tick(self, *, success: bool | None, cost: float | None) -> None: + self.done += 1 + if success: + self.passed += 1 + if cost is not None: + self.cost += cost + if self.mode == "bars": + print("\r" + self._line(), end="", flush=True) + elif self.done < self.total and self._elapsed - self._last_print >= self._THROTTLE_S: + self._last_print = self._elapsed + print(" " + self._line(), flush=True) + + def _line(self) -> str: + pct = round(100 * self.done / self.total) if self.total else 100 + parts = [f"{self.label:>14} {pct:>3}%|{_progress_bar(self.done, self.total)}| {self.done}/{self.total}"] + parts.append(f"[{_fmt_secs(self._elapsed)}]") + if self.track_success: + parts.append(f"success {_fmt_rate(self.passed / self.done if self.done else None)} (mean)") + parts.append(f"{_fmt_cost(self.cost)} total") + return " ".join(parts) + + def _surface_error(self, outcome: RunOutcome) -> None: + """Let genuine harness failures through even in bars mode; ordinary test fails just lower the rate.""" + detail = outcome.payload.get("error") + if outcome.reason == "provider_exhausted": + msg = f"provider usage/credit exhausted: {detail or 'no provider detail'}" + elif outcome.reason == "sandbox_blocked": + msg = f"agent sandbox refused an outbound host (harness bug): {detail or 'no sandbox detail'}" + else: + return + print(f"{chr(10) if self.mode == 'bars' else ''}error: {msg}", file=sys.stderr, flush=True) + + def finish(self) -> None: + """Leave the completed bar on screen (bars) or print the final line (plain).""" + print(("\r" if self.mode == "bars" else " ") + self._line(), flush=True) + + +class _Spinner: + """An indeterminate elapsed-time spinner for the single-agent improve step (no sub-progress). + + In ``bars`` mode a daemon thread rewrites the line while ``improve_skill`` blocks the main + thread; other modes just print a start line. Use as a context manager around the blocking call. + """ + + def __init__(self, label: str, mode: str) -> None: + self.label = label + self.mode = mode + self._t0 = time.monotonic() + self._stop = threading.Event() + self._thread: threading.Thread | None = None + + def __enter__(self) -> _Spinner: + if self.mode == "bars": + self._thread = threading.Thread(target=self._spin, daemon=True) + self._thread.start() + else: + print(f" {self.label} ...", flush=True) + return self + + def _spin(self) -> None: + for frame in itertools.cycle(_SPINNER_FRAMES): + if self._stop.is_set(): + break + print(f"\r {self.label} {frame} {_fmt_secs(time.monotonic() - self._t0)}", end="", flush=True) + self._stop.wait(0.1) + + def __exit__(self, *exc: object) -> None: + self._stop.set() + if self._thread is not None: + self._thread.join() + print("\r\033[K", end="", flush=True) # clear the spinner line for the done line + + @dataclass(frozen=True) class _Arm: """One arm of a pass: its version, its loaded skill, and its matrix.""" @@ -317,7 +512,7 @@ def _resolve_arms(cfg: Config, tasks: Sequence[Task], args: argparse.Namespace) arms = [] for version in versions: skill = None if version is None else load_skill(args.skills, version, expect_name=cfg.skill_name) - planned = build_matrix(cfg, tasks, skill=version, splits=args.split or SPLITS, task_ids=args.task) + planned = build_matrix(cfg, tasks, skill=version, splits=args.split or BENCH_SPLITS, task_ids=args.task) todo = pending(planned, args.runs, resume=not args.no_resume) arms.append(_Arm(version=version, skill=skill, planned=planned, todo=todo)) return arms @@ -366,11 +561,16 @@ def _print_run_summary(outcomes: Sequence[RunOutcome], elapsed: float, *, label: print(f"\n{prefix}{passed}/{len(outcomes)} passed in {_fmt_secs(elapsed)} ({cost_summary}, {breakdown})") -def _print_skill_loading(outcomes: Sequence[RunOutcome], arm: _Arm, skill_name: str) -> None: - """Say whether the skill reached the agent — the comparison means nothing otherwise.""" +def _print_skill_loading(outcomes: Sequence[RunOutcome], arm: _Arm, skill_name: str, *, quiet: bool = False) -> None: + """Say whether the skill reached the agent — the comparison means nothing otherwise. + + ``quiet`` drops the routine "loaded in N/M" line (the bars keep phase output compact) but + still raises the warnings, which signal a broken comparison and must never be swallowed. + """ loaded = sum(1 for o in outcomes if o.payload.get("skill_loaded")) if arm.skill is not None: - print(f"skill loaded in {loaded}/{len(outcomes)} runs") + if not quiet: + print(f"skill loaded in {loaded}/{len(outcomes)} runs") if loaded == 0: print( f"warning: {arm.name} never loaded the skill — that arm is not measuring the skill", @@ -380,6 +580,146 @@ def _print_skill_loading(outcomes: Sequence[RunOutcome], arm: _Arm, skill_name: print(f"warning: {skill_name} loaded in {loaded} baseline runs", file=sys.stderr) +def _resolve_bench_auth(models: set[str], auth: str) -> dict[AgentProvider, AuthMode]: + """Resolve one auth mode per provider present, checking each CLI and printing the choice.""" + providers = {provider_for_model(model) for model in models} + auth_modes = {provider: resolve_auth_mode(auth, provider=provider) for provider in providers} + for provider in sorted(providers): + check_agent_cli(provider) + _print_auth(auth_modes[provider], provider) + _warn_codex_accounting(provider) + if "session" in auth_modes.values(): + print( + "note: cost_usd for session-billed runs is what they would have cost at API " + "rates, not metered spend; each run records its auth_mode", + file=sys.stderr, + ) + return auth_modes + + +def _execute_arms( + arms: Sequence[_Arm], + *, + cfg: Config, + target, + runs_root: Path, + auth_modes: dict[AgentProvider, AuthMode], + prices: PriceTable, + keep_sandboxes: bool, + progress: _Progress | _PhaseBar | None = None, + quiet: bool = False, +) -> list[RunOutcome]: + """Run each arm's pending runs sequentially, sharing one progress counter across them. + + Arms run one after another: every run in a matrix shares one skill, and a sequential pass + keeps each arm's tally readable while the progress counter spans the whole thing. ``quiet`` + (epoch/fit bar mode) suppresses the per-arm banner and tally so the phase bar is the only + output; harness warnings still surface. Raises :class:`BenchmarkInvalidError` if a harness + failure decides the pass. + """ + running = [arm for arm in arms if arm.todo] + todo = [item for arm in running for item in arm.todo] + if not todo: + return [] + progress = progress or _Progress(len(todo)) + collected: list[RunOutcome] = [] + for arm in running: + if len(running) > 1 and not quiet: + print(f"\n=== arm {arm.name}: {len(arm.todo)} runs ===", flush=True) + started = time.monotonic() + outcomes = asyncio.run( + run_matrix( + arm.todo, + target=target, + runs_root=runs_root, + max_concurrency=cfg.max_concurrency, + auth_modes=auth_modes, + skill=arm.skill, + skill_name=cfg.skill_name, + keep_sandbox=keep_sandboxes, + stderr=StderrFilter(), + on_start=progress.on_start, + on_done=progress.on_done, + env_passthrough=cfg.env_passthrough, + prices=prices, + ) + ) + collected.extend(outcomes) + if not quiet: + _print_run_summary(outcomes, time.monotonic() - started, label=arm.name if len(running) > 1 else "") + _print_skill_loading(outcomes, arm, cfg.skill_name, quiet=quiet) + return collected + + +def _build_arms( + specs: Sequence[tuple[str | None, Sequence[Split]]], + *, + cfg: Config, + tasks: Sequence[Task], + runs_root: Path, + skills_root: Path, + resume: bool = True, + task_ids: Sequence[str] | None = None, +) -> list[_Arm]: + """Build arms for an explicit list of ``(version, splits)`` specs (for the epoch orchestrator).""" + arms = [] + for version, splits in specs: + skill = None if version is None else load_skill(skills_root, version, expect_name=cfg.skill_name) + planned = build_matrix(cfg, tasks, skill=version, splits=splits, task_ids=task_ids) + todo = pending(planned, runs_root, resume=resume) + arms.append(_Arm(version=version, skill=skill, planned=planned, todo=todo)) + return arms + + +def _run_test_phase( + best: str, + *, + cfg: Config, + tasks: Sequence[Task], + runs_root: Path, + skills_root: Path, + target, + auth_modes: dict[AgentProvider, AuthMode], + prices: PriceTable, + mode: str, +) -> None: + """Bench the held-out test split once, on the best kept version and the baseline. + + Nothing in the fit loop ever benches ``test`` (epochs run only train and valid, and the wiki + and improver read train alone), so this final pass is the skill's first and only contact with + it. Runs resume via ``pending``, so re-running an already-tested fit does nothing. + """ + specs: list[tuple[str | None, Sequence[Split]]] = [(best, ["test"]), (None, ["test"])] + arms = _build_arms(specs, cfg=cfg, tasks=tasks, runs_root=runs_root, skills_root=skills_root) + todo = sum(len(arm.todo) for arm in arms) + if not todo: + return + if mode == "verbose": + print(f"\nheld-out test: benching [{best}] and [{arm_name(None)}] on the test split ...", flush=True) + _print_plan(arms) + bar = None if mode == "verbose" else _PhaseBar("held-out test", todo, mode) + _execute_arms( + arms, + cfg=cfg, + target=target, + runs_root=runs_root, + auth_modes=auth_modes, + prices=prices, + keep_sandboxes=False, + progress=bar, + quiet=mode != "verbose", + ) + if bar is not None: + bar.finish() + + +def _invalid_bench_note() -> None: + print( + "Fix or replenish that credential, then rerun the same command; invalid and cancelled cells remain pending.", + file=sys.stderr, + ) + + def _cmd_bench(args: argparse.Namespace) -> int: cfg = load_config(args.config) tasks = load_tasks(args.tasks) @@ -400,18 +740,7 @@ def _cmd_bench(args: argparse.Namespace) -> int: return 0 # One resolved mode per provider in the matrix, so a mixed pass bills each side correctly. - providers = {provider_for_model(item.model) for item in todo} - auth_modes = {provider: resolve_auth_mode(args.auth, provider=provider) for provider in providers} - for provider in sorted(providers): - check_agent_cli(provider) - _print_auth(auth_modes[provider], provider) - _warn_codex_accounting(provider) - if "session" in auth_modes.values(): - print( - "note: cost_usd for session-billed runs is what they would have cost at API " - "rates, not metered spend; each run records its auth_mode", - file=sys.stderr, - ) + auth_modes = _resolve_bench_auth({item.model for item in todo}, args.auth) # Before the target is built and before any agent runs: an unreachable pricing page # must cost nothing, and a pass must never be priced by a table it cannot date. try: @@ -430,44 +759,23 @@ def _cmd_bench(args: argparse.Namespace) -> int: target = prepare_target(cfg, args.cache, refresh=args.refresh_target) print(f"target ready: {target.fingerprint} @ {target.commit[:8]} (venv {target.venv_dir})", flush=True) - # Arms run one after another: every run in a matrix shares one skill, and a sequential - # pass keeps each arm's tally readable while the progress counter spans the whole thing. running = [arm for arm in arms if arm.todo] print(f"running {len(todo)} runs, up to {cfg.max_concurrency} at a time:", flush=True) progress = _Progress(len(todo)) - collected: list[RunOutcome] = [] try: - for arm in running: - if len(running) > 1: - print(f"\n=== arm {arm.name}: {len(arm.todo)} runs ===", flush=True) - started = time.monotonic() - outcomes = asyncio.run( - run_matrix( - arm.todo, - target=target, - runs_root=args.runs, - max_concurrency=cfg.max_concurrency, - auth_modes=auth_modes, - skill=arm.skill, - skill_name=cfg.skill_name, - keep_sandbox=args.keep_sandboxes, - stderr=StderrFilter(), - on_start=progress.on_start, - on_done=progress.on_done, - env_passthrough=cfg.env_passthrough, - prices=prices, - ) - ) - collected.extend(outcomes) - _print_run_summary(outcomes, time.monotonic() - started, label=arm.name if len(running) > 1 else "") - _print_skill_loading(outcomes, arm, cfg.skill_name) + collected = _execute_arms( + arms, + cfg=cfg, + target=target, + runs_root=args.runs, + auth_modes=auth_modes, + prices=prices, + keep_sandboxes=args.keep_sandboxes, + progress=progress, + ) except BenchmarkInvalidError as err: print(f"\nerror: {err}", file=sys.stderr) - print( - "Fix or replenish that credential, then rerun the same command; invalid and " - "cancelled cells remain pending.", - file=sys.stderr, - ) + _invalid_bench_note() return 2 if len(running) > 1: @@ -476,20 +784,37 @@ def _cmd_bench(args: argparse.Namespace) -> int: return 0 -def _cmd_draft(args: argparse.Namespace) -> int: +def _cmd_improve(args: argparse.Namespace) -> int: cfg = load_config(args.config) + tasks = load_tasks(args.tasks) if args.model: cfg = replace(cfg, meta_model=args.model) - existing = available_versions(args.skills) - if existing and not args.force: + # Parent is an explicit --from, else the latest version, else None (create the first skill + # from the noskill wiki). improve_skill resolves and validates the parent itself. + parent = args.from_version or latest_version(args.skills) + parent_skill = None if parent is None else load_skill(args.skills, parent, expect_name=cfg.skill_name) + # Fail before the costly target prep if there is nothing to learn from: the improver needs the + # parent arm's train runs (and the wiki built from them). + parent_arm = arm_name(parent) + if not collect_arm_runs(args.runs, parent_arm, tasks, split="train"): + hint = ( + "acumen bench --no-skill --split train" + if parent is None + else f"acumen bench --skill {parent} --split train" + ) print( - f"skills already exist ({', '.join(existing)}) — drafting would add " - f"another version. Pass --force to draft anyway, or use `acumen improve` " - f"to build on {existing[-1]}.", + f"no train-split runs found for {parent_arm} under {args.runs / parent_arm / 'train'} — " + f"run `{hint}` and `acumen wiki` first", file=sys.stderr, ) return 2 + if parent_skill is None: + print(f"no skill versions under {args.skills} — creating the first skill from the wiki with {cfg.meta_model}") + else: + print( + f"improving {parent_skill.version} ({parent_skill.name}, {parent_skill.hash[:19]}…) with {cfg.meta_model}" + ) provider = provider_for_model(cfg.meta_model) check_agent_cli(provider) @@ -499,55 +824,127 @@ def _cmd_draft(args: argparse.Namespace) -> int: print(f"preparing target {cfg.repo}@{cfg.ref} ...", flush=True) target = prepare_target(cfg, args.cache, refresh=args.refresh_target) print(f"target ready: {target.fingerprint} @ {target.commit[:8]}", flush=True) - print(f"drafting with {cfg.meta_model} (this reads the package source) ...", flush=True) - log = LiveLog.open(args.log_dir, "draft", stream=args.stream) + log = LiveLog.open(args.log_dir, "improve", stream=args.stream) print(f"log → {log.jsonl_path}", flush=True) with log: result = asyncio.run( - draft_skill( + improve_skill( cfg=cfg, prices=_agent_prices(cfg, model=cfg.meta_model), target=target, skills_root=args.skills, + runs_root=args.runs, + wiki_root=args.wiki, + tasks=tasks, auth_mode=auth_mode, + parent_version=parent, max_turns=args.max_turns, max_usd=args.max_usd, feedback=args.feedback, log=log, ) ) - skill = result.skill - files = sorted(p.relative_to(skill.directory).as_posix() for p in skill.directory.rglob("*") if p.is_file()) - print(f"\nwrote {skill.directory}") - print(f" name: {skill.name}") - print(f" description: {skill.description}") - print(f" hash: {skill.hash}") + new = result.skill + files = sorted(p.relative_to(new.directory).as_posix() for p in new.directory.rglob("*") if p.is_file()) + parent_label = result.parent or "noskill" + print(f"\nwrote {new.directory} (parent {parent_label})") + print(f" name: {new.name}") + print(f" description: {new.description}") + print(f" hash: {new.hash}") print(f" files: {', '.join(files)}") + print(f" evidence: {result.n_train_runs} train runs ({result.n_train_failures} failing)") print(f" cost: {_fmt_cost(result.cost_usd)} over {result.turns} turns") + if parent_skill is not None and new.hash == parent_skill.hash: + print( + "warning: the new version is byte-identical to its parent — the improver changed nothing", + file=sys.stderr, + ) _print_log_result(log) - print(f"\nnext: acumen bench --skill {skill.version}") + print(f"\nnext: acumen bench --skill {new.version} && acumen report") return 0 -def _cmd_improve(args: argparse.Namespace) -> int: +def _wiki_version(args: argparse.Namespace) -> str: + """Resolve the version label a wiki update records, from ``--skill``/``--no-skill``/default.""" + if args.skill is not None: + return args.skill if args.skill.startswith("v") else f"v{args.skill}" + if args.no_skill: + return arm_name(None) + return latest_version(args.skills) or arm_name(None) + + +def _print_wiki_task(result) -> None: + """Print one task's wiki-update outcome as it lands, with any brevity warnings.""" + print(f" wiki [{result.version}] {result.task_id}: {_fmt_cost(result.cost_usd)} over {result.turns} turns") + for warning in result.warnings: + print(f" warning: {warning}", file=sys.stderr) + + +def _run_wiki( + *, + cfg: Config, + tasks: Sequence[Task], + target, + version: str, + runs_root: Path, + wiki_root: Path, + skills_root: Path, + prices: PriceTable, + auth_mode: AuthMode, + log_dir: Path, + stream: bool, + mode: str = "verbose", +): + """Update the wiki for one arm and report progress; returns the task results. + + ``mode`` (``verbose``/``bars``/``plain``) picks the reporting style — a per-task tally in + verbose, a single progress bar otherwise. The bar's denominator is the count of tasks the + pass will actually run, reported once via ``update_wiki``'s ``on_plan`` callback. + """ + bar = None if mode == "verbose" else _PhaseBar("wiki", 0, mode, track_success=False) + if mode == "verbose": + print(f"updating wiki for [{version}] with {cfg.meta_model} (one agent per task) ...", flush=True) + results = asyncio.run( + update_wiki( + cfg=cfg, + target=target, + runs_root=runs_root, + wiki_root=wiki_root, + skills_root=skills_root, + tasks=tasks, + version=version, + prices=prices, + auth_mode=auth_mode, + max_concurrency=cfg.max_concurrency, + log_dir=log_dir, + stream=stream, + on_plan=(bar.set_total if bar is not None else None), + on_task_done=(_print_wiki_task if bar is None else bar.on_wiki_task), + ) + ) + if bar is not None: + bar.finish() + elif not results: + print(f" wiki already had [{version}] for every task — nothing to do") + else: + total = sum(r.cost_usd or 0.0 for r in results) + print(f"wiki: updated {len(results)} task(s) for [{version}] ({_fmt_cost(total)})") + return results + + +def _cmd_wiki(args: argparse.Namespace) -> int: cfg = load_config(args.config) tasks = load_tasks(args.tasks) + if args.max_concurrency: + cfg = replace(cfg, max_concurrency=args.max_concurrency) if args.model: cfg = replace(cfg, meta_model=args.model) - versions = available_versions(args.skills) - if not versions: - print( - f"no skill versions under {args.skills} — run `acumen draft` first, then bench it", - file=sys.stderr, - ) - return 2 - parent = args.from_version or latest_version(args.skills) - # Immutability guard: the improved version is always the next unused directory, - # so an existing version is never in the write path. Say the parent plainly up front. - skill = load_skill(args.skills, parent, expect_name=cfg.skill_name) - print(f"improving {skill.version} ({skill.name}, {skill.hash[:19]}…) with {cfg.meta_model}") + version = _wiki_version(args) + if version != arm_name(None): + # Validate a named skill version exists before the costly target prep. + load_skill(args.skills, version, expect_name=cfg.skill_name) provider = provider_for_model(cfg.meta_model) check_agent_cli(provider) @@ -558,41 +955,352 @@ def _cmd_improve(args: argparse.Namespace) -> int: target = prepare_target(cfg, args.cache, refresh=args.refresh_target) print(f"target ready: {target.fingerprint} @ {target.commit[:8]}", flush=True) - log = LiveLog.open(args.log_dir, "improve", stream=args.stream) - print(f"log → {log.jsonl_path}", flush=True) - with log: - result = asyncio.run( - improve_skill( + _run_wiki( + cfg=cfg, + tasks=tasks, + target=target, + version=version, + runs_root=args.runs, + wiki_root=args.wiki, + skills_root=args.skills, + prices=_agent_prices(cfg, model=cfg.meta_model), + auth_mode=auth_mode, + log_dir=args.log_dir, + stream=args.stream, + ) + print(f"\nwiki written to {args.wiki.resolve()}") + return 0 + + +def _prepare_pass(cfg: Config, args: argparse.Namespace): + """Resolve auth, freeze prices, and build the target once for a pass that spends on benches. + + Shared by ``epoch`` and ``fit`` so a multi-epoch run prepares the target and rates once rather + than per epoch. Raises :class:`PriceFeedError` if rates cannot be established (benchmark cost is + frozen into every result, so a pass must not run without them). + """ + auth_modes = _resolve_bench_auth(set(cfg.models) | {cfg.meta_model}, args.auth) + meta_auth = auth_modes[provider_for_model(cfg.meta_model)] + prices = _bench_prices(cfg) + _warn_unpriced(set(cfg.models) | {cfg.meta_model}, prices) + print(f"preparing target {cfg.repo}@{cfg.ref} ...", flush=True) + target = prepare_target(cfg, args.cache, refresh=args.refresh_target) + print(f"target ready: {target.fingerprint} @ {target.commit[:8]} (venv {target.venv_dir})", flush=True) + return auth_modes, meta_auth, prices, target + + +def _valid_complete_fn(cfg: Config, tasks: list[Task], runs_root: Path) -> Callable[[str], bool]: + """Build the "does this version have a complete ``valid`` bench?" predicate. + + Shared by epoch resolution (``resolve_epoch``/``completed_epochs``) so the two never + disagree on what "done" means. + """ + + def valid_complete(version: str) -> bool: + planned = build_matrix(cfg, tasks, skill=version, splits=["valid"]) + return not pending(planned, runs_root, resume=True) + + return valid_complete + + +def _run_one_epoch( + args: argparse.Namespace, + *, + cfg: Config, + tasks: list[Task], + target, + prices: PriceTable, + auth_modes: dict[AgentProvider, AuthMode], + meta_auth: AuthMode, + mode: str = "verbose", +) -> EpochPlan: + """Run one training epoch on an already-prepared pass, returning the resolved plan. + + The four steps: bench the parent arm on train (the first epoch also benches the noskill + baseline on valid), distil that arm into the wiki, create or improve the skill (skipped when + the version already exists — a resumed epoch), then bench the new version on valid. + + Auth, prices, and the target are passed in, not re-derived, so ``fit`` runs many epochs against + one prepared target. Raises :class:`BenchmarkInvalidError` if a harness failure decides a bench; + the caller reports it. + """ + runs_root = args.runs + + valid_complete = _valid_complete_fn(cfg, tasks, runs_root) + plan = resolve_epoch(args.skills, valid_complete=valid_complete) + parent_label = plan.parent_version or arm_name(None) + tail = " (resuming)" if plan.resumed else "" + print(f"epoch: learning from [{parent_label}] → producing {plan.new_version}{tail}") + + def bench_phase(specs, *, label: str, verbose_banner: str) -> None: + """Run one bench step: a numbered banner + plan + tally in verbose, a phase bar otherwise.""" + if mode == "verbose": + print(verbose_banner, flush=True) + bench_arms = _build_arms(specs, cfg=cfg, tasks=tasks, runs_root=runs_root, skills_root=args.skills) + if mode == "verbose": + _print_plan(bench_arms) + bar = None if mode == "verbose" else _PhaseBar(label, sum(len(a.todo) for a in bench_arms), mode) + _execute_arms( + bench_arms, + cfg=cfg, + target=target, + runs_root=runs_root, + auth_modes=auth_modes, + prices=prices, + keep_sandboxes=False, + progress=bar, + quiet=mode != "verbose", + ) + if bar is not None: + bar.finish() + + # Step 1 — bench the parent arm on the training signal. The first epoch also benches the + # noskill baseline on valid, so the report has an in-epoch baseline to compare against. + if plan.first: + train_specs: list[tuple[str | None, Sequence[Split]]] = [(None, ["train", "valid"])] + else: + train_specs = [(plan.parent_version, ["train"])] + bench_phase( + train_specs, + label="bench baseline" if plan.first else "bench train", + verbose_banner=f"\n[1/4] benchmarking [{parent_label}] on the training signal ...", + ) + + # Step 2 — distil that arm into the wiki (idempotent: recorded arms are skipped). + if mode == "verbose": + print(f"\n[2/4] updating the wiki for [{parent_label}] ...", flush=True) + _run_wiki( + cfg=cfg, + tasks=tasks, + target=target, + version=parent_label, + runs_root=runs_root, + wiki_root=args.wiki, + skills_root=args.skills, + prices=prices, + auth_mode=meta_auth, + log_dir=args.log_dir, + stream=args.stream, + mode=mode, + ) + + # Step 3 — create or improve the skill (skipped when the version already exists: a resumed + # epoch that crashed after improve). + if skill_dir(args.skills, plan.new_version).exists(): + resumed_msg = f"{plan.new_version} already exists — skipping improve (resumed epoch)" + print(f"\n[3/4] {resumed_msg}" if mode == "verbose" else f" {resumed_msg}", flush=True) + else: + verb = "creating" if plan.first else "improving" + log = LiveLog.open(args.log_dir, "improve", stream=args.stream) + if mode == "verbose": + print(f"\n[3/4] {verb} the skill → {plan.new_version} with {cfg.meta_model} ...", flush=True) + print(f"log → {log.jsonl_path}", flush=True) + spinner = nullcontext() if mode == "verbose" else _Spinner(f"inferring skill → {plan.new_version}", mode) + improve_started = time.monotonic() + with log, spinner: + result = asyncio.run( + improve_skill( + cfg=cfg, + prices=prices, + target=target, + skills_root=args.skills, + runs_root=runs_root, + wiki_root=args.wiki, + tasks=tasks, + auth_mode=meta_auth, + parent_version=plan.parent_version, + feedback=args.feedback, + log=log, + ) + ) + improve_elapsed = _fmt_secs(time.monotonic() - improve_started) + new = result.skill + if mode == "verbose": + print(f"wrote {new.directory} (parent {result.parent or 'noskill'})") + print(f" description: {new.description}") + print(f" cost: {_fmt_cost(result.cost_usd)} over {result.turns} turns in {improve_elapsed}") + _print_log_result(log) + else: + print( + f" inferring skill → {new.version} done " + f"({improve_elapsed}, {_fmt_cost(result.cost_usd)}, {result.turns} turns)", + flush=True, + ) + + # Step 4 — bench the new version on the held-out valid signal (completes it if unfinished). + bench_phase( + [(plan.new_version, ["valid"])], + label="bench valid", + verbose_banner=f"\n[4/4] benchmarking {plan.new_version} on the held-out valid signal ...", + ) + return plan + + +def _cmd_epoch(args: argparse.Namespace) -> int: + cfg = load_config(args.config) + tasks = load_tasks(args.tasks) + if args.max_concurrency: + cfg = replace(cfg, max_concurrency=args.max_concurrency) + if args.replicates: + cfg = replace(cfg, n_replicates=args.replicates) + if args.model: + cfg = replace(cfg, meta_model=args.model) + + try: + auth_modes, meta_auth, prices, target = _prepare_pass(cfg, args) + except PriceFeedError as err: + print(f"error: {err}", file=sys.stderr) + print("Retry when the pricing pages are reachable, or pin rates in config.yaml.", file=sys.stderr) + return 2 + + try: + plan = _run_one_epoch( + args, + cfg=cfg, + tasks=tasks, + target=target, + prices=prices, + auth_modes=auth_modes, + meta_auth=meta_auth, + mode=_progress_mode(args), + ) + except BenchmarkInvalidError as err: + print(f"\nerror: {err}", file=sys.stderr) + _invalid_bench_note() + return 2 + + print(f"\nepoch complete: {plan.new_version} produced and benched on valid.") + print("next: `acumen report` to see it, or `acumen epoch` again for another round") + return 0 + + +def _cmd_fit(args: argparse.Namespace) -> int: + cfg = load_config(args.config) + tasks = load_tasks(args.tasks) + if args.max_concurrency: + cfg = replace(cfg, max_concurrency=args.max_concurrency) + if args.replicates: + cfg = replace(cfg, n_replicates=args.replicates) + if args.model: + cfg = replace(cfg, meta_model=args.model) + + fixed = args.epochs is not None + # A GLOBAL target across invocations, not a per-run count: --epochs/--max-epochs is the total + # number of epochs to end up with. It is taken from this invocation's flag and not persisted, + # so re-run with the same flag to keep continuing toward the same target. + total = args.epochs if fixed else args.max_epochs + if total < 1: + print("error: nothing to run — --epochs/--max-epochs must be >= 1", file=sys.stderr) + return 2 + + try: + auth_modes, meta_auth, prices, target = _prepare_pass(cfg, args) + except PriceFeedError as err: + print(f"error: {err}", file=sys.stderr) + print("Retry when the pricing pages are reachable, or pin rates in config.yaml.", file=sys.stderr) + return 2 + + # How many epochs are already fully done on disk — the loop resumes from the next one. + completed = completed_epochs(args.skills, valid_complete=_valid_complete_fn(cfg, tasks, args.runs)) + remaining = max(0, total - completed) + prior = [row for row in build_training_rows(args.runs, cfg) if row.epoch <= completed] if completed else [] + stopping = "early stopping disabled" if fixed else f"early stopping with patience {args.patience}" + if completed: + print(f"fit: target {total} epoch(s) total ({stopping}); {completed} done, {remaining} to go") + # Show the progress of the epochs already finished (skip any in-progress/partial version). + prior_best = best_version(prior) + print(f"\nresuming: {completed} epoch(s) already complete (target {total})") + for index, row in enumerate(prior): + since = epochs_since_best([r.valid_success for r in prior[: index + 1]]) + print( + _epoch_bar( + row.epoch, total, row, best=prior_best, patience=args.patience, since_best=since, fixed=fixed + ) + ) + else: + print(f"fit: target {total} epoch(s) total ({stopping})") + + # A finished epoch that already hit 100% means there is nothing left to gain — do not resume + # into more epochs. Mirrors the in-loop is_perfect stop, which never sees a prior epoch. + already_perfect = next((row for row in prior if is_perfect(row.valid_success)), None) + if already_perfect is not None: + print(f"\nearly stop: validation success already reached 100% at {already_perfect.version}.") + elif remaining == 0: + print(f"\nnothing to do: already at {completed} epoch(s) (target {total}).") + + mode = _progress_mode(args) + epochs = range(completed + 1, total + 1) if already_perfect is None else range(0) + for epoch_no in epochs: + _epoch_header(mode, epoch_no, total) + try: + plan = _run_one_epoch( + args, cfg=cfg, - prices=_agent_prices(cfg, model=cfg.meta_model), + tasks=tasks, target=target, - skills_root=args.skills, - runs_root=args.runs, + prices=prices, + auth_modes=auth_modes, + meta_auth=meta_auth, + mode=mode, + ) + except BenchmarkInvalidError as err: + print(f"\nerror: {err}", file=sys.stderr) + _invalid_bench_note() + return 2 + + # Rebuild the whole training curve from runs/ each epoch, so the CSV is always consistent + # with what exists and a resumed fit produces the same table. + rows = build_training_rows(args.runs, cfg) + write_training_csv(rows, args.out) + current = next((row for row in rows if row.version == plan.new_version), None) + valids = [row.valid_success for row in rows] + best = best_version(rows) + since = epochs_since_best(valids) + print(_epoch_bar(epoch_no, total, current, best=best, patience=args.patience, since_best=since, fixed=fixed)) + + # A perfect validation score leaves nothing to gain — stop even under a fixed --epochs. + if current is not None and is_perfect(current.valid_success): + print(f"\nearly stop: validation success reached 100% at {current.version}.") + break + + if not fixed and patience_exhausted(valids, args.patience): + print(f"\nearly stop: validation mean success did not improve in {args.patience} epoch(s).") + break + + rows = build_training_rows(args.runs, cfg) + best = best_version(rows) + if best is not None: + try: + _run_test_phase( + best, + cfg=cfg, tasks=tasks, - auth_mode=auth_mode, - parent_version=parent, - max_turns=args.max_turns, - max_usd=args.max_usd, - feedback=args.feedback, - log=log, + runs_root=args.runs, + skills_root=args.skills, + target=target, + auth_modes=auth_modes, + prices=prices, + mode=mode, ) - ) - new = result.skill - files = sorted(p.relative_to(new.directory).as_posix() for p in new.directory.rglob("*") if p.is_file()) - print(f"\nwrote {new.directory} (parent {result.parent})") - print(f" name: {new.name}") - print(f" description: {new.description}") - print(f" hash: {new.hash}") - print(f" files: {', '.join(files)}") - print(f" evidence: {result.n_train_runs} train runs ({result.n_train_failures} failing)") - print(f" cost: {_fmt_cost(result.cost_usd)} over {result.turns} turns") - if new.hash == skill.hash: - print( - "warning: the new version is byte-identical to its parent — the improver changed nothing", - file=sys.stderr, - ) - _print_log_result(log) - print(f"\nnext: acumen bench --skill {new.version} && acumen report") + except BenchmarkInvalidError as err: + print(f"\nerror: {err}", file=sys.stderr) + _invalid_bench_note() + return 2 + best_row = next(row for row in rows if row.version == best) + print(f"\nfit complete: best version {best} (valid {_fmt_rate(best_row.valid_success)}).") + else: + print("\nfit complete.") + print(f"training curve → {args.out.resolve()}") + + # A fit's whole point is the comparison, so build the report without being asked. It is a + # summary of what already ran, so a failure here must not fail the fit itself. + report_out = Path("report.html") + skills_root = args.skills if args.skills.is_dir() else None + try: + build_report(args.runs, report_out, tasks, skills_root=skills_root) + print(f"report → {report_out.resolve()}") + except ReportError as err: + print(f"note: report skipped — {err}", file=sys.stderr) return 0 @@ -668,7 +1376,7 @@ def _cmd_tasks(args: argparse.Namespace) -> int: "and split, so it was not kept", file=sys.stderr, ) - print("\nnext: review the tasks, then `acumen check`, `acumen draft` and `acumen bench`") + print("\nnext: review the tasks, then `acumen check`, then `acumen fit`") return 0 @@ -1082,7 +1790,7 @@ def _cmd_init(args: argparse.Namespace) -> int: written = scaffold(args.directory, force=args.force) for path in written: print(f"wrote {path}") - print("\nnext: edit config.yaml (repo) and tasks.yaml, then `acumen draft`") + print("\nnext: edit config.yaml (repo) and tasks.yaml, then `acumen fit`") return 0 @@ -1167,25 +1875,18 @@ def build_parser() -> argparse.ArgumentParser: _add_bench_args(bench) bench.set_defaults(func=_cmd_bench) - draft = sub.add_parser("draft", help="draft a skill from the target package's source") - draft.add_argument("--config", type=Path, default=Path("config.yaml"), help="path to config.yaml") - draft.add_argument("--skills", type=Path, default=Path("skills"), help="root of the skill tree") - draft.add_argument("--model", help="override config meta_model") - draft.add_argument("--max-turns", type=int, help="cap turns for the drafting agent (default: unbounded)") - draft.add_argument("--max-usd", type=float, help="cap spend for the drafting agent (default: unbounded)") - draft.add_argument("--cache", type=Path, default=DEFAULT_CACHE_ROOT, help="target cache root") - draft.add_argument("--refresh-target", action="store_true", help="rebuild the target checkout and venv") - draft.add_argument("--force", action="store_true", help="draft another version even if some already exist") - _add_auth_arg(draft) - _add_feedback_arg(draft, extra=" (e.g. package context, what the skill should emphasise)") - _add_log_args(draft) - draft.set_defaults(func=_cmd_draft) - - improve = sub.add_parser("improve", help="improve the current skill into a new version from its train results") + improve = sub.add_parser( + "improve", + help="create or improve the skill from the knowledge wiki", + description="Read the knowledge wiki (per-task observations/hypotheses distilled from train " + "runs) and the filtered package source, then create the first skill (when none exist) or " + "improve the latest into the next version. The held-out valid split is never reachable.", + ) improve.add_argument("--config", type=Path, default=Path("config.yaml"), help="path to config.yaml") improve.add_argument("--tasks", type=Path, default=Path("tasks.yaml"), help="path to tasks.yaml") improve.add_argument("--skills", type=Path, default=Path("skills"), help="root of the skill tree") improve.add_argument("--runs", type=Path, default=Path("runs"), help="root of the run tree") + improve.add_argument("--wiki", type=Path, default=Path("wiki"), help="root of the knowledge wiki") improve.add_argument("--from", dest="from_version", metavar="VERSION", help="version to improve (default: latest)") improve.add_argument("--model", help="override config meta_model") improve.add_argument("--max-turns", type=int, help="cap turns for the improving agent (default: unbounded)") @@ -1195,11 +1896,92 @@ def build_parser() -> argparse.ArgumentParser: _add_auth_arg(improve) _add_feedback_arg( improve, - extra=" (e.g. what to fix or emphasise; do NOT paste test-split answers — that defeats the held-out split)", + extra=" (e.g. what to fix or emphasise; do NOT paste valid-split answers — that defeats the held-out split)", ) _add_log_args(improve) improve.set_defaults(func=_cmd_improve) + wiki = sub.add_parser( + "wiki", + help="distil an arm's train runs into the knowledge wiki (one agent per task)", + description="For each task, read that arm's train-split runs across models and replicates " + "and append a terse [version][model] block to wiki//observations.md and hypothesis.md. " + "Cumulative and idempotent: an arm already recorded for a task is skipped.", + ) + wiki.add_argument("--config", type=Path, default=Path("config.yaml"), help="path to config.yaml") + wiki.add_argument("--tasks", type=Path, default=Path("tasks.yaml"), help="path to tasks.yaml") + wiki.add_argument("--runs", type=Path, default=Path("runs"), help="root of the run tree") + wiki.add_argument("--wiki", type=Path, default=Path("wiki"), help="root of the knowledge wiki") + wiki.add_argument("--skills", type=Path, default=Path("skills"), help="root of the skill tree") + wiki_arm = wiki.add_mutually_exclusive_group() + wiki_arm.add_argument("--no-skill", action="store_true", help="record the baseline (noskill) arm") + wiki_arm.add_argument("--skill", metavar="VERSION", help="record one skill version, e.g. v1 (default: latest)") + wiki.add_argument("--model", help="override config meta_model") + wiki.add_argument("--max-concurrency", type=int, help="override config max_concurrency") + wiki.add_argument("--cache", type=Path, default=DEFAULT_CACHE_ROOT, help="target cache root") + wiki.add_argument("--refresh-target", action="store_true", help="rebuild the target checkout and venv") + _add_auth_arg(wiki) + _add_log_args(wiki) + wiki.set_defaults(func=_cmd_wiki) + + epoch = sub.add_parser( + "epoch", + help="run one training epoch: bench train, update wiki, improve, bench valid", + description="One training epoch end to end: bench the current arm on the training signal, " + "distil it into the wiki, create/improve the skill, then bench the new version on the " + "held-out valid signal. Fully resumable — re-run to continue a crashed epoch or start the " + "next one.", + ) + epoch.add_argument("--config", type=Path, default=Path("config.yaml"), help="path to config.yaml") + epoch.add_argument("--tasks", type=Path, default=Path("tasks.yaml"), help="path to tasks.yaml") + epoch.add_argument("--runs", type=Path, default=Path("runs"), help="root of the run tree") + epoch.add_argument("--skills", type=Path, default=Path("skills"), help="root of the skill tree") + epoch.add_argument("--wiki", type=Path, default=Path("wiki"), help="root of the knowledge wiki") + epoch.add_argument("--model", help="override config meta_model") + epoch.add_argument("--max-concurrency", type=int, help="override config max_concurrency") + epoch.add_argument("--replicates", type=int, help="override config n_replicates") + epoch.add_argument("--cache", type=Path, default=DEFAULT_CACHE_ROOT, help="target cache root") + epoch.add_argument("--refresh-target", action="store_true", help="rebuild the target checkout and venv") + epoch.add_argument("--verbose", action="store_true", help="use the full scrolling logs instead of progress bars") + _add_auth_arg(epoch) + _add_feedback_arg(epoch, extra=" (passed to the improver; do NOT paste valid-split answers)") + _add_log_args(epoch) + epoch.set_defaults(func=_cmd_epoch) + + fit = sub.add_parser( + "fit", + help="run many training epochs with early stopping (like training a model)", + description="Run `acumen epoch` back to back until validation stops improving (patience) or " + "a hard cap is hit, writing a per-epoch training curve to training.csv and a progress line " + "each epoch. Fully resumable — re-run to continue.", + ) + fit.add_argument("--config", type=Path, default=Path("config.yaml"), help="path to config.yaml") + fit.add_argument("--tasks", type=Path, default=Path("tasks.yaml"), help="path to tasks.yaml") + fit.add_argument("--runs", type=Path, default=Path("runs"), help="root of the run tree") + fit.add_argument("--skills", type=Path, default=Path("skills"), help="root of the skill tree") + fit.add_argument("--wiki", type=Path, default=Path("wiki"), help="root of the knowledge wiki") + fit.add_argument("--out", type=Path, default=Path("training.csv"), help="training-curve CSV to write") + fit.add_argument( + "--patience", type=int, default=2, help="stop after this many epochs with no validation gain (default: 2)" + ) + fit.add_argument("--max-epochs", type=int, default=10, help="hard cap on epochs (default: 10)") + fit.add_argument( + "--epochs", + type=int, + default=None, + help="run exactly this many epochs (disables early stopping and ignores --max-epochs)", + ) + fit.add_argument("--model", help="override config meta_model") + fit.add_argument("--max-concurrency", type=int, help="override config max_concurrency") + fit.add_argument("--replicates", type=int, help="override config n_replicates") + fit.add_argument("--cache", type=Path, default=DEFAULT_CACHE_ROOT, help="target cache root") + fit.add_argument("--refresh-target", action="store_true", help="rebuild the target checkout and venv") + fit.add_argument("--verbose", action="store_true", help="use the full scrolling logs instead of progress bars") + _add_auth_arg(fit) + _add_feedback_arg(fit, extra=" (passed to the improver; do NOT paste valid-split answers)") + _add_log_args(fit) + fit.set_defaults(func=_cmd_fit) + tasks_cmd = sub.add_parser("tasks", help="autonomously generate a tasks.yaml from the target package") tasks_cmd.add_argument("--config", type=Path, default=Path("config.yaml"), help="path to config.yaml") tasks_cmd.add_argument("--out", type=Path, default=Path("tasks.yaml"), help="tasks.yaml to write") @@ -1325,8 +2107,8 @@ def main(argv: list[str] | None = None) -> int: AgentError, PriceFeedError, SkillError, - DraftError, ImproveError, + WikiError, TaskGenError, ReviewError, ShipError, diff --git a/src/acumen/draft.py b/src/acumen/draft.py deleted file mode 100644 index b77db42..0000000 --- a/src/acumen/draft.py +++ /dev/null @@ -1,235 +0,0 @@ -"""The drafting agent: write ``skills/v1/`` from the target package's source. - -Unlike a benchmark agent, the drafter reads the target's source — it is documenting -the package, so it needs to see it. It is otherwise held to the same isolation: scrubbed -env, throwaway config dir, no user settings or memories. - -What it does *not* see is agent guidance the target already ships. A skill drafted from the -maintainer's own skill is not the independent artifact the benchmark then reports on: the arms -would be comparing that skill, re-served, against no skill at all. So the drafter reads a -filtered copy of the checkout with skills and agent-instruction files stripped -(:func:`acumen.scrub.build_filtered_source`) and is denied the original tree, exactly as -``tasks`` is. The real checkout is never modified. - -The agent writes into a staging directory, not into ``skills/`` directly. Only a skill -that loads and validates is promoted to a version, so a failed or half-finished draft -never leaves a broken ``skills/vN/`` behind — versions are immutable, which means -they must be right the moment they appear. -""" - -from __future__ import annotations - -import shutil -import tempfile -from dataclasses import dataclass -from pathlib import Path - -from acumen.agents import AgentOptions, AgentResult, provider_for_model, run_agent -from acumen.config import Config -from acumen.env import AuthMode, Target, build_agent_env -from acumen.logs import LiveLog -from acumen.prices import PriceTable, price_usage, pricer, resolve_cost -from acumen.procs import label_env, reap -from acumen.prompts import draft_prompt -from acumen.scrub import build_filtered_source, make_skill_guard -from acumen.skills import ( - SKILL_FILE, - Skill, - SkillError, - load_skill, - next_version, - skill_dir, - write_meta, -) - - -class DraftError(RuntimeError): - """Raised when a skill could not be drafted.""" - - -@dataclass(frozen=True) -class DraftResult: - """A drafted skill version and what it cost to produce.""" - - skill: Skill - cost_usd: float | None - turns: int - #: Live log paths for this run, when a :class:`LiveLog` was attached. - log_jsonl: Path | None = None - log_html: Path | None = None - - -def _validate_staged(staging: Path, skill_name: str) -> None: - """Fail loudly if the agent's output isn't a usable skill, before it becomes a version.""" - if not (staging / SKILL_FILE).is_file(): - raise DraftError( - f"the drafting agent did not write {SKILL_FILE} — nothing to promote. Inspect the run log or the prompt." - ) - # load_skill enforces the frontmatter contract (name matches, description present). - try: - load_skill(staging.parent, staging.name, expect_name=skill_name) - except SkillError as err: - raise DraftError(f"the drafted skill is not valid: {err}") from err - - -async def draft_skill( - *, - cfg: Config, - target: Target, - skills_root: Path, - auth_mode: AuthMode = "session", - prices: PriceTable | None = None, - model: str | None = None, - max_turns: int | None = None, - max_usd: float | None = None, - rationale: str = "initial draft", - feedback: str | None = None, - log: LiveLog | None = None, -) -> DraftResult: - """Draft a new skill version from the target package's source. - - Parameters - ---------- - cfg - The pass config; supplies ``skill_name`` and ``meta_model``. - target - The prepared target, supplying the source checkout and interpreter. - skills_root - The ``skills/`` root. The new version is the next unused one. - auth_mode - Which credential the drafting agent authenticates with — ``"session"`` (the Claude - subscription) or ``"api"`` (see :func:`acumen.env.build_agent_env`). - model - Override for the drafting model; defaults to the config's. - max_turns, max_usd - Turn and budget caps for the drafting agent. **Unbounded by default** — the config's - ``max_turns``/``max_usd`` cap benchmark agents only; pass explicit caps to bound the - drafter. - rationale - Recorded in ``meta.json`` as why this version exists. - feedback - Optional maintainer guidance, injected into the draft prompt as a subordinated block - and recorded in ``meta.json`` as provenance. - log - A :class:`LiveLog` to stream the agent's messages to and render an HTML log from. - - Returns - ------- - The drafted skill, loaded and validated. - """ - version = next_version(skills_root) - dest = skill_dir(skills_root, version) - if dest.exists(): - raise DraftError(f"{dest} already exists — skill versions are immutable and never overwritten") - - holder = Path(tempfile.mkdtemp(prefix="acumen-draft-")) - try: - # The staging dir is named for the version so load_skill can validate it in place. - work = holder / "work" - staging = work / version - home = holder / "home" - selected_model = model or cfg.meta_model - table = prices if prices is not None else PriceTable(overrides=cfg.prices) - provider = provider_for_model(selected_model) - config_dir = home / (".claude" if provider == "claude" else ".codex") - for path in (staging, home, config_dir, home / "tmp"): - path.mkdir(parents=True, exist_ok=True) - - # Marks the agent's processes so the teardown below can find what it leaves running. - env = label_env( - build_agent_env( - config_dir=config_dir, - home=home, - extra_path=[target.bin_dir], - auth_mode=auth_mode, - extra_allow=cfg.env_passthrough, - provider=provider, - ), - holder, - ) - - # A copy of the source with the target's own skills and agent guidance stripped. The - # drafter reads this, never the real checkout. - source_copy = build_filtered_source(target.src_dir, holder / "source") - - prompt = draft_prompt( - package=target.pkg_name, - version=target.pkg_version, - src=source_copy, - python=target.python, - out=staging, - skill_name=cfg.skill_name, - feedback=feedback, - ) - options = AgentOptions( - cwd=work, - env=env, - model=selected_model, - # No default turn or budget cap: only bound the agent if the caller asked. The - # config's ``max_turns``/``max_usd`` cap benchmark agents only. - max_turns=max_turns, - max_usd=max_usd, - # Codex reports no billed figure, so a budget cap needs the run's own rate table. - price_usd=pricer(selected_model, table), - # The drafter reads the target source; benchmark agents never do. It points at the - # *filtered* copy, not the real checkout. - read_dirs=(source_copy, target.venv_dir), - write_dirs=(work,), - discover_skills=True, - # Belt-and-braces over the filtered copy: deny any call that reaches an existing - # skill/guidance artifact or the original unfiltered source, wherever pointed. Built - # only for Claude — the hook is an SDK object, and Codex gets ``deny_paths`` below. - claude_hooks={"PreToolUse": [make_skill_guard(target.src_dir)]} if provider == "claude" else None, - # Codex reads the filtered copy and is denied the original checkout. - deny_paths=(target.src_dir.resolve(),), - ) - - result: AgentResult | None = None - agent_error: Exception | None = None - try: - result = await run_agent( - prompt, - options=options, - on_event=log.append if log is not None else None, - ) - except Exception as err: # noqa: BLE001 - a failed draft is an error to report, re-raised below - agent_error = err - finally: - # Render the HTML log while the throwaway config dir still holds the native - # transcript — in a finally so an aborted run (the SDK raises on a cap breach, - # after yielding the result) is still inspectable. - if log is not None: - log.finalize(config_dir=config_dir, work_dir=work, result=result) - - if agent_error is not None: - raise DraftError(f"the drafting agent failed: {type(agent_error).__name__}: {agent_error}") from agent_error - if result is None: - raise DraftError("the drafting agent produced no result message") - if result.is_error: - raise DraftError(f"the drafting agent errored: {result.subtype} {result.errors or ''}".strip()) - - _validate_staged(staging, cfg.skill_name) - - dest.parent.mkdir(parents=True, exist_ok=True) - shutil.copytree(staging, dest) - write_meta(dest, parent=None, rationale=rationale, feedback=feedback) - skill = load_skill(skills_root, version, expect_name=cfg.skill_name) - return DraftResult( - skill=skill, - cost_usd=resolve_cost( - result.total_cost_usd, - price_usage( - result.usage, - model=selected_model, - provider=result.provider, - prices=table, - ), - ).cost_usd, - turns=result.num_turns, - log_jsonl=log.jsonl_path if log is not None else None, - log_html=log.html_path if log is not None and log.html_rendered else None, - ) - finally: - # Kill anything the agent left running before removing the directory it runs in. - reap(holder) - shutil.rmtree(holder, ignore_errors=True) diff --git a/src/acumen/env.py b/src/acumen/env.py index 536e104..bafee3b 100644 --- a/src/acumen/env.py +++ b/src/acumen/env.py @@ -99,6 +99,13 @@ class Target: pkg_name: str pkg_version: str + @property + def is_remote(self) -> bool: + """Whether ``source`` is a remote URL (fetchable) rather than a local path.""" + from acumen.config import _looks_remote + + return _looks_remote(self.source) + @property def bin_dir(self) -> Path: """The venv's ``bin`` directory — what goes on an agent's PATH.""" diff --git a/src/acumen/epoch.py b/src/acumen/epoch.py new file mode 100644 index 0000000..4a47042 --- /dev/null +++ b/src/acumen/epoch.py @@ -0,0 +1,73 @@ +"""Resolve which training epoch an ``acumen epoch`` invocation is on, from on-disk state. + +An epoch benches the current arm on ``train``, distils it into the wiki, produces the next skill +version, and benches that version on ``valid``. The resolver here decides — purely from what is on +disk — whether a fresh invocation *starts* a new epoch or *resumes* one that crashed partway, so a +crash between "improve wrote the version" and "valid bench finished" re-enters the SAME epoch and +completes it rather than starting another. +""" + +from __future__ import annotations + +from collections.abc import Callable +from dataclasses import dataclass + +from acumen.skills import available_versions, next_version + + +@dataclass(frozen=True) +class EpochPlan: + """What one ``acumen epoch`` invocation will do, resolved from state.""" + + #: The arm learned from this epoch: a skill version, or ``None`` for the ``noskill`` baseline. + parent_version: str | None + #: The version this epoch produces, e.g. ``"v1"`` or ``"v2"``. + new_version: str + #: Whether this is the first epoch (``parent_version is None``), which also benches the + #: ``noskill`` baseline on ``valid`` for the report. + first: bool + #: Whether ``new_version`` already exists on disk (a resumed epoch), so ``improve`` is skipped. + resumed: bool + + +def resolve_epoch(skills_root, *, valid_complete: Callable[[str], bool]) -> EpochPlan: + """Decide the epoch to run from the skills on disk and each version's valid-bench state. + + Parameters + ---------- + skills_root + The ``skills/`` root. + valid_complete + Predicate: does this version have a complete ``valid`` bench? Injected so the resolver + stays pure and unit-testable (the caller builds it from ``build_matrix``/``pending``). + + Returns + ------- + The resolved :class:`EpochPlan`. + + Notes + ----- + - No versions yet → first epoch: ``parent=None`` (noskill), ``new=v1``. + - Latest version exists but its valid bench is unfinished → resume that epoch: ``new=latest``, + ``parent`` is the version below it (or ``None`` when latest is ``v1``), ``resumed=True``. + - Otherwise → fresh epoch: ``parent=latest``, ``new=next``. + """ + versions = available_versions(skills_root) + latest = versions[-1] if versions else None + + if latest is not None and not valid_complete(latest): + parent = versions[-2] if len(versions) >= 2 else None + return EpochPlan(parent_version=parent, new_version=latest, first=parent is None, resumed=True) + + new = next_version(skills_root) + return EpochPlan(parent_version=latest, new_version=new, first=latest is None, resumed=False) + + +def completed_epochs(skills_root, *, valid_complete: Callable[[str], bool]) -> int: + """How many epochs are fully finished — versions on disk whose ``valid`` bench is complete. + + Used to resume ``fit`` toward a global epoch target: the next epoch to run is + ``completed_epochs(...) + 1``. Takes the same injected ``valid_complete`` predicate as + :func:`resolve_epoch`, so it stays pure and unit-testable. + """ + return sum(1 for version in available_versions(skills_root) if valid_complete(version)) diff --git a/src/acumen/grade.py b/src/acumen/grade.py index d7d5858..c6c5f39 100644 --- a/src/acumen/grade.py +++ b/src/acumen/grade.py @@ -26,12 +26,14 @@ "error", "provider_exhausted", "sandbox_blocked", + "connection_error", ] #: Reasons that mark a run as a harness failure rather than evidence about a model. A run #: with one of these is recorded for diagnosis but must never reach a report, an improve -#: pass, or a resume as though it measured something. -INVALID_REASONS: frozenset[Reason] = frozenset({"provider_exhausted", "sandbox_blocked"}) +#: pass, or a resume as though it measured something — and it stays pending so a later run +#: retries it (``connection_error`` is a transient network drop, not the model's fault). +INVALID_REASONS: frozenset[Reason] = frozenset({"provider_exhausted", "sandbox_blocked", "connection_error"}) _FENCE_RE = re.compile(r"^\s*```[^\n]*\n(?P.*?)\n?\s*```\s*$", re.DOTALL) _HEADER_LINE_RE = re.compile(r"^\s*#+\s") diff --git a/src/acumen/htmldiff.py b/src/acumen/htmldiff.py new file mode 100644 index 0000000..a7fb1d5 --- /dev/null +++ b/src/acumen/htmldiff.py @@ -0,0 +1,144 @@ +"""A side-by-side (split) HTML diff renderer, shared across acumen's HTML outputs. + +The report diffs each skill version against its parent; the transcript diffs a file edit's before +against its after. Both want the same thing — old on the left, new on the right, changed words +lit up — so the renderer lives here and neither page owns it. It is pure stdlib +(``difflib``/``re``/``html``) so the transcript can use it without importing the report (and its +matplotlib/pandas weight). The styling lives in :data:`acumen.theme.DIFF_CSS`; this module emits +only class names. +""" + +from __future__ import annotations + +import difflib +import html +import re +from dataclasses import dataclass + +#: Unchanged lines kept on either side of a change, as in ``diff -u``. Longer untouched runs +#: collapse to a single marker row, so a change reads as only what actually moved. +_DIFF_CONTEXT = 3 + +#: Below this line-similarity the two sides of a replaced row are treated as unrelated text, +#: so the row is tinted whole instead of being picked apart into confetti-sized highlights. +_DIFF_INLINE_RATIO = 0.5 + + +@dataclass(frozen=True) +class _DiffRow: + """One row of a split diff: the same logical line on each side, either side possibly absent. + + ``kind`` is the change type from :class:`difflib.SequenceMatcher` (``equal``, ``replace``, + ``delete``, ``insert``), plus ``gap`` for the marker standing in for an elided run of + unchanged lines. A side is ``None`` where that version has no line there at all — a pure + insertion has no left-hand text — and renders as an inert filler cell. + """ + + kind: str + left_no: int | None + left: str | None + right_no: int | None + right: str | None + + +def _equal_rows(before: list[str], after: list[str], i: int, j: int, count: int) -> list[_DiffRow]: + """``count`` rows of unchanged text, starting at line ``i`` on the left and ``j`` on the right.""" + return [_DiffRow("equal", i + k + 1, before[i + k], j + k + 1, after[j + k]) for k in range(count)] + + +def _split_diff_rows(before: list[str], after: list[str]) -> list[_DiffRow]: + """Align two versions of a file into side-by-side rows, old on the left, new on the right. + + Changed runs pair off line by line, so a rewritten paragraph sits opposite its replacement + rather than being stacked below it; where one side runs out, the other continues against + filler. Unchanged stretches beyond :data:`_DIFF_CONTEXT` lines from any change collapse to + a gap row. + """ + rows: list[_DiffRow] = [] + opcodes = difflib.SequenceMatcher(a=before, b=after, autojunk=False).get_opcodes() + for index, (tag, i1, i2, j1, j2) in enumerate(opcodes): + if tag == "equal": + head = _DIFF_CONTEXT if index > 0 else 0 + tail = _DIFF_CONTEXT if index < len(opcodes) - 1 else 0 + if i2 - i1 > head + tail: + rows += _equal_rows(before, after, i1, j1, head) + rows.append(_DiffRow("gap", None, None, None, None)) + rows += _equal_rows(before, after, i2 - tail, j2 - tail, tail) + else: + rows += _equal_rows(before, after, i1, j1, i2 - i1) + continue + left, right = before[i1:i2], after[j1:j2] + for k in range(max(len(left), len(right))): + has_left, has_right = k < len(left), k < len(right) + rows.append( + _DiffRow( + tag, + i1 + k + 1 if has_left else None, + left[k] if has_left else None, + j1 + k + 1 if has_right else None, + right[k] if has_right else None, + ) + ) + return rows + + +def _inline_pair(left: str, right: str) -> tuple[str, str]: + """Both sides of a replaced line, escaped, with the words that differ wrapped in ````. + + The comparison runs over words rather than characters, so a changed word lights up whole + instead of down to the letters it happens to share with its replacement. Lines too + dissimilar to be a rewrite of one another are left unmarked — highlighting nearly every + word says less than the row tint already does. + """ + a, b = re.findall(r"\w+|\W", left), re.findall(r"\w+|\W", right) + matcher = difflib.SequenceMatcher(a=a, b=b, autojunk=False) + if matcher.ratio() < _DIFF_INLINE_RATIO: + return html.escape(left), html.escape(right) + marked = ["", ""] + for tag, i1, i2, j1, j2 in matcher.get_opcodes(): + for side, tokens, start, end in ((0, a, i1, i2), (1, b, j1, j2)): + chunk = html.escape("".join(tokens[start:end])) + if chunk: + marked[side] += chunk if tag == "equal" else f"{chunk}" + return marked[0], marked[1] + + +def _diff_cells(number: int | None, text: str | None, marked: str | None, cls: str) -> str: + """The line-number and content cell for one side of a row, or filler where that side is empty.""" + if text is None: + return '' + body = marked if marked is not None else html.escape(text) + return f'{number}{body or " "}' + + +def split_diff_table(rel: str, before: list[str], after: list[str], labels: tuple[str, str]) -> str: + """One file's split diff as a table: line numbers and text for each version, side by side.""" + body: list[str] = [] + for row in _split_diff_rows(before, after): + if row.kind == "gap": + body.append('…') + continue + left_mark = right_mark = None + if row.kind == "replace" and row.left is not None and row.right is not None: + left_mark, right_mark = _inline_pair(row.left, row.right) + left_cls = "diff-ctx" if row.kind == "equal" else "diff-del" + right_cls = "diff-ctx" if row.kind == "equal" else "diff-add" + body.append( + "" + + _diff_cells(row.left_no, row.left, left_mark, left_cls) + + _diff_cells(row.right_no, row.right, right_mark, right_cls) + + "" + ) + head = ( + f'{html.escape(labels[0])}' + f'{html.escape(labels[1])}' + ) + return ( + f'
{html.escape(rel)}
' + f'{head}' + f"{''.join(body)}
" + ) + + +#: Backwards-compatible alias — ``report.py`` historically named this ``_split_diff_table``. +_split_diff_table = split_diff_table diff --git a/src/acumen/improve.py b/src/acumen/improve.py index 23ee1b0..3dbadbd 100644 --- a/src/acumen/improve.py +++ b/src/acumen/improve.py @@ -1,26 +1,25 @@ -"""The improving agent: read train evidence, write ``skills/v{n+1}/``. - -Where the drafter reads the package *source* to write the first skill, the improver -reads how the current skill *performed* — on the TRAIN split only — and edits it. Versions -are immutable: ``improve`` always writes a new directory, never mutates an existing one. - -The load-bearing constraint is that the improver must never see test results, or the whole -benchmark is worthless. This is enforced two ways, not one: - -1. **Structurally.** The agent's ``cwd`` is a throwaway work dir and its readable material - is a *curated copy* of train-split runs. The real ``runs/`` tree is not on its ``cwd``, - not in the read-only roots, and not in scope — so there is no test directory to reach in the - first place. -2. **Belt-and-braces.** A ``PreToolUse`` hook denies any tool call whose path resolves under - a ``runs/*/test/`` subtree, so even a call that names an absolute path back into the - project runs tree is refused. See :func:`find_test_access`. - -Prompt-level instruction alone is explicitly *not* sufficient. +"""The improving agent: read the knowledge wiki, write the next ``skills/vN/``. + +The improver no longer reads a raw dump of the parent skill's runs. It reads the **wiki** — the +distilled, per-task ``observations.md``/``hypothesis.md`` notes that :mod:`acumen.wiki` builds from +train-split runs and accumulates across versions — plus the parent skill (in improve mode), the +raw train transcripts (for drill-down), and the *filtered* package source. On the first epoch there +is no parent skill: the wiki holds only the ``noskill`` baseline and the improver **creates** v1. + +Two load-bearing isolations, both structural rather than by prompt: + +1. **No held-out (valid) results.** The improver's readable material is the wiki + staged train + transcripts — the real ``runs/`` tree is denied wholesale, and a ``PreToolUse`` hook refuses any + call that resolves under ``runs/*/valid/`` wherever it is pointed (see :func:`find_valid_access`). +2. **No skill bias from the target.** A package may already ship its own skill/agent guidance; the + improver reads the *filtered* source (:func:`acumen.scrub.build_filtered_source`) with those + artifacts stripped, and the raw checkout denied, so the optimization never re-serves a skill the + package already carries. This matters precisely because acumen's job is to ship skills into + packages. """ from __future__ import annotations -import json import shutil import tempfile from dataclasses import dataclass @@ -36,31 +35,31 @@ from acumen.logs import LiveLog from acumen.paths import ( ANSWER_FILE, - RESULT_FILE, SCRIPT_FILE, TRANSCRIPT_HTML, - TRANSCRIPT_JSONL, arm_name, - parse_run_dir, ) from acumen.prices import PriceTable, price_usage, pricer, resolve_cost from acumen.procs import label_env, reap from acumen.prompts import improve_prompt +from acumen.scrub import build_filtered_source, make_skill_guard from acumen.skills import ( SKILL_FILE, Skill, SkillError, content_files, + latest_version, load_skill, next_version, skill_dir, write_meta, ) from acumen.tasks import Task +from acumen.wiki import WIKI_DIRNAME, collect_arm_runs -#: The subtree component that marks a held-out split. A path under ``runs//test/…`` is +#: The subtree component that marks the held-out split. A path under ``runs//valid/…`` is #: what the improver must never reach. -_TEST_SPLIT = "test" +_VALID_SPLIT = "valid" #: tool_input keys that carry a filesystem path across the default Claude Code toolset. _PATH_KEYS = ("file_path", "path", "notebook_path", "filename") @@ -75,35 +74,13 @@ class ImproveError(RuntimeError): """Raised when a skill could not be improved.""" -@dataclass(frozen=True) -class TrainRun: - """One train-split benchmark run of the parent skill, as evidence for the improver. - - ``success`` and ``skill_loaded`` are independent outcomes with independent fixes: a run - that never loaded the skill says nothing about the skill's body, only about its - ``description``. Both reach the improver so it can tell the two apart. - """ - - task_id: str - model: str - rep: int - prompt: str - expected: str - answer: str | None - reason: str - success: bool - directory: Path - #: Whether the agent actually invoked the skill. ``None`` when the transcript could not - #: be read — undetermined, which is not the same as "did not load". - skill_loaded: bool | None = None - - @dataclass(frozen=True) class ImproveResult: - """A newly improved skill version and what it cost to produce.""" + """A newly created or improved skill version and what it cost to produce.""" skill: Skill - parent: str + #: The parent version, or ``None`` when this is the first (created) skill. + parent: str | None cost_usd: float | None turns: int n_train_runs: int @@ -113,198 +90,64 @@ class ImproveResult: log_html: Path | None = None -# ── Train evidence ───────────────────────────────────────────────────────────────────── - - -def collect_train_runs(runs_root: Path, arm: str, tasks: list[Task]) -> list[TrainRun]: - """Collect the parent arm's train-split runs into evidence for the improver. +# ── Staging ────────────────────────────────────────────────────────────────────────────── - Walks ``runs_root//train`` for ``result.json`` files and pairs each with its task's - train prompt and ground-truth answer. Only train runs are ever collected — the test - subtree is never even visited. - Parameters - ---------- - runs_root - The ``runs/`` root. - arm - The arm whose train runs are the improvement signal, e.g. ``"skill_v1"``. - tasks - The loaded tasks, to recover each run's prompt and expected answer. +def _seed_staging(staging: Path, parent: Skill) -> None: + """Pre-fill the staging dir with the parent skill's content, so the agent edits in place. - Returns - ------- - The train runs, failures first then by task/rep — so a reader sees what went wrong up - front. + Only content files are copied — ``meta.json`` is acumen bookkeeping and is written fresh once + the new version is promoted. """ - train_root = runs_root / arm / "train" - by_id = {task.id: task for task in tasks} - runs: list[TrainRun] = [] - if not train_root.is_dir(): - return runs - for result_path in sorted(train_root.rglob(RESULT_FILE)): - try: - data = json.loads(result_path.read_text()) - except (OSError, ValueError) as err: - raise ImproveError(f"cannot read {result_path}: {err}") from err - if data.get("valid", True) is False: - raise ImproveError( - "cannot improve from an infrastructure-invalid benchmark result: " - f"{result_path} ({data.get('reason')}). Fix the harness — replenish the provider " - "credential, or report the sandbox refusal as a bug if it refused a host — " - "and resume the benchmark first." - ) - key = parse_run_dir(runs_root, result_path.parent) - task = by_id.get(key.task_id) - if task is None: - # A run for a task no longer in tasks.yaml is stale evidence; skip it rather - # than guess a prompt for it. - continue - runs.append( - TrainRun( - task_id=key.task_id, - model=str(data.get("model", key.model)), - rep=key.rep, - prompt=task.train.prompt.strip(), - expected=task.train.answer.strip(), - answer=data.get("answer"), - reason=str(data.get("reason", "")), - success=bool(data.get("success", False)), - directory=result_path.parent, - skill_loaded=_loaded(data.get("skill_loaded")), - ) - ) - runs.sort(key=lambda r: (r.success, r.task_id, r.model, r.rep)) - return runs - - -def _loaded(value: Any) -> bool | None: - """Coerce a ``result.json`` ``skill_loaded`` field, preserving "undetermined" as ``None``.""" - return None if value is None else bool(value) - - -def load_rates(runs: list[TrainRun]) -> dict[str, tuple[int, int, int]]: - """Count how often the skill actually loaded, per model. - - Reported per model rather than pooled because the load rate varies far more across models - than across skill versions — a pooled number invites the improver to chase a model whose - behaviour no wording can reach. + staging.mkdir(parents=True, exist_ok=True) + for src in content_files(parent.directory): + dest = staging / src.relative_to(parent.directory) + dest.parent.mkdir(parents=True, exist_ok=True) + shutil.copyfile(src, dest) - Parameters - ---------- - runs - The train runs to tally. - Returns - ------- - ``{model: (loaded, total, undetermined)}``, where ``undetermined`` counts runs whose - transcript could not be read and which are therefore excluded from ``loaded``. - """ - rates: dict[str, tuple[int, int, int]] = {} - for run in runs: - loaded, total, unknown = rates.get(run.model, (0, 0, 0)) - rates[run.model] = ( - loaded + (run.skill_loaded is True), - total + 1, - unknown + (run.skill_loaded is None), - ) - return dict(sorted(rates.items())) - - -def _load_mark(loaded: bool | None) -> str: - """The phrase naming a run's load outcome, used identically in SUMMARY.md and run.md.""" - if loaded is None: - return "skill load UNDETERMINED" - return "skill LOADED" if loaded else "skill NOT LOADED" - - -def _load_section(runs: list[TrainRun]) -> list[str]: - """The SUMMARY.md section reporting per-model load rates.""" - rates = load_rates(runs) - lines = [ - "## Did the skill load at all?", - "", - "A run where the skill never loaded is not evidence about the skill's body — the agent", - "never read it. Only the `description` in the frontmatter decides whether a skill loads.", - "", - "| model | loaded | runs | rate |", - "| --- | ---: | ---: | ---: |", - ] - total_loaded = total_runs = total_unknown = 0 - for model, (loaded, total, unknown) in rates.items(): - total_loaded += loaded - total_runs += total - total_unknown += unknown - lines.append(f"| `{model}` | {loaded} | {total} | {loaded / total:.0%} |") - if total_runs: - lines.extend(["", f"**Overall: {total_loaded}/{total_runs} ({total_loaded / total_runs:.0%}).**"]) - if total_unknown: - noun = "run" if total_unknown == 1 else "runs" - lines.append( - f"({total_unknown} {noun} had no readable transcript, so loading is undetermined and is " - "counted as not loaded above.)" - ) - lines.append("") - return lines +def _stage_wiki(wiki_root: Path, dest: Path) -> None: + """Copy the whole wiki into the agent's work dir, so it never reaches back into the project.""" + dest.mkdir(parents=True, exist_ok=True) + if wiki_root.is_dir(): + shutil.copytree(wiki_root, dest, dirs_exist_ok=True) -def _write_material(train_dir: Path, runs: list[TrainRun]) -> None: - """Lay out the curated train evidence the improver reads. +def _stage_transcripts(runs_root: Path, arm: str, tasks: list[Task], dest: Path) -> int: + """Copy the parent arm's train transcripts into the work dir for drill-down. - A ``SUMMARY.md`` digest (per-model load rates, then runs with failures first) plus one - directory per run holding the run's ``script.py``, transcript, and a ``run.md`` stating - prompt/expected/actual/outcome and whether the skill loaded. Copied, not linked, so the - agent never has a path back into the real runs tree. + Returns the number of runs staged. Copies transcript/script/answer per run into + ``dest//__rep_/`` — copies, so there is no path back into the real tree. """ - train_dir.mkdir(parents=True, exist_ok=True) - n_fail = sum(1 for r in runs if not r.success) - noun = "run" if len(runs) == 1 else "runs" - lines = [ - "# Train-split evidence for the current skill", - "", - "Each run below benchmarked the current skill on one train task. Two things were", - "recorded per run, and they fail for different reasons and take different fixes:", - "whether the agent got the answer RIGHT, and whether the agent LOADED the skill at", - "all. Read the load rates first, then the failing runs. Open a run's `run.md`,", - "`script.py`, and `transcript.html` for the detail.", - "", - f"**{len(runs)} {noun}, {n_fail} failing.**", - "", - *_load_section(runs), - "## Runs", - "", - ] + runs = collect_arm_runs(runs_root, arm, tasks, split="train") for run in runs: - slug = f"{run.task_id}__{run.model}__rep_{run.rep}" - mark = "PASS" if run.success else "FAIL" - lines.append( - f"- `{slug}/` — **{mark}** ({run.reason}) — {_load_mark(run.skill_loaded)} — " - f"task `{run.task_id}` on `{run.model}` — expected `{run.expected}`, got `{run.answer!r}`" - ) - run_out = train_dir / slug + run_out = dest / run.task_id / f"{run.model}__rep_{run.rep}" run_out.mkdir(parents=True, exist_ok=True) - (run_out / "run.md").write_text( - f"# {run.task_id} — {mark} ({run.reason})\n\n" - f"- Model: `{run.model}`\n" - f"- Skill: {_load_mark(run.skill_loaded)}\n\n" - f"## Task prompt\n\n{run.prompt}\n\n" - f"## Expected answer\n\n{run.expected}\n\n" - f"## Answer the agent gave\n\n{run.answer!r}\n\n" - f"## Files\n\n- `script.py` — the script the agent wrote\n" - f"- `transcript.html` — the full run transcript\n" - ) - for name in (SCRIPT_FILE, ANSWER_FILE, TRANSCRIPT_HTML, TRANSCRIPT_JSONL): + for name in (TRANSCRIPT_HTML, SCRIPT_FILE, ANSWER_FILE): src = run.directory / name if src.is_file(): shutil.copyfile(src, run_out / name) - (train_dir / "SUMMARY.md").write_text("\n".join(lines) + "\n") + return len(runs) -# ── Test-access guard ────────────────────────────────────────────────────────────────── +def _validate_staged(staging: Path, skill_name: str) -> None: + """Fail loudly if the agent's output isn't a usable skill, before it becomes a version.""" + if not (staging / SKILL_FILE).is_file(): + raise ImproveError( + f"the improving agent left no {SKILL_FILE} in the staging directory — nothing to " + "promote. Inspect the run log or the prompt." + ) + try: + load_skill(staging.parent, staging.name, expect_name=skill_name) + except SkillError as err: + raise ImproveError(f"the improved skill is not valid: {err}") from err + + +# ── Valid-access guard ───────────────────────────────────────────────────────────────────── def _blocks(candidate: str, runs_root: Path) -> bool: - """Return whether ``candidate`` resolves under a ``runs/*/test/`` subtree.""" + """Return whether ``candidate`` resolves under a ``runs/*/valid/`` subtree.""" try: resolved = Path(candidate).expanduser().resolve() except (OSError, RuntimeError, ValueError): @@ -314,17 +157,17 @@ def _blocks(candidate: str, runs_root: Path) -> bool: except ValueError: return False parts = rel.parts - # runs_root//test/... — the split is the second component. - return len(parts) >= 2 and parts[1] == _TEST_SPLIT + # runs_root//valid/... — the split is the second component. + return len(parts) >= 2 and parts[1] == _VALID_SPLIT -def find_test_access(tool_name: str, tool_input: dict[str, Any], runs_root: Path) -> str | None: - """Return the first path in a tool call that reaches held-out test results, else ``None``. +def find_valid_access(tool_name: str, tool_input: dict[str, Any], runs_root: Path) -> str | None: + """Return the first path in a tool call that reaches held-out valid results, else ``None``. - Pure and side-effect free, so the enforcement can be exercised directly without - standing up an agent. Checks the path-bearing tool_input keys, and — for shell tools — - the whitespace/metacharacter-split tokens of the command, since a Bash call can name a - path no structured field would. + Pure and side-effect free, so the enforcement can be exercised directly without standing up an + agent. Checks the path-bearing tool_input keys, and — for shell tools — the whitespace/ + metacharacter-split tokens of the command, since a Bash call can name a path no structured + field would. Parameters ---------- @@ -337,7 +180,7 @@ def find_test_access(tool_name: str, tool_input: dict[str, Any], runs_root: Path Returns ------- - The offending path string, or ``None`` if the call touches no test results. + The offending path string, or ``None`` if the call touches no valid results. """ for key in _PATH_KEYS: value = tool_input.get(key) @@ -352,8 +195,8 @@ def find_test_access(tool_name: str, tool_input: dict[str, Any], runs_root: Path return None -def make_test_guard(runs_root: Path) -> HookMatcher: - """Build the ``PreToolUse`` hook that denies any access to held-out test results. +def make_valid_guard(runs_root: Path) -> HookMatcher: + """Build the ``PreToolUse`` hook that denies any access to held-out valid results. ``matcher=None`` fires the hook for every tool. The hook resolves paths against the real project ``runs/`` root, so it holds regardless of the agent's ``cwd``. @@ -365,7 +208,7 @@ def make_test_guard(runs_root: Path) -> HookMatcher: root = runs_root.resolve() async def guard(input_data: dict[str, Any], tool_use_id: str | None, context: Any) -> dict[str, Any]: - hit = find_test_access( + hit = find_valid_access( input_data.get("tool_name", ""), input_data.get("tool_input", {}) or {}, root, @@ -377,7 +220,7 @@ async def guard(input_data: dict[str, Any], tool_use_id: str | None, context: An "hookEventName": "PreToolUse", "permissionDecision": "deny", "permissionDecisionReason": ( - f"acumen blocks the improver from reading held-out test results ({hit}). " + f"acumen blocks the improver from reading held-out valid results ({hit}). " "Only train-split evidence is available to you." ), } @@ -386,33 +229,7 @@ async def guard(input_data: dict[str, Any], tool_use_id: str | None, context: An return HookMatcher(matcher=None, hooks=[guard]) -# ── Orchestration ────────────────────────────────────────────────────────────────────── - - -def _seed_staging(staging: Path, parent: Skill) -> None: - """Pre-fill the staging dir with the parent skill's content, so the agent edits in place. - - Only content files are copied — ``meta.json`` is acumen bookkeeping and is written - fresh once the new version is promoted. - """ - staging.mkdir(parents=True, exist_ok=True) - for src in content_files(parent.directory): - dest = staging / src.relative_to(parent.directory) - dest.parent.mkdir(parents=True, exist_ok=True) - shutil.copyfile(src, dest) - - -def _validate_staged(staging: Path, skill_name: str) -> None: - """Fail loudly if the agent's output isn't a usable skill, before it becomes a version.""" - if not (staging / SKILL_FILE).is_file(): - raise ImproveError( - f"the improving agent left no {SKILL_FILE} in the staging directory — nothing to " - "promote. Inspect the run log or the prompt." - ) - try: - load_skill(staging.parent, staging.name, expect_name=skill_name) - except SkillError as err: - raise ImproveError(f"the improved skill is not valid: {err}") from err +# ── Orchestration ────────────────────────────────────────────────────────────────────────── async def improve_skill( @@ -421,6 +238,7 @@ async def improve_skill( target: Target, skills_root: Path, runs_root: Path, + wiki_root: Path, tasks: list[Task], auth_mode: AuthMode = "session", prices: PriceTable | None = None, @@ -431,55 +249,67 @@ async def improve_skill( feedback: str | None = None, log: LiveLog | None = None, ) -> ImproveResult: - """Improve the current skill into the next version from its train-split evidence. + """Create or improve a skill from the knowledge wiki. + + When ``parent_version`` is ``None`` and no skill versions exist, the wiki holds only the + ``noskill`` baseline and the agent CREATES v1 from it + the filtered source. Otherwise it + improves ``parent_version`` (default: latest) into the next version, editing a pre-filled copy. Parameters ---------- cfg The pass config; supplies ``skill_name`` and ``meta_model``. target - The prepared target, supplying the interpreter (for verifying claims). Its source - is **not** exposed — the improver works from evidence, not code. + The prepared target. Its source is exposed only as a FILTERED copy (bundled skills + stripped); the interpreter is available for verifying claims. skills_root The ``skills/`` root; the new version is the next unused one. runs_root - The ``runs/`` root, read for train evidence and guarded against test access. + The ``runs/`` root, read for train transcripts and guarded against valid access. + wiki_root + The ``wiki/`` root, the improver's primary distilled signal. tasks - The loaded tasks, to pair runs with their prompts and expected answers. + The loaded tasks, to locate the parent arm's train transcripts. auth_mode - Which credential the improving agent authenticates with — ``"session"`` (the Claude - subscription) or ``"api"`` (see :func:`acumen.env.build_agent_env`). + Which credential the improving agent authenticates with. parent_version - The version to improve; defaults to the latest present. + The version to improve; defaults to the latest present, or ``None`` (create) when none + exist. model Override for the improving model; defaults to the config's. max_turns, max_usd - Turn and budget caps for the improving agent. **Unbounded by default** — the config's - ``max_turns``/``max_usd`` cap benchmark agents only; pass explicit caps to bound the - improver. + Turn and budget caps. Unbounded by default — the config's caps bound benchmark agents only. feedback - Optional maintainer guidance, injected into the improve prompt as a subordinated block - and recorded in ``meta.json`` as provenance. It is prompt text only — it cannot reach - the held-out test split, which stays blocked structurally and by the guard hook. + Optional maintainer guidance, subordinated below the hard rules and recorded in + ``meta.json``. It cannot reach the held-out valid split. log A :class:`LiveLog` to stream the agent's messages to and render an HTML log from. Returns ------- - The improved skill, loaded and validated, with its parent and cost. + The created/improved skill, loaded and validated, with its parent and cost. """ - parent = load_skill(skills_root, parent_version or _latest_or_error(skills_root), expect_name=cfg.skill_name) + # Resolve parent: an explicit version, else the latest present, else None (create the first). + resolved_parent = parent_version if parent_version is not None else latest_version(skills_root) + parent = None if resolved_parent is None else load_skill(skills_root, resolved_parent, expect_name=cfg.skill_name) + parent_ver = parent.version if parent is not None else None + new_version = next_version(skills_root) dest = skill_dir(skills_root, new_version) if dest.exists(): raise ImproveError(f"{dest} already exists — skill versions are immutable and never overwritten") - arm = arm_name(parent.version) - train_runs = collect_train_runs(runs_root, arm, tasks) + parent_arm = arm_name(parent_ver) + train_runs = collect_arm_runs(runs_root, parent_arm, tasks, split="train") if not train_runs: + hint = ( + "acumen bench --no-skill --split train" + if parent_ver is None + else f"acumen bench --skill {parent_ver} --split train" + ) raise ImproveError( - f"no train-split runs found for {parent.version} under {runs_root / arm / 'train'} — " - f"run `acumen bench --skill {parent.version}` first so the improver has evidence to work from" + f"no train-split runs found for {parent_arm} under {runs_root / parent_arm / 'train'} — " + f"run `{hint}` first so the wiki and improver have evidence to work from" ) n_failures = sum(1 for r in train_runs if not r.success) @@ -487,7 +317,8 @@ async def improve_skill( try: work = holder / "work" staging = work / new_version - train_dir = work / "train" + wiki_dir = work / WIKI_DIRNAME + transcripts_dir = work / "transcripts" rationale_path = work / "rationale.md" home = holder / "home" selected_model = model or cfg.meta_model @@ -497,8 +328,14 @@ async def improve_skill( for path in (work, home, config_dir, home / "tmp"): path.mkdir(parents=True, exist_ok=True) - _seed_staging(staging, parent) - _write_material(train_dir, train_runs) + if parent is not None: + _seed_staging(staging, parent) + else: + staging.mkdir(parents=True, exist_ok=True) + _stage_wiki(wiki_root, wiki_dir) + _stage_transcripts(runs_root, parent_arm, tasks, transcripts_dir) + # The filtered source: a package's own shipped skill must not bias the optimization. + source_copy = build_filtered_source(target.src_dir, holder / "source") # Marks the agent's processes so the teardown below can find what it leaves running. env = label_env( @@ -516,35 +353,40 @@ async def improve_skill( prompt = improve_prompt( package=target.pkg_name, version=target.pkg_version, + src=source_copy, python=target.python, skill_dir=staging, - train_dir=train_dir, + wiki_dir=wiki_dir, + transcripts_dir=transcripts_dir, rationale_path=rationale_path, skill_name=cfg.skill_name, - parent_version=parent.version, new_version=new_version, + parent_version=parent_ver, feedback=feedback, ) options = AgentOptions( cwd=work, env=env, model=selected_model, - # No default turn or budget cap: only bound the agent if the caller asked. The - # config's ``max_turns``/``max_usd`` cap benchmark agents only. + # No default turn or budget cap: only bound the agent if the caller asked. max_turns=max_turns, max_usd=max_usd, # Codex reports no billed figure, so a budget cap needs the run's own rate table. price_usd=pricer(selected_model, table), - read_dirs=(target.venv_dir,), + # The filtered source + venv, for verifying claims. Never the raw checkout. + read_dirs=(source_copy, target.venv_dir), write_dirs=(work,), discover_skills=True, # Belt-and-braces over the structural isolation: refuse any call that reaches a - # held-out test result, wherever the agent points it. Built only for Claude — the - # hook is an SDK object, and Codex gets the equivalent from ``deny_paths`` below. - claude_hooks={"PreToolUse": [make_test_guard(runs_root)]} if provider == "claude" else None, - # Codex gets only the staged train evidence and is denied the original - # run tree wholesale, which includes every held-out test artifact. - deny_paths=(runs_root.resolve(),), + # held-out valid result, or an unfiltered source artifact. Built only for Claude. + claude_hooks=( + {"PreToolUse": [make_valid_guard(runs_root), make_skill_guard(target.src_dir)]} + if provider == "claude" + else None + ), + # Codex gets only the staged evidence and filtered source; deny the run tree and the + # original checkout wholesale. + deny_paths=(runs_root.resolve(), target.src_dir.resolve()), ) result: AgentResult | None = None @@ -575,14 +417,14 @@ async def improve_skill( _validate_staged(staging, cfg.skill_name) - rationale = _read_rationale(rationale_path, result, parent.version, new_version) + rationale = _read_rationale(rationale_path, result, parent_ver, new_version) dest.parent.mkdir(parents=True, exist_ok=True) shutil.copytree(staging, dest) - write_meta(dest, parent=parent.version, rationale=rationale, feedback=feedback) + write_meta(dest, parent=parent_ver, rationale=rationale, feedback=feedback) skill = load_skill(skills_root, new_version, expect_name=cfg.skill_name) return ImproveResult( skill=skill, - parent=parent.version, + parent=parent_ver, cost_usd=resolve_cost( result.total_cost_usd, price_usage( @@ -604,20 +446,11 @@ async def improve_skill( shutil.rmtree(holder, ignore_errors=True) -def _latest_or_error(skills_root: Path) -> str: - from acumen.skills import latest_version - - latest = latest_version(skills_root) - if latest is None: - raise ImproveError(f"no skill versions found under {skills_root} — run `acumen draft` first, then bench it") - return latest - - -def _read_rationale(path: Path, result: AgentResult, parent: str, new_version: str) -> str: +def _read_rationale(path: Path, result: AgentResult, parent: str | None, new_version: str) -> str: """Recover the agent's rationale for ``meta.json``. - Prefers the ``rationale.md`` the prompt asks for; falls back to the agent's final - message, then to a plain provenance note, so a version always records why it exists. + Prefers the ``rationale.md`` the prompt asks for; falls back to the agent's final message, then + to a plain provenance note, so a version always records why it exists. """ if path.is_file(): text = path.read_text().strip() @@ -626,4 +459,4 @@ def _read_rationale(path: Path, result: AgentResult, parent: str, new_version: s final = (result.result or "").strip() if final: return final - return f"improved from {parent} to {new_version}" + return f"created {new_version}" if parent is None else f"improved from {parent} to {new_version}" diff --git a/src/acumen/logs.py b/src/acumen/logs.py index fb32df6..1300155 100644 --- a/src/acumen/logs.py +++ b/src/acumen/logs.py @@ -164,13 +164,17 @@ def finalize(self, *, config_dir: Path, work_dir: Path, result: ResultMessage | if result is None: return False prompt = getattr(result, "prompt", "") + # The run's authoritative usage — what it is billed on — so the footer matches result.json + # rather than re-summing the session file's per-message usage (which overcounts, since each + # turn re-counts the cached context). + usage = getattr(result, "usage", None) if getattr(result, "provider", "claude") == "codex": - traj = from_codex_events(getattr(result, "transcript", []), prompt=prompt) + traj = from_codex_events(getattr(result, "transcript", []), prompt=prompt, usage=usage) else: if not result.session_id: return False native = locate_transcript(config_dir, work_dir, result.session_id) - traj = from_claude_transcript(native, prompt=prompt) if native and native.is_file() else None + traj = from_claude_transcript(native, prompt=prompt, usage=usage) if native and native.is_file() else None if traj is None: return False write_trajectory_json(traj, self.jsonl_path.with_suffix(".trajectory.json")) diff --git a/src/acumen/markdown.py b/src/acumen/markdown.py new file mode 100644 index 0000000..2c5e371 --- /dev/null +++ b/src/acumen/markdown.py @@ -0,0 +1,158 @@ +"""A small, safe markdown-to-HTML renderer shared across acumen's HTML outputs. + +Both the transcript renderer (:mod:`acumen.trajectory`) and the report +(:mod:`acumen.report`) turn agent- or maintainer-authored markdown into HTML, and +neither wants an external renderer dependency. The renderer here is deliberately a +common-case subset of CommonMark — paragraphs, headings, bullet/ordered lists, fenced +code, pipe tables and inline emphasis — with everything HTML-escaped first, so no input +can inject markup and anything unrecognized falls through as plain text. +""" + +from __future__ import annotations + +import re +from html import escape + +#: How much of one fenced code block the page keeps. The full text lives in the source +#: beside the rendered HTML (e.g. the transcript JSONL). +OUTPUT_CAP = 20_000 + + +def clip(text: str) -> str: + """Truncate ``text`` to :data:`OUTPUT_CAP`, noting how much was dropped.""" + if len(text) <= OUTPUT_CAP: + return text + return text[:OUTPUT_CAP] + f"\n… {len(text) - OUTPUT_CAP} more characters" + + +def _md_inline(text: str) -> str: + """Render inline markdown (code, bold, italic, links) on one line, escaping HTML first.""" + codes: list[str] = [] + + def stash(match: re.Match[str]) -> str: + codes.append(match.group(1)) + return f"\x00{len(codes) - 1}\x00" + + text = re.sub(r"`([^`]+)`", stash, text) + text = escape(text) + text = re.sub(r"\[([^\]]+)\]\((https?://[^\s)]+)\)", r'\1', text) + text = re.sub(r"\*\*([^*]+)\*\*", r"\1", text) + text = re.sub(r"__([^_]+)__", r"\1", text) + text = re.sub(r"(?\1", text) + text = re.sub(r"(?\1", text) + return re.sub(r"\x00(\d+)\x00", lambda m: f"{escape(codes[int(m.group(1))])}", text) + + +def _table_cells(line: str) -> list[str]: + """Split one pipe-table row into cells, dropping the optional outer pipes.""" + stripped = line.strip() + stripped = stripped.removeprefix("|").removesuffix("|") + return stripped.split("|") + + +def _is_table_delimiter(line: str) -> bool: + """Whether ``line`` is a GFM table delimiter row (``| --- | :--: |``).""" + cells = _table_cells(line) + return "|" in line and bool(cells) and all(re.fullmatch(r"\s*:?-+:?\s*", cell) for cell in cells) + + +def _table_align(spec: str) -> str: + spec = spec.strip() + left, right = spec.startswith(":"), spec.endswith(":") + if left and right: + return "center" + if right: + return "right" + return "left" if left else "" + + +def _render_table(header: list[str], aligns: list[str], rows: list[list[str]]) -> str: + def cell(tag: str, text: str, index: int) -> str: + align = aligns[index] if index < len(aligns) else "" + style = f' style="text-align:{align}"' if align else "" + return f"<{tag}{style}>{_md_inline(text.strip())}" + + head = "" + "".join(cell("th", value, i) for i, value in enumerate(header)) + "" + body = "".join("" + "".join(cell("td", value, i) for i, value in enumerate(row)) + "" for row in rows) + return f"{head}{body}
" + + +def render_markdown(text: str) -> str: + """A small, safe markdown-to-HTML renderer — headings, lists, code, tables, inline. + + Deliberately a common-case subset (not full CommonMark): the input is paragraphs, bullet + lists, fenced code and inline emphasis. Everything is HTML-escaped, so no input can inject + markup, and anything unrecognized falls through as plain text. + """ + lines = text.replace("\r\n", "\n").replace("\r", "\n").split("\n") + out: list[str] = [] + para: list[str] = [] + items: list[str] = [] + list_tag = "" + + def flush_para() -> None: + if para: + out.append("

" + "
".join(_md_inline(line) for line in para) + "

") + para.clear() + + def flush_list() -> None: + nonlocal list_tag + if items: + out.append(f"<{list_tag}>" + "".join(f"
  • {_md_inline(it)}
    • " for it in items) + f"") + items.clear() + list_tag = "" + + i = 0 + while i < len(lines): + line = lines[i] + stripped = line.strip() + if stripped.startswith("```"): + flush_para() + flush_list() + i += 1 + code: list[str] = [] + while i < len(lines) and not lines[i].strip().startswith("```"): + code.append(lines[i]) + i += 1 + i += 1 # skip the closing fence + out.append(f"
    {escape(clip(chr(10).join(code)))}
    ") + continue + if "|" in stripped and i + 1 < len(lines) and _is_table_delimiter(lines[i + 1]): + flush_para() + flush_list() + header = _table_cells(line) + aligns = [_table_align(spec) for spec in _table_cells(lines[i + 1])] + i += 2 + rows: list[list[str]] = [] + while i < len(lines) and lines[i].strip() and "|" in lines[i]: + rows.append(_table_cells(lines[i])) + i += 1 + out.append(_render_table(header, aligns, rows)) + continue + if not stripped: + flush_para() + flush_list() + elif heading := re.match(r"(#{1,6})\s+(.*)", stripped): + flush_para() + flush_list() + level = min(len(heading.group(1)) + 2, 6) # start at h3 so content never out-shouts the page + out.append(f"{_md_inline(heading.group(2))}") + elif bullet := re.match(r"[-*+]\s+(.*)", stripped): + flush_para() + if list_tag and list_tag != "ul": + flush_list() + list_tag = "ul" + items.append(bullet.group(1)) + elif ordered := re.match(r"\d+[.)]\s+(.*)", stripped): + flush_para() + if list_tag and list_tag != "ol": + flush_list() + list_tag = "ol" + items.append(ordered.group(1)) + else: + flush_list() + para.append(stripped) + i += 1 + flush_para() + flush_list() + return "".join(out) diff --git a/src/acumen/paths.py b/src/acumen/paths.py index 18c4392..ae6a94c 100644 --- a/src/acumen/paths.py +++ b/src/acumen/paths.py @@ -13,9 +13,17 @@ from pathlib import Path from typing import Literal -Split = Literal["train", "test"] +# ``valid`` is held out from the improver but still picks the best version, so it is really a +# validation split. ``test`` is the true holdout: nothing in the loop ever sees it, and it is +# benched only once at the end of a fit on the single best version and the baseline. +Split = Literal["train", "valid", "test"] -SPLITS: tuple[Split, ...] = ("train", "test") +#: Every legal split. Used for path validation, task parsing, generation and checking. +SPLITS: tuple[Split, ...] = ("train", "valid", "test") + +#: The splits benched during normal epochs and by ``acumen bench`` with no ``--split``. ``test`` +#: is deliberately excluded so it stays untouched until the final held-out evaluation. +BENCH_SPLITS: tuple[Split, ...] = ("train", "valid") NOSKILL_ARM = "noskill" diff --git a/src/acumen/prompts.py b/src/acumen/prompts.py index 0ba5c20..bb2e842 100644 --- a/src/acumen/prompts.py +++ b/src/acumen/prompts.py @@ -19,12 +19,12 @@ def feedback_block(feedback: str | None) -> str: ```` delimiters and placed (by the templates) *after* the hard rules, with explicit wording that it is guidance and does NOT override anything above it. That subordination is deliberate: the feedback comes from a trusted maintainer, but it must not be - able to talk an agent out of the isolation rules (test-split, skill-bias) or the anti-overfit + able to talk an agent out of the isolation rules (valid-split, skill-bias) or the anti-overfit rules — those are enforced structurally regardless, and the prompt says so. Note: feedback steers *within* each command's methodology; it cannot redefine it. It won't, for instance, stop ``tasks`` from running the package to verify answers, nor let ``improve`` - reach the test split to cheat. + reach the valid split to cheat. Parameters ---------- @@ -61,7 +61,10 @@ def feedback_block(feedback: str | None) -> str: - The target package (`{package}`) is already installed. Run Python with `{python}`, which is also `python` on your PATH. Do not create virtualenvs and do not install or upgrade packages. -- You have web access. Use it if it helps. +- Use only the already-installed `{package}` — it is the source of truth. Do NOT clone, + download, fetch, or `pip install`/`pip download` its source repository or source + distribution (for example from GitHub or PyPI). +- You have web access for general reference. Use it if it helps. # What you must leave behind @@ -102,190 +105,214 @@ def feedback_block(feedback: str | None) -> str: """ -DRAFT_PROMPT = """\ -You are writing an agent skill for the Python package `{package}` (version {version}). +#: Shared guidance for the description trigger and anti-overfit body rules, used by both the +#: create and improve prompts so the two never drift apart. +_SKILL_CRAFT = """\ +# The description is the trigger + +The `description` is the ONE sentence an agent sees before deciding whether to open the skill. +It is the entire loading decision. Its job is to make the skill load exactly when it is +relevant and not otherwise. + +- **State the goals a user would actually phrase**, in their words, not the package's. An agent + matches the description against the task in front of it. Name the outcomes and the kinds of + question the skill answers. +- **Do not overfit it.** Widening the description with the specific datasets, methods, or + phrasings from the tasks is cheating: it buys train load rate and loses on the held-out valid + split. Widen to the CATEGORY of goal, never to the instances you saw. +- **Do not oversell.** A description that claims coverage the body does not deliver loads the + skill on tasks it cannot help with, and costs every one of those runs its tokens for nothing. + +# How to write the body -A skill is documentation written for an agent, not for a human. Its only purpose is to -make an agent that has never used `{package}` succeed at real tasks with it on the first -try. It is not a tutorial, not a README, and not a sales pitch. +- **Organize around goals, not modules.** Structure the skill so a stated goal maps to the right + entry point and the right sequence of steps — do not just mirror the module layout. +- **Write what is not guessable.** The agent already knows Python and can read a traceback. Spend + words on what it would get WRONG by guessing: non-obvious defaults, required preprocessing, the + function that looks right but isn't, where results land, argument shapes, the right order of steps. +- **Generalise — never overfit.** NEVER name a specific dataset, parameter value, column, expected + answer, or task in the skill. It must help on tasks you have not seen; enumerating the cases you + saw is cheating and fails the valid split. +- **Progressive disclosure.** Keep `SKILL.md` short and route depth to `references/*.md`. An agent + pays for every token of `SKILL.md` on every task, including the ones where it is irrelevant. +- **Prefer removing text over adding it.** A correct short code example beats a paragraph. If a + passage changed no outcome, cut it. +- **Verify before you write.** Do not assert a default, a return type, or where an output lands + without confirming it in the installed package or the docs. No hedging — say what to do.""" -The agent arrives with a goal stated in plain English — what a user wants done, not which -function to call. The skill's job is to get it from that goal to a working result: route it -to the right entry point and the right sequence of steps, so it never has to reverse-engineer -that from the module layout. + +IMPROVE_PROMPT = """\ +You are improving an agent skill for the Python package `{package}` (version {version}). + +A skill is documentation written for an agent, not a human. Its only purpose is to make an +agent that has never used `{package}` succeed at real tasks with it on the first try. + +You are producing version {new_version} — an improvement of version {parent_version}. # What you can read -- The package's source is at `{src}`. Read it — the source, the docstrings, the examples, - the docs directory. This is the ground truth about how the package behaves. -- `{package}` is also installed; run `{python}` to check anything you are unsure about. - Verify claims before you write them down. +- `{skill_dir}` — the current skill ({parent_version}). This directory has been pre-filled with a + copy of it. EDIT THESE FILES IN PLACE; what they contain when you finish becomes {new_version}. +- `{wiki_dir}` — the KNOWLEDGE WIKI: one directory per task, each with `observations.md` (what + agents did across models and replicates, how often the skill loaded, and whether loading led to + success) and `hypothesis.md` (why it did or did not work). Entries are tagged `[version][model]` + and accumulate across versions — `noskill` is the baseline, then `v1`, `v2`, … READ THIS FIRST. + It is the distilled signal of what the skill gets right and wrong; the trend across versions + tells you what past changes did. +- `{transcripts_dir}` — the raw TRAIN-split transcripts behind the wiki, for the parent skill. + Drill in here only when the wiki is not specific enough to act on. +- `{src}` — the package source (filtered: any skill or agent-guidance files a package ships have + been stripped and are BLOCKED — write from the API, not from someone else's skill). `{package}` + is installed; run `{python}` to verify a claim before writing it. Do not install packages. +- You have web access if the published docs help. -# Ignore any existing skills or agent instructions — deliberately hidden +# What you are NOT allowed to see -The package may already ship skills or agent-instruction files written for it (`SKILL.md`, -`.agents/skills/`, `.claude/skills/`, `.codex/`, `CLAUDE.md`, `AGENTS.md`, `.cursor/`, -Copilot instructions). These have been stripped from the source above, and any attempt to reach -them — or the original unfiltered checkout — is BLOCKED. This is on purpose: a skill copied from -pre-written guidance is not a skill written from the package, and this one is about to be -measured against no skill at all. Write yours from the API, the source, and the user-facing docs. +You are optimising against the TRAIN split. A separate, held-out VALID split measures whether your +changes generalise rather than memorise these tasks. Any tool call that reaches valid results is +BLOCKED. Do not attempt it — reaching valid data would invalidate the whole benchmark. -# What you must write +# Two different failures, two different fixes -Your working directory is `{out}`. Write: +The wiki records whether the agent LOADED the skill and whether it SUCCEEDED. Separate them before +changing anything — they look identical in a list of failures and have opposite fixes. -1. `{out}/SKILL.md` — required. It must begin with YAML frontmatter, exactly: +1. **The skill never loaded.** The agent never read the body, so nothing in the body caused this + and nothing you write in the body can fix it. The only lever is the `description`. +2. **The skill loaded and the run still failed.** Now the body is on trial. Fix it, using the + cause the wiki (or a transcript) actually shows — not one you imagine. +3. **The skill loaded and the run passed.** Evidence the body works. Do not rewrite it for style. ---- -name: {skill_name} -description: ---- +A skill that is never loaded scores exactly like no skill at all. If the wiki shows load rates low +across the board, that is the biggest available win — and the only lever is the `description`. If +one model loads reliably and another almost never does, that gap is mostly the model's behaviour; +do not contort the sentence chasing it. - The `name` must be exactly `{skill_name}`. - - The `description` is load-bearing and must be HONEST. It is the only part of the skill - an agent sees before deciding whether to open it, and it is the only thing that gets - the skill loaded at the right moment. - State what the skill covers and the goals it applies to — name the outcomes a user would - actually phrase, so the skill loads when one of them comes up. Do not oversell it, and do - not claim coverage the body does not deliver. Use this format: - [What it does] + [When to use it: the goals/triggers it fires on] + [Key capabilities] - -2. `{out}/references/*.md` — optional. Use these for detail that only some tasks need. - -# How to write it - -- **Organize around goals, not modules.** Work out what people actually use `{package}` to - accomplish — read its examples, tutorials, and docs, not just its API — and structure the - skill so a stated goal maps to the right entry point and the right sequence of steps. Do not - just mirror the package's module layout. -- **Write what is not guessable.** An agent already knows Python and can read a - traceback. Spend your words on what it would get WRONG by guessing: non-obvious - defaults, required preprocessing, the function that looks right but isn't, where - results are written, argument shapes and orientation, footguns the API invites, right order - of steps to follow. -- **Generalize within each goal.** Organize around categories of goal, but keep the guidance - under each one general enough to cover any task in that category. Don't enumerate one-off - recipes — that's the failure mode on the other side of module-mirroring. -- **Progressive disclosure.** `SKILL.md` should be short and route to `references/` for - depth. An agent pays for every token of it on every task, including the tasks where it - is irrelevant. If `SKILL.md` is long, you are taxing every run. -- **Be concrete.** A correct short code example beats a paragraph of prose. Show the real - call, with the arguments that matter. -- **Prefer removing text over adding it.** Anything that merely restates the obvious is - worse than nothing: it costs tokens and buries the parts that matter. -- **No hedging.** Say what to do. - -# Verify before you finish - -Do not write claims you have not checked. If you assert a default value, a return type, -or where an output lands, confirm it in the source or by running `{python}`. A skill that -confidently states something false is worse than no skill at all — it will send an agent -in the wrong direction with full confidence. +{craft} + +# What you must write + +1. Edit the skill in place under `{skill_dir}`. When you finish, `{skill_dir}/SKILL.md` must still + begin with YAML frontmatter whose `name` is exactly `{skill_name}`. +2. `{rationale_path}` — one short paragraph stating WHAT you changed and WHY, grounded in the wiki. + Write it OUTSIDE the skill directory; do not put it inside `{skill_dir}`. {feedback} -When you are done, `{out}/SKILL.md` must exist and start with the frontmatter above. +When you are done, `{skill_dir}/SKILL.md` exists and starts with the frontmatter above, and +`{rationale_path}` contains your rationale. """ -IMPROVE_PROMPT = """\ -You are improving an agent skill for the Python package `{package}` (version {version}). - -A skill is documentation written for an agent, not a human. Its only purpose is to make an -agent that has never used `{package}` succeed at real tasks with it on the first try. +CREATE_PROMPT = """\ +You are writing the FIRST agent skill (version {new_version}) for the Python package `{package}` +(version {version}). -You are producing version {new_version} — an improvement of version {parent_version}. +A skill is documentation written for an agent, not a human. Its only purpose is to make an agent +that has never used `{package}` succeed at real tasks with it on the first try. It is not a +tutorial, not a README, not a sales pitch. The agent arrives with a goal in plain English; the +skill routes it to the right entry point and sequence of steps. # What you can read -- `{skill_dir}` — the current skill ({parent_version}). This directory has been pre-filled - with a copy of it. EDIT THESE FILES IN PLACE; what they contain when you finish becomes - version {new_version}. -- `{train_dir}` — evidence from benchmarking the current skill on the TRAIN split. Read - `{train_dir}/SUMMARY.md` first. For each run you will find the task prompt, the expected - answer, the answer the agent actually gave, whether it passed, WHETHER THE AGENT LOADED THE - SKILL AT ALL, which model ran it, the `script.py` it wrote, and its full transcript. This is - your only signal about what the skill gets right and what it gets wrong. -- `{package}` is installed; run `{python}` (also `python` on your PATH) to verify any - claim about the API before you write it down. Do not install or upgrade packages. +- `{wiki_dir}` — the KNOWLEDGE WIKI: one directory per task, each with `observations.md` (what + agents did WITHOUT any skill — the `noskill` baseline — across models and replicates, and how + often they got it right) and `hypothesis.md` (why they succeeded or failed). READ THIS FIRST: it + tells you exactly where an unaided agent goes wrong, which is precisely what your skill must fix. +- `{transcripts_dir}` — the raw TRAIN-split transcripts behind the wiki. Drill in when the wiki is + not specific enough. +- `{src}` — the package source (filtered: any skill or agent-guidance files a package ships have + been stripped and are BLOCKED — write from the API, the source, and the user-facing docs, not + from someone else's skill). `{package}` is installed; run `{python}` to verify claims. Do not + install packages. - You have web access if the published docs help. # What you are NOT allowed to see -You are optimising against a TRAIN split. A separate, held-out TEST split is what measures -whether your changes actually generalise rather than memorising these particular tasks. You -must not see the test split, and any tool call that reaches test results will be BLOCKED. -Do not attempt it — reaching test data would invalidate the whole benchmark. +The wiki and transcripts are from the TRAIN split only. A held-out VALID split measures whether +your skill generalises. Any tool call that reaches valid results is BLOCKED — do not attempt it. # What you must write -1. Edit the skill in place under `{skill_dir}`. When you finish, `{skill_dir}/SKILL.md` - must still begin with YAML frontmatter whose `name` is exactly `{skill_name}` and whose - `description` is an honest one-sentence statement of what the skill covers and when to - use it. The `description` is the ONLY thing that decides whether the skill loads — see - below. - -2. `{rationale_path}` — one short paragraph stating WHAT you changed and WHY, grounded in - the train evidence. Write it here, OUTSIDE the skill directory. Do not put the rationale - inside `{skill_dir}`. - -# Two different failures, two different fixes +Your staging directory is `{skill_dir}`. Write: -Every run records whether the agent LOADED the skill and whether it SUCCEEDED. Separate them -before you change anything — they look identical in a list of failures and have opposite fixes. +1. `{skill_dir}/SKILL.md` — required. It must begin with YAML frontmatter, exactly: -1. **The skill never loaded.** The agent never read a word of the body, so nothing in the body - caused this and nothing you write in the body can fix it. The only lever is the - `description`. Editing the body in response to these runs is wasted work. -2. **The skill loaded and the run still failed.** Now the body is on trial: it steered the - agent wrong, or it was silent where it should have spoken. Fix the body, using the cause - you can see in that run's transcript — not one you imagine. -3. **The skill loaded and the run passed.** Evidence the body works. Do not rewrite it for - style. +--- +name: {skill_name} +description: +--- -A skill that is never loaded scores exactly like no skill at all, however good its body is. If -the load rate in `SUMMARY.md` is low, that is the biggest available win — and the only thing -you can do about it is the `description`. + The `name` must be exactly `{skill_name}`. The `description` is load-bearing and must be HONEST — + it is the only thing an agent sees before deciding whether to open the skill. -# The description is the trigger +2. `{skill_dir}/references/*.md` — optional, for detail only some tasks need. -The `description` is the ONE sentence an agent sees before deciding whether to open the skill. -It is the entire loading decision. Its job is to make the skill load exactly when it is -relevant and not otherwise. - -- **Read the load rates per model.** They are in `SUMMARY.md`, broken down by model. If one - model loads the skill reliably and another almost never does, that gap is mostly the model's - behaviour, not your wording — do not contort the sentence chasing it. Act on a rate that is - low across the board. -- **State the goals a user would actually phrase**, in their words, not the package's. An - agent matches the description against the task in front of it. Name the outcomes and the - kinds of question the skill answers. -- **Do not overfit it to the train tasks.** Widening the description with the specific - datasets, methods, or phrasings you just read in the train runs is the same cheating as - putting them in the body: it buys train load rate and loses on the test split. Widen to - the CATEGORY of goal, never to the instances you saw. -- **Do not oversell.** A description that claims coverage the body does not deliver loads the - skill on tasks it cannot help with, and costs every one of those runs its tokens for nothing. +3. `{rationale_path}` — one short paragraph on what the skill covers and why, grounded in the + baseline wiki. Write it OUTSIDE `{skill_dir}`. -# How to improve the body - -- **Fix what the evidence shows is broken.** Work from the runs where the skill loaded and - the agent still failed. Address the cause visible in the transcript, not one you imagine. -- **Generalise — never overfit.** NEVER name a specific dataset, parameter value, column, - expected answer, or task from the train runs in the skill. The skill must help on tasks - you have not seen. Guidance that enumerates these particular cases is cheating and will - fail the test split. -- **Prefer removing text over adding it.** A shorter skill that an agent reads and follows - beats a longer one it skims. Every token is paid on every task, including the ones where - the skill is irrelevant. If a passage did not change any outcome, cut it. -- **Verify before you write.** Do not assert a default, a return type, or where an output - lands without confirming it in the installed package or the docs. -- **No hedging.** Say what to do. +{craft} {feedback} When you are done, `{skill_dir}/SKILL.md` exists and starts with the frontmatter above, and `{rationale_path}` contains your rationale. """ +WIKI_PROMPT = """\ +You are a benchmark analyst keeping a running knowledge wiki about how well an agent skill for the +Python package `{package}` helps on ONE task. Your job this round: read what happened when the +`[{version}]` arm was benchmarked on this task across several models and replicates, and record it +BRIEFLY into two files. + +`[{version}]` is the skill version under review. `noskill` means no skill was installed (the +baseline). {skill_note} + +# What you are given + +- `{evidence_dir}/INDEX.md` — every run of `[{version}]` on task `{task_id}`, one line each: pass or + fail, whether the skill LOADED, the model, the expected answer, and the answer given. +- `{evidence_dir}/__rep_/` — that run's `transcript.html` (what the agent actually did), + its `answer.md`, and its `script.py`. Open a few to understand the pattern; you do NOT need to + read every one. +- The package source is at `{src}` and it is installed — run `{python}` if a fact helps you explain + an outcome. Optional; do not go down a rabbit hole. +{skill_body_note} +# What you must write — APPEND, do not rewrite + +Two files already exist and may contain entries from earlier arms. LEAVE those untouched and ADD +your new entries at the end. `noskill` is the first arm; later arms are `v1`, `v2`, … + +1. `{observations_path}` — add ONE line per model, in EXACTLY this format: + + - [{version}][]: + + Say the general idea, not a play-by-play. For `noskill`, "loaded" does not apply — just say what + the agents did and how often they got it right. + +2. `{hypothesis_path}` — add ONE line per model, in EXACTLY this format: + + - [{version}][]: + +# BE BRIEF — this is the whole point + +This wiki is read in full by the skill improver every round and grows forever. A verbose wiki is +WORSE than a short one. Hard rules: + +- One to two lines per entry. No transcript quotes, no step-by-step, no per-run breakdown, no + restating the task. Compress across replicates into the pattern. +- The hypothesis is one or two short sentences. No essays, no hedging. + +GOOD observation: + - [{version}][claude-opus-5]: Loaded 3/3; agents used the right entry point and passed every run. +GOOD hypothesis: + - [{version}][claude-opus-5]: The skill named the correct function and output location, which is + the step agents otherwise guess wrong. +BAD (too verbose): a paragraph recounting each replicate's tool calls and reasoning. + +When you are done, `{observations_path}` and `{hypothesis_path}` each contain your new +`[{version}]` line(s) appended after whatever was already there — and nothing else changed. +""" + + TASKGEN_PROMPT = """\ You are writing a benchmark of real analysis tasks for a Python package (`{package}`). Each task states a GOAL a user has, in plain language, plus the single answer a correct analysis @@ -351,11 +378,13 @@ def feedback_block(feedback: str | None) -> str: - GOOD (a lazy goal): "Using the pbmc3k data, find which transcription factor is most active in the monocytes. Give only the factor's symbol." -# Train and test variants +# Train, valid and test variants -Give each task a train and a test variant of the SAME goal, differing only in the input or the -target it asks about (a different cell type, group, condition, or dataset). Two instances of one -analysis with two different correct answers — so a skill cannot pass by memorising one answer. +Give each task a train, a valid and a test variant of the SAME goal, differing only in the input +or the target it asks about (a different cell type, group, condition, or dataset). Three instances +of one analysis with three different correct answers — so a skill cannot pass by memorising one. +Train drives learning, valid picks the best version, and test is the untouched held-out check, so +all three must be genuinely different instances. # Ground truth by execution @@ -369,8 +398,9 @@ def feedback_block(feedback: str | None) -> str: The script you run to obtain an answer is NOT scratch. Save one per split to `{scripts_dir}/-.py`, using the SAME `id` you gave the task in `{out}` — so the task -`bulk` needs `{scripts_dir}/bulk-train.py` and `{scripts_dir}/bulk-test.py`. These are what -`acumen check` reruns later to confirm the answer still holds, so each one must: +`bulk` needs `{scripts_dir}/bulk-train.py`, `{scripts_dir}/bulk-valid.py` and +`{scripts_dir}/bulk-test.py`. These are what `acumen check` reruns later to confirm the answer +still holds, so each one must: - Be SELF-CONTAINED and runnable from ANY empty working directory: `