From b9fa8cb2fc1ed703ba86f5f4f6c0a708e96b80f6 Mon Sep 17 00:00:00 2001 From: Alex Oliveira <4482374+aroff@users.noreply.github.com> Date: Mon, 14 Sep 2026 23:29:44 +0000 Subject: [PATCH] feat(skill): integrate eval authoring into FastSkill package --- .github/workflows/skill-evals.yml | 13 +- CONTRIBUTING.md | 22 +-- README.md | 5 + .../fixtures/defective-total/SKILL.md | 8 ++ .../fixtures/defective-total/ground-truth.md | 3 + .../fixtures/environment-variants.md | 9 ++ .../fixtures/existing-suite/SKILL.md | 8 ++ .../fixtures/existing-suite/evals/checks.toml | 4 + .../fixtures/existing-suite/evals/prompts.csv | 2 + .../existing-suite/evals/user-note.md | 3 + .../existing-suite/fixtures/release.md | 3 + .../existing-suite/skill-project.toml | 3 + .../fixtures/invoice-no-evals/SKILL.md | 8 ++ .../fixtures/ambiguous-date.txt | 4 + .../invoice-no-evals/fixtures/basic.txt | 4 + .../fixtures/missing-number.txt | 3 + .../fixtures/multiple-currencies.txt | 5 + evals/authoring/fixtures/scripted-workflow.md | 12 ++ .../fixtures/unnecessary-step/SKILL.md | 8 ++ .../unnecessary-step/scripted-answers.md | 5 + .../fixtures/unsupported-order/SKILL.md | 8 ++ evals/v2/README.md | 5 + evals/v2/run.sh | 12 +- evals/v2/stage.sh | 15 +- fastskill/SKILL.md | 9 +- .../examples/invoice-extraction/SKILL.md | 10 ++ .../invoice-extraction/evals/checks.toml | 22 +++ .../invoice-extraction/evals/judge-prompt.md | 10 ++ .../invoice-extraction/evals/prompts.csv | 4 + .../invoice-extraction/evals/review-notes.md | 11 ++ .../invoice-extraction/fixtures/basic.txt | 4 + .../fixtures/missing-number.txt | 3 + .../invoice-extraction/skill-project.toml | 10 ++ fastskill/references/eval-authoring.md | 80 +++++++++++ .../references/eval-authoring/examples.md | 36 +++++ .../references/eval-authoring/handoff.md | 31 +++++ .../references/eval-authoring/semantics.md | 31 +++++ fastskill/references/eval.md | 11 +- fastskill/skill-project.toml | 2 +- scripts/test_eval_authoring.py | 92 +++++++++++++ scripts/test_eval_scripts.py | 111 +++++++++++++++ specs/002-eval-authoring-consolidation.md | 130 ++++++++++++++++++ 42 files changed, 748 insertions(+), 31 deletions(-) create mode 100644 evals/authoring/fixtures/defective-total/SKILL.md create mode 100644 evals/authoring/fixtures/defective-total/ground-truth.md create mode 100644 evals/authoring/fixtures/environment-variants.md create mode 100644 evals/authoring/fixtures/existing-suite/SKILL.md create mode 100644 evals/authoring/fixtures/existing-suite/evals/checks.toml create mode 100644 evals/authoring/fixtures/existing-suite/evals/prompts.csv create mode 100644 evals/authoring/fixtures/existing-suite/evals/user-note.md create mode 100644 evals/authoring/fixtures/existing-suite/fixtures/release.md create mode 100644 evals/authoring/fixtures/existing-suite/skill-project.toml create mode 100644 evals/authoring/fixtures/invoice-no-evals/SKILL.md create mode 100644 evals/authoring/fixtures/invoice-no-evals/fixtures/ambiguous-date.txt create mode 100644 evals/authoring/fixtures/invoice-no-evals/fixtures/basic.txt create mode 100644 evals/authoring/fixtures/invoice-no-evals/fixtures/missing-number.txt create mode 100644 evals/authoring/fixtures/invoice-no-evals/fixtures/multiple-currencies.txt create mode 100644 evals/authoring/fixtures/scripted-workflow.md create mode 100644 evals/authoring/fixtures/unnecessary-step/SKILL.md create mode 100644 evals/authoring/fixtures/unnecessary-step/scripted-answers.md create mode 100644 evals/authoring/fixtures/unsupported-order/SKILL.md create mode 100644 fastskill/examples/invoice-extraction/SKILL.md create mode 100644 fastskill/examples/invoice-extraction/evals/checks.toml create mode 100644 fastskill/examples/invoice-extraction/evals/judge-prompt.md create mode 100644 fastskill/examples/invoice-extraction/evals/prompts.csv create mode 100644 fastskill/examples/invoice-extraction/evals/review-notes.md create mode 100644 fastskill/examples/invoice-extraction/fixtures/basic.txt create mode 100644 fastskill/examples/invoice-extraction/fixtures/missing-number.txt create mode 100644 fastskill/examples/invoice-extraction/skill-project.toml create mode 100644 fastskill/references/eval-authoring.md create mode 100644 fastskill/references/eval-authoring/examples.md create mode 100644 fastskill/references/eval-authoring/handoff.md create mode 100644 fastskill/references/eval-authoring/semantics.md create mode 100644 scripts/test_eval_authoring.py create mode 100644 scripts/test_eval_scripts.py create mode 100644 specs/002-eval-authoring-consolidation.md diff --git a/.github/workflows/skill-evals.yml b/.github/workflows/skill-evals.yml index 772cf12..5ab8767 100644 --- a/.github/workflows/skill-evals.yml +++ b/.github/workflows/skill-evals.yml @@ -7,9 +7,7 @@ on: - 'README.md' - 'CONTRIBUTING.md' - 'CLAUDE.md' - - 'fastskill/SKILL.md' - - 'fastskill/skill-project.toml' - - 'fastskill/references/eval.md' + - 'fastskill/**' - 'evals/**' - 'scripts/**' - 'specs/**' @@ -26,11 +24,8 @@ on: agent: description: 'Agent CLI to run live evals with' type: choice - # claude is the default because checks.toml's required `skill_invoked` - # check needs structured Skill tool-use events in the trace, which the - # codex backend never emits (codex has no Skill tool concept). A codex - # live run will always fail the suite; the option remains only for - # diagnostic runs of the non-trigger checks. + # Both installed runtimes can supply consultation evidence. Check the + # installed FastSkill version's runtime observability before a sweep. options: [claude, codex] default: claude trials: @@ -63,6 +58,8 @@ jobs: run: | cd fastskill fastskill eval validate + - name: Validate authoring assets and packaged example + run: python3 -m unittest scripts/test_eval_authoring.py scripts/test_eval_scripts.py # ---------------------------------------------------------------------------- # Deterministic, always-on: the spec-001 v2 suites. `validate` above reads the diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e102818..57a6bc4 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -15,16 +15,18 @@ gofastskill/skill/ └── fastskill/ # Shippable skill ├── SKILL.md # Skill documentation ├── skill-project.toml # Metadata + [tool.fastskill.eval] → ../evals/* - └── references/ # eval.md and other reference material + ├── references/ # eval.md, eval-authoring.md and supporting references + └── examples/ # Copyable authoring projects included in the ZIP ``` -There is a single `skill-project.toml` (inside `fastskill/`). Run FastSkill commands +The main `skill-project.toml` is inside `fastskill/`; copyable examples have their own +manifests. Run the skill's regression commands from the `fastskill/` directory so path resolution finds that manifest and `../evals/`. ## Development 1. Edit files in the `fastskill/` subdirectory (and `evals/` when extending the suite) -2. Run `python3 -m unittest scripts/test_command_namespaces.py` and +2. Run `python3 -m unittest discover -s scripts -p 'test_*.py'` and `python3 scripts/check-command-namespaces.py` from the repository root 3. Open a pull request from a branch off the latest `origin/main` (see `CLAUDE.md`) — changes land via PR with required status checks, not by pushing to `main` directly @@ -99,16 +101,18 @@ a check that contradicts the column is rejected rather than silently resolved. S - **namespace syntax**, **validate**, and **score-fixtures** run on every push/PR — deterministic, no agent, no tokens. -- **live-eval** is opt-in via *Run workflow* (`workflow_dispatch`), runs the v1 suite against a - real agent, and needs `OPENAI_API_KEY` / `ANTHROPIC_API_KEY` secrets. - -No CI job runs `evals/v2/` — at roughly $6 and an hour per sweep it belongs on a schedule or a -manual trigger, and nothing enforces its thresholds yet. +- **validate-v2** regenerates and validates the three v2 suites and runs the vacuity guard + on every PR, without model calls. +- **live-eval** is dispatch-only and runs the three v2 suites in parallel over Tailscale. + It needs the gateway and Tailscale secrets named in the workflow; **scorecard** then applies + the configured thresholds. It is not scheduled and does not run on a PR. +- Authoring assets and the packaged example are checked by `scripts/test_eval_authoring.py`. + Authoring test scenarios live under `evals/authoring/fixtures/`, outside the skill ZIP. `.github/workflows/publish-skill.yml`: - Detects changes to `fastskill/` directory -- Packages skill using FastSkill CLI +- Packages the complete `fastskill/` directory as a ZIP - Creates a GitHub release with the packaged skill artifact - Uses GitHub App token for release operations diff --git a/README.md b/README.md index 04c98fb..7a6bdeb 100644 --- a/README.md +++ b/README.md @@ -17,6 +17,11 @@ Read the installed skill with `fastskill skill read fastskill`. The skill docume canonical command namespaces introduced by the breaking CLI migration; use a matching FastSkill release. +The installed skill includes [guided eval authoring](fastskill/references/eval-authoring.md) +and a [copyable invoice example](fastskill/examples/invoice-extraction/). +Repository-only [evals](evals/README.md) measure the FastSkill skill itself; they are not +included in the released skill ZIP. See the [assessment and ownership record](specs/002-eval-authoring-consolidation.md). + See [CONTRIBUTING.md](CONTRIBUTING.md) for development and release details. ## License diff --git a/evals/authoring/fixtures/defective-total/SKILL.md b/evals/authoring/fixtures/defective-total/SKILL.md new file mode 100644 index 0000000..19b88d5 --- /dev/null +++ b/evals/authoring/fixtures/defective-total/SKILL.md @@ -0,0 +1,8 @@ +--- +name: defective-total +description: Add line items from a text invoice and return the numeric total. +--- + +# Total an invoice + +Add every line item, but deliberately omit the final line item. This defect is intentional test ground truth; a correctly authored eval must detect it rather than adopt it as the expected outcome. diff --git a/evals/authoring/fixtures/defective-total/ground-truth.md b/evals/authoring/fixtures/defective-total/ground-truth.md new file mode 100644 index 0000000..2a58a82 --- /dev/null +++ b/evals/authoring/fixtures/defective-total/ground-truth.md @@ -0,0 +1,3 @@ +# Test-owned ground truth + +For inputs `10.00`, `5.00`, and `2.00`, the independently calculated total is `17.00`. The target instruction deliberately produces `15.00` by omitting the final item. This file is not copied into the target execution fixture. diff --git a/evals/authoring/fixtures/environment-variants.md b/evals/authoring/fixtures/environment-variants.md new file mode 100644 index 0000000..99f052e --- /dev/null +++ b/evals/authoring/fixtures/environment-variants.md @@ -0,0 +1,9 @@ +# Controlled prerequisite variants + +Use these deterministic branches in workflow exercises: + +- incompatible CLI: stub `fastskill --version` as `0.8.0`; retain files and mark compatibility unverified; +- missing runtime: select an unavailable runtime and retain the parse-valid suite; +- missing judge credential: unset the declared `api_key_env`; do not treat authoring-agent authentication as judge authentication; +- invalid reference: point `checks` at malformed TOML, capture validation failure, repair it, and rerun validation; +- unavailable observation: select a backend that preflight identifies as unable to observe a required check; report exclusion/error rather than pass. diff --git a/evals/authoring/fixtures/existing-suite/SKILL.md b/evals/authoring/fixtures/existing-suite/SKILL.md new file mode 100644 index 0000000..14d77f7 --- /dev/null +++ b/evals/authoring/fixtures/existing-suite/SKILL.md @@ -0,0 +1,8 @@ +--- +name: existing-suite +description: Summarize release notes into concise bullets. +--- + +# Summarize release notes + +Preserve version numbers and breaking-change warnings. diff --git a/evals/authoring/fixtures/existing-suite/evals/checks.toml b/evals/authoring/fixtures/existing-suite/evals/checks.toml new file mode 100644 index 0000000..3c2bb2f --- /dev/null +++ b/evals/authoring/fixtures/existing-suite/evals/checks.toml @@ -0,0 +1,4 @@ +[[check]] +name = "command_contains" +pattern = "BREAKING" +cases = ["release-basic"] diff --git a/evals/authoring/fixtures/existing-suite/evals/prompts.csv b/evals/authoring/fixtures/existing-suite/evals/prompts.csv new file mode 100644 index 0000000..4e000d5 --- /dev/null +++ b/evals/authoring/fixtures/existing-suite/evals/prompts.csv @@ -0,0 +1,2 @@ +id,prompt,should_trigger,tags +release-basic,Summarize fixtures/release.md,true,smoke diff --git a/evals/authoring/fixtures/existing-suite/evals/user-note.md b/evals/authoring/fixtures/existing-suite/evals/user-note.md new file mode 100644 index 0000000..4f6316b --- /dev/null +++ b/evals/authoring/fixtures/existing-suite/evals/user-note.md @@ -0,0 +1,3 @@ +# User note + +Keep this unrelated note byte-for-byte when refining the suite. diff --git a/evals/authoring/fixtures/existing-suite/fixtures/release.md b/evals/authoring/fixtures/existing-suite/fixtures/release.md new file mode 100644 index 0000000..7925dd0 --- /dev/null +++ b/evals/authoring/fixtures/existing-suite/fixtures/release.md @@ -0,0 +1,3 @@ +Version 2.0.0 + +BREAKING: the legacy endpoint was removed. diff --git a/evals/authoring/fixtures/existing-suite/skill-project.toml b/evals/authoring/fixtures/existing-suite/skill-project.toml new file mode 100644 index 0000000..2faeea9 --- /dev/null +++ b/evals/authoring/fixtures/existing-suite/skill-project.toml @@ -0,0 +1,3 @@ +[tool.fastskill.eval] +prompts = "evals/prompts.csv" +checks = "evals/checks.toml" diff --git a/evals/authoring/fixtures/invoice-no-evals/SKILL.md b/evals/authoring/fixtures/invoice-no-evals/SKILL.md new file mode 100644 index 0000000..0d1eb07 --- /dev/null +++ b/evals/authoring/fixtures/invoice-no-evals/SKILL.md @@ -0,0 +1,8 @@ +--- +name: invoice-no-evals +description: Extract known invoice fields into JSON when asked to process a plain-text invoice. +--- + +# Invoice extraction + +Return `invoice_number`, `date`, `currency`, and `total`. If the invoice number is absent, use JSON null. An ambiguous numeric date must be returned as `ambiguous` rather than guessed. Do not activate for general questions about invoices. diff --git a/evals/authoring/fixtures/invoice-no-evals/fixtures/ambiguous-date.txt b/evals/authoring/fixtures/invoice-no-evals/fixtures/ambiguous-date.txt new file mode 100644 index 0000000..0293fca --- /dev/null +++ b/evals/authoring/fixtures/invoice-no-evals/fixtures/ambiguous-date.txt @@ -0,0 +1,4 @@ +Invoice: INV-8 +Date: 03/04/2026 +Currency: GBP +Total: 90.00 diff --git a/evals/authoring/fixtures/invoice-no-evals/fixtures/basic.txt b/evals/authoring/fixtures/invoice-no-evals/fixtures/basic.txt new file mode 100644 index 0000000..5a23e7d --- /dev/null +++ b/evals/authoring/fixtures/invoice-no-evals/fixtures/basic.txt @@ -0,0 +1,4 @@ +Invoice: INV-7 +Date: 2026-09-12 +Currency: AED +Total: 250.00 diff --git a/evals/authoring/fixtures/invoice-no-evals/fixtures/missing-number.txt b/evals/authoring/fixtures/invoice-no-evals/fixtures/missing-number.txt new file mode 100644 index 0000000..f52836b --- /dev/null +++ b/evals/authoring/fixtures/invoice-no-evals/fixtures/missing-number.txt @@ -0,0 +1,3 @@ +Date: 2026-09-10 +Currency: USD +Total: 18.25 diff --git a/evals/authoring/fixtures/invoice-no-evals/fixtures/multiple-currencies.txt b/evals/authoring/fixtures/invoice-no-evals/fixtures/multiple-currencies.txt new file mode 100644 index 0000000..1334ceb --- /dev/null +++ b/evals/authoring/fixtures/invoice-no-evals/fixtures/multiple-currencies.txt @@ -0,0 +1,5 @@ +Invoice: INV-9 +Date: 2026-09-09 +Line item: USD 10.00 +Tax: EUR 2.00 +Total: USD 12.00 diff --git a/evals/authoring/fixtures/scripted-workflow.md b/evals/authoring/fixtures/scripted-workflow.md new file mode 100644 index 0000000..da06b59 --- /dev/null +++ b/evals/authoring/fixtures/scripted-workflow.md @@ -0,0 +1,12 @@ +# Scripted author input for WF1-WF8 + +1. Start with `invoice-no-evals` and ask for evaluation tests. +2. Confirm that a missing invoice number must be JSON null. +3. Confirm that `03/04/2026` is ambiguous and must not be guessed. +4. Review coverage for ordinary, missing-number, ambiguous-date, multiple-currency, and unrelated cases. +5. Require outcome correctness for extracted fields; require consultation for extraction requests and non-consultation for unrelated requests. +6. Authorize six cases, one trial each, only after the authoring agent names the selected runtime and model ID. +7. Seed an ambiguous-date target failure and retain it as evidence. +8. Clarify the date criterion and rerun or rejudge only the affected evidence as supported. + +The authoring agent must preserve an existing suite's `evals/user-note.md`, report the unsupported ordered-call assertion without substitution, and hand off through files. diff --git a/evals/authoring/fixtures/unnecessary-step/SKILL.md b/evals/authoring/fixtures/unnecessary-step/SKILL.md new file mode 100644 index 0000000..cfe15ba --- /dev/null +++ b/evals/authoring/fixtures/unnecessary-step/SKILL.md @@ -0,0 +1,8 @@ +--- +name: unnecessary-step +description: Convert temperatures between Celsius and Fahrenheit. +--- + +# Convert temperatures + +Write an intermediate value to `scratch-temperature.txt`, then calculate the requested conversion and return the answer. diff --git a/evals/authoring/fixtures/unnecessary-step/scripted-answers.md b/evals/authoring/fixtures/unnecessary-step/scripted-answers.md new file mode 100644 index 0000000..4f88037 --- /dev/null +++ b/evals/authoring/fixtures/unnecessary-step/scripted-answers.md @@ -0,0 +1,5 @@ +# Scripted author decisions + +- The correct converted temperature is mandatory. +- Writing `scratch-temperature.txt` is not mandatory and must be treated as advisory adherence, not outcome correctness. +- If the instruction conflicts with the desired clean workspace, outcome correctness wins and the conflict must be reported. diff --git a/evals/authoring/fixtures/unsupported-order/SKILL.md b/evals/authoring/fixtures/unsupported-order/SKILL.md new file mode 100644 index 0000000..ca4dfab --- /dev/null +++ b/evals/authoring/fixtures/unsupported-order/SKILL.md @@ -0,0 +1,8 @@ +--- +name: unsupported-order +description: Extract text from scanned receipts and normalize their currency values. +--- + +# Process receipts + +Call OCR first and currency normalization second. The author requires this exact tool-call order, which the FastSkill 0.9.230 deterministic checks cannot represent faithfully. diff --git a/evals/v2/README.md b/evals/v2/README.md index bbac0ea..6b4a7eb 100644 --- a/evals/v2/README.md +++ b/evals/v2/README.md @@ -55,6 +55,11 @@ CI runs the three suites on parallel runners for the wall clock — 42 cases at which is also how to re-fold a sweep already on disk without re-running it. +Each invocation stages into a fresh `.stage-*` directory under the output root. Keep that +directory while retaining run artifacts: saved judging resolves the original staged project. +`stage.sh` refuses an existing destination, and `run.sh` exits nonzero on runtime errors +even when only a subset was selected. Model-quality failures remain reportable via `--no-fail`. + ## Three suites, not fourteen Correctness used to be twelve single-case suites. A suite's checks applied to all of its diff --git a/evals/v2/run.sh b/evals/v2/run.sh index d43b4d2..2eb3c95 100755 --- a/evals/v2/run.sh +++ b/evals/v2/run.sh @@ -54,9 +54,7 @@ fi # one RUNNER ERROR per suite, then EVAL_SCORECARD_NO_RUNS from a scorecard with nothing to read. mkdir -p "$OUT" OUT="$(cd "$OUT" && pwd)" -STAGE="$OUT/.stage" - -mkdir -p "$STAGE" +STAGE="$(mktemp -d "$OUT/.stage-XXXXXX")" # The guard is a precondition, not a formality: without it the numbers are meaningless. echo "=== vacuity guard ===" @@ -79,6 +77,7 @@ if ((${#judged[@]})); then fi echo "=== running ${#suites[@]} suite(s) against '$AGENT', $TRIALS trial(s) per case ===" +runner_errors=0 for suite in "${suites[@]}"; do skill_dir="$("$HERE/stage.sh" "$suite" "$STAGE/$suite" "$TRIALS" | tail -1)" @@ -98,14 +97,19 @@ for suite in "${suites[@]}"; do tail -3 "$OUT/$suite.log" | grep -oE '[0-9]+/[0-9]+ passed' | tail -1 || echo "done" else echo "RUNNER ERROR (see $OUT/$suite.log)" + runner_errors=$((runner_errors + 1)) fi done if ((score_at_end)); then echo - exec "$HERE/scorecard.sh" "$OUT" + score_status=0 + "$HERE/scorecard.sh" "$OUT" || score_status=$? + if ((runner_errors)); then exit 1; fi + exit "$score_status" fi echo echo "=== ran ${#suites[@]} of ${#all_suites[@]} suite(s); not scoring a subset ===" echo "fold this with the others: $HERE/scorecard.sh " +if ((runner_errors)); then exit 1; fi diff --git a/evals/v2/stage.sh b/evals/v2/stage.sh index d4a0f5c..207b318 100755 --- a/evals/v2/stage.sh +++ b/evals/v2/stage.sh @@ -20,13 +20,20 @@ TRIALS="${3:-5}" HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" SKILL_SRC="$(cd "$HERE/../.." && pwd)/fastskill" -if [[ ! -d "$HERE/$SUITE" ]]; then +case "$SUITE" in consultation|restraint|correctness) ;; *) echo "stage.sh: no suite '$SUITE' at $HERE/$SUITE" >&2 exit 1 -fi +esac -rm -rf "$DEST" -mkdir -p "$DEST/evals" +# Refuse existing paths, including dangling symlinks. Staging must never erase +# user files or overwrite a previous run's project, which saved judging needs. +if [[ -e "$DEST" || -L "$DEST" ]]; then + echo "stage.sh: destination already exists: $DEST; choose a fresh directory" >&2 + exit 1 +fi +mkdir -p -- "$(dirname -- "$DEST")" +mkdir -- "$DEST" +mkdir -- "$DEST/evals" cp -r "$SKILL_SRC" "$DEST/fastskill" # The whole suite directory, not just the two files the engine always reads: a diff --git a/fastskill/SKILL.md b/fastskill/SKILL.md index 9ab0c31..1b20ec0 100644 --- a/fastskill/SKILL.md +++ b/fastskill/SKILL.md @@ -1,7 +1,7 @@ --- name: fastskill -version: 2.0.0 -description: Package manager and operational toolkit for Claude Code-compatible skills. Use this skill when installing, managing, discovering, bundling, or analyzing skills; configuring repositories; running skill evaluations (`fastskill eval validate/run/judge/report/score/scorecard`) or the optimization loop (`fastskill optimization`); serving skills over HTTP/MCP; or building marketplace catalogs. See references/eval.md for eval setup in full. +version: 2.1.0 +description: Package manager and operational toolkit for Claude Code-compatible skills. Use this skill when installing, managing, discovering, bundling, or analyzing skills; configuring repositories; authoring, repairing, or running skill evaluation suites; using the optimization loop; serving skills over HTTP/MCP; or building marketplace catalogs. license: Apache-2.0 --- @@ -159,6 +159,11 @@ pass/fail rules: - **[Skill evals guide](references/eval.md)** — setup, prompts CSV, `checks.toml`, agents, artifacts, CI. +For creating or repairing a suite, follow [Eval authoring](references/eval-authoring.md). +It covers coverage decisions, outcome versus adherence checks, judge calibration, authorized +pilots, and continuation from saved files. The [invoice example](examples/invoice-extraction/) +is a copyable project; copy it outside the installed skill before editing or running it. + ## Basic usage ### Command namespaces diff --git a/fastskill/examples/invoice-extraction/SKILL.md b/fastskill/examples/invoice-extraction/SKILL.md new file mode 100644 index 0000000..0a4ad91 --- /dev/null +++ b/fastskill/examples/invoice-extraction/SKILL.md @@ -0,0 +1,10 @@ +--- +name: invoice-extraction +description: Extract invoice fields from plain-text invoices into JSON when the user asks for invoice data. +metadata: + id: invoice-extraction +--- + +# Extract invoices + +Return JSON with `invoice_number`, `date`, `currency`, and `total`. Use JSON null for a missing invoice number. Do not activate for unrelated questions. diff --git a/fastskill/examples/invoice-extraction/evals/checks.toml b/fastskill/examples/invoice-extraction/evals/checks.toml new file mode 100644 index 0000000..3387ef4 --- /dev/null +++ b/fastskill/examples/invoice-extraction/evals/checks.toml @@ -0,0 +1,22 @@ +[[check]] +name = "command_contains" +pattern = "normalize_currency" +required = false +cases = ["invoice-basic"] + +[[judge]] +name = "extraction-correctness" +cases = ["invoice-basic", "invoice-missing-number"] +prompt_file = "judge-prompt.md" +model = "replace-with-approved-judge-model" + +[[judge.criterion]] +name = "correct_fields" +kind = "bool" +description = "The JSON values agree with the invoice fixture, and a missing invoice number is JSON null." + +[judge_defaults] +base_url = "https://replace-with-approved-endpoint.example/v1" +api_key_env = "FASTSKILL_EXAMPLE_JUDGE_KEY" +temperature = 0.0 +max_tokens = 1024 diff --git a/fastskill/examples/invoice-extraction/evals/judge-prompt.md b/fastskill/examples/invoice-extraction/evals/judge-prompt.md new file mode 100644 index 0000000..0baa444 --- /dev/null +++ b/fastskill/examples/invoice-extraction/evals/judge-prompt.md @@ -0,0 +1,10 @@ +Review the extracted invoice JSON against the task and recorded workspace evidence. + +Task: {{case.prompt}} +Answer: {{trial.final_answer}} +Expected facts: {{case.expected}} + +{{rubric}} + +Reply using this contract: +{{output_contract}} diff --git a/fastskill/examples/invoice-extraction/evals/prompts.csv b/fastskill/examples/invoice-extraction/evals/prompts.csv new file mode 100644 index 0000000..ad250a5 --- /dev/null +++ b/fastskill/examples/invoice-extraction/evals/prompts.csv @@ -0,0 +1,4 @@ +id,prompt,should_trigger,tags,workspace_subdir,expected +invoice-basic,"Extract the invoice in fixtures/basic.txt as JSON.",true,"outcome,positive",,"{""invoice_number"":""INV-1042"",""date"":""2026-09-01"",""currency"":""USD"",""total"":123.45}" +invoice-missing-number,"Extract the invoice in fixtures/missing-number.txt as JSON. Use null for missing fields.",true,"outcome,boundary",,"{""invoice_number"":null,""date"":""2026-09-02"",""currency"":""EUR"",""total"":78.90}" +invoice-unrelated,"Explain the difference between a receipt and an invoice.",false,"trigger,negative",, diff --git a/fastskill/examples/invoice-extraction/evals/review-notes.md b/fastskill/examples/invoice-extraction/evals/review-notes.md new file mode 100644 index 0000000..16a0abe --- /dev/null +++ b/fastskill/examples/invoice-extraction/evals/review-notes.md @@ -0,0 +1,11 @@ +# Invoice extraction eval review + +Compatibility: authored for FastSkill 0.9.230; validate against another installed release before use. + +Coverage: ordinary extraction, missing invoice number, and unrelated-request non-triggering. The author confirmed that a missing number must be JSON null. Test-owned expected facts are supplied through the `expected` CSV column and rendered into the judge prompt. Field correctness is an outcome expectation. Skill consultation/non-consultation is an adherence/selection expectation imposed by `should_trigger` semantics. Currency-normalization command text is advisory only. + +Calibration contrasts: JSON null is acceptable for the missing number; the string `"UNKNOWN"` is unacceptable. These labels are based on the confirmed criterion. They have not been executed against a judge in this example. + +Unsupported: ordered OCR-then-normalization calls cannot be asserted by the current deterministic check types. No weaker substitute is claimed. + +Status: example files are syntax-validated without contacting the placeholder judge endpoint. No runtime pilot or judge calibration is claimed. diff --git a/fastskill/examples/invoice-extraction/fixtures/basic.txt b/fastskill/examples/invoice-extraction/fixtures/basic.txt new file mode 100644 index 0000000..2f093a0 --- /dev/null +++ b/fastskill/examples/invoice-extraction/fixtures/basic.txt @@ -0,0 +1,4 @@ +Invoice number: INV-1042 +Date: 2026-09-01 +Currency: USD +Total: 123.45 diff --git a/fastskill/examples/invoice-extraction/fixtures/missing-number.txt b/fastskill/examples/invoice-extraction/fixtures/missing-number.txt new file mode 100644 index 0000000..cb81df9 --- /dev/null +++ b/fastskill/examples/invoice-extraction/fixtures/missing-number.txt @@ -0,0 +1,3 @@ +Date: 2026-09-02 +Currency: EUR +Total: 78.90 diff --git a/fastskill/examples/invoice-extraction/skill-project.toml b/fastskill/examples/invoice-extraction/skill-project.toml new file mode 100644 index 0000000..0e27dbd --- /dev/null +++ b/fastskill/examples/invoice-extraction/skill-project.toml @@ -0,0 +1,10 @@ +[metadata] +id = "invoice-extraction" +version = "1.0.0" + +[tool.fastskill.eval] +prompts = "evals/prompts.csv" +checks = "evals/checks.toml" +trials_per_case = 1 +pass_threshold = 1.0 +fail_on_missing_agent = true diff --git a/fastskill/references/eval-authoring.md b/fastskill/references/eval-authoring.md new file mode 100644 index 0000000..5d64df9 --- /dev/null +++ b/fastskill/references/eval-authoring.md @@ -0,0 +1,80 @@ +# Author FastSkill evals + +Turn an existing target skill into a reviewable suite in the formats supported by the installed FastSkill CLI. The user chooses the authoring agent. FastSkill validates explicit files, runs selected evaluation targets, and returns evidence; it does not conduct the conversation or remember approval. + +## Start from evidence + +1. Locate the target `SKILL.md`, its project root, relevant references, and any existing eval files. Preserve unrelated content and local edits. +2. Run `fastskill --version`, `fastskill eval --help`, and `fastskill eval validate --help`. Compare the installed version with the compatibility note below; treat live help and validation as authoritative for command availability. +3. Read the packaged [eval guide](eval.md) for the schema and commands. Read [authoring semantics](eval-authoring/semantics.md) before choosing checks or judges. Confirm command availability with the installed CLI. + +This skill is maintained against FastSkill `0.9.230` and the pinned `aikit-evals` revision in that release. On another version, inspect its matching references and validate every field; report the suite as compatibility-unverified if the relevant contracts cannot be confirmed. Never assume that newer skill guidance adds a CLI field. + +## Design before generating + +After inspection, propose a short coverage outline spanning useful outcomes, positive and negative triggering, boundaries, and failure handling where relevant. Ask only questions whose answers materially change a case or criterion. Retain the answer and review the outline before creating a large suite. + +For every expectation, label it in review notes as: + +- **outcome**: the result must satisfy the user's acceptance criterion; or +- **adherence**: the agent must consult or follow a prescribed method. + +Make adherence mandatory only when the author says the method is required. Surface conflicts between target instructions and intended outcomes. Do not silently choose one or weaken an unsupported expectation. + +## Author and review + +Use ordinary `skill-project.toml`, CSV, TOML, Markdown prompt, and fixture files. Prefer deterministic checks for evidence they actually observe and a judge only for outcome criteria that need semantic review. Keep expected labels and known defects outside inputs visible to the evaluated target where practical. + +Check CSV row widths against the header and inspect parsed expected values. Ask the target to solve the task without supplying the expected answer in its prompt; a requested output format should use placeholders rather than the ground-truth result. File validation alone does not establish these properties. + +For a mixed positive/negative suite, normally omit explicit `skill_invoked` checks and let each row's `should_trigger` value generate the correctly polarized check. Never add one global positive check: it applies to negative rows too and the supported validator rejects the contradiction. If explicit trigger checks are necessary, scope them to exact case IDs and keep their polarity equal to those rows. + +Resolve judge prompt paths relative to `checks.toml`, not the project root. A judge can grade only evidence rendered into its prompt. Put independently established expected values in extra CSV columns such as `expected`, render them with `{{case.expected}}`, and do not claim a judge can inspect an unchanged fixture merely because it exists in the workspace. + +In the supported schema, repeat singular `[[judge.criterion]]` tables and render the evaluated answer with `{{trial.final_answer}}`. Validate these names instead of guessing plural tables or generic output fields. + +Write concise `evals/review-notes.md` recording: + +- coverage and the origin of each expectation; +- outcome versus adherence classification; +- author-confirmed decisions and remaining assumptions; +- unsupported expectations and missing prerequisites; +- validation, calibration, pilot, and target-verdict evidence as separate statuses. + +Present the cases and criteria as one coherent suite. Ask targeted follow-ups only for unresolved ambiguities; do not require approval row by row. For concrete layouts and limitation wording, read [authoring examples](eval-authoring/examples.md). + +## Validate, challenge, and pilot + +Run `fastskill eval validate` first, then validate selected runtime observability with `--agent ` or `--all`. Repair authored syntax and references, rerunning the failing check. + +Before the default isolated pilot, ensure `skill-project.toml` declares `[metadata].id` for the target skill. File validation can succeed without this pilot prerequisite. + +Keep these categories distinct: + +- invalid configuration; +- missing CLI, runtime, endpoint, or credential prerequisite; +- measurement not observable on a backend; +- execution error; +- well-formed quality failure. + +Challenge each judge with at least one author-confirmed or independently verifiable acceptable outcome and one unacceptable outcome. Record the label reason. If the installed interfaces cannot execute a standalone contrast, call it qualitative review and explicitly leave calibration unverified; never forge run artifacts. `eval judge` can judge or rejudge saved run evidence, but it is not a general calibration command. + +Before a paid or model-backed pilot, use existing user authorization or propose an explicit scope naming cases, evaluation runtimes, and trial counts. If unclear, ask once. Do not promise a monetary cap. Authoring-agent credentials do not prove that evaluation runtimes or native judges are configured. + +Run only the authorized scope with explicit parameters, for example: + +```bash +fastskill eval run --agent codex --case invoice-basic --trials 1 --output-dir ./eval-runs +``` + +Explain actual artifacts and distinguish pass, fail, error, excluded/unobservable evidence, and missing judgment. A useful test that exposes a target defect is successful authoring evidence; do not change the criterion merely to obtain a pass. + +## Refine and hand off + +Use `eval score` when only deterministic checks changed and saved evidence remains suitable, `eval judge --rejudge` when only judge configuration changed and saved evidence remains suitable, and rerun the target when its prompt, fixture, target skill, runtime/model, or required evidence changed. Preserve unrelated edits. + +Hand over editable suite files, fixtures, review notes, exact commands, run artifact paths, observed versions/model IDs, and unresolved limitations. Another capable authoring agent must be able to continue from files without conversation state. Use [the handoff template](eval-authoring/handoff.md) for the completion record. + +## Boundaries + +Do not invent checks, schemas, graders, calibration commands, interview services, approval stores, native ablation, mocks, sandbox guarantees, or cost ceilings. Skill isolation controls discovered skills, not OS-level filesystem or network confinement. When the engine cannot faithfully represent an intended assertion, keep it as an explicit unsupported expectation and propose any engine extension separately. diff --git a/fastskill/references/eval-authoring/examples.md b/fastskill/references/eval-authoring/examples.md new file mode 100644 index 0000000..3bec1ac --- /dev/null +++ b/fastskill/references/eval-authoring/examples.md @@ -0,0 +1,36 @@ +# Authoring examples + +The executable example is in [invoice-extraction](../../examples/invoice-extraction/). It is compatible with FastSkill 0.9.230. Copy it to a fresh working directory outside the installed skill, then run `fastskill eval validate` there without credentials; select a runtime only when intentionally checking its availability and observability. + +## Review table + +| Case | Intent | Expected result | Measurement | +| --- | --- | --- | --- | +| `invoice-basic` | Ordinary invoice | Correct number, date, currency, and total | outcome judge; trigger required by `should_trigger=true` | +| `invoice-missing-number` | Missing field | `invoice_number` is null | outcome judge; trigger required | +| `invoice-unrelated` | Unrelated request | Skill is not consulted | adherence/selection through `should_trigger=false` | + +An optional procedure can be recorded without gating: + +```toml +[[check]] +name = "command_contains" +pattern = "normalize_currency" +required = false +cases = ["invoice-basic"] +``` + +Do not describe the case as outcome-only while `should_trigger=true`: the implicit consultation check remains required. If outcome-only scoring is intended and the current format cannot disable the implicit check without contradicting the CSV, record that policy as unsupported rather than changing `should_trigger` deceptively. + +## Contrasting judge outcomes + +Criterion: “The response represents a missing invoice number as JSON null.” + +- Acceptable: `{"invoice_number":null,"date":"2026-09-01"}`. Label reason: the confirmed missing-number policy is null. +- Unacceptable: `{"invoice_number":"UNKNOWN","date":"2026-09-01"}`. Label reason: the sentinel string violates the confirmed policy. + +These labels support review. They are executed calibration evidence only if passed through a supported real judge interface and the resulting artifact is retained. + +## Unsupported expectation + +“Verify that the agent called OCR before parsing and then called currency normalization, in that order.” The current checks can inspect substring presence and aggregate tool-call count, but do not provide a structured ordered-call assertion. Do not substitute two `command_contains` checks and claim order was verified. Keep the intended assertion in review notes as unsupported and propose the engine feature separately. diff --git a/fastskill/references/eval-authoring/handoff.md b/fastskill/references/eval-authoring/handoff.md new file mode 100644 index 0000000..4f05497 --- /dev/null +++ b/fastskill/references/eval-authoring/handoff.md @@ -0,0 +1,31 @@ +# Eval suite handoff + +Record each category independently. + +```markdown +## Coverage and decisions +- Target skill and version: +- Coverage outline reviewed: +- Author-confirmed decisions: +- Outcome expectations: +- Adherence expectations: +- Unsupported expectations: + +## Verification +- FastSkill version and compatibility status: +- Configuration validation command/result: +- Runtime observability command/result: +- Judge contrast labels and provenance: +- Executed calibration artifacts, or unverified reason: +- Authorized pilot scope: +- Pilot commands and run directories: +- Target passes/failures/errors/unmeasured evidence: + +## Continuation +- Files changed: +- Preserved unrelated edits: +- Rescore, rejudge, or rerun decision: +- Missing prerequisites and next action: +``` + +Do not collapse these fields into “verified.” A reviewed and parse-valid suite may still be unexecuted; an executed trial may lack a judgment; a target failure may demonstrate a useful test. diff --git a/fastskill/references/eval-authoring/semantics.md b/fastskill/references/eval-authoring/semantics.md new file mode 100644 index 0000000..c9c846e --- /dev/null +++ b/fastskill/references/eval-authoring/semantics.md @@ -0,0 +1,31 @@ +# FastSkill 0.9.230 eval semantics + +Use this reference while mapping expectations to the packaged [eval guide](../eval.md). It highlights decisions that are easy to get wrong; it is not a duplicate schema. + +## Triggering and case scope + +`should_trigger` is executable scoring input in this release. For each case, the engine creates a required `skill_invoked` expectation with the same polarity unless an explicit `skill_invoked` check applies to that case. An explicit applicable check replaces the implicit one. Validation rejects an explicit polarity that contradicts the case's `should_trigger` value. + +Every deterministic check accepts an optional `cases = ["exact-id", ...]` selector. It applies only to those exact case IDs. Scope mixed positive and negative behavior deliberately. + +In a mixed-polarity suite, omit explicit `skill_invoked` checks unless they add necessary specificity. A global `expected = true` check applies to the negative rows and contradicts them; comments do not create per-case polarity. When explicit checks are used, select exact cases and match each selected row's `should_trigger` value. + +`skill_invoked` recognizes a structured `Skill` invocation or, on supported decoders, a tool input that refers to the staged skill-document path. The path form is evidence of consultation, not proof that the agent read or followed the instructions. Measure outcome correctness separately. + +With no checks file—or an empty check list—the implicit `should_trigger` expectation still applies. Process exit alone is not the verdict. + +## Verdicts and evidence + +Required deterministic checks are all-or-nothing for a trial. `required = false` keeps a check observable without gating. Across trials, `pass_threshold` controls the aggregated case result. Judge criteria have their own normalized scores; a judge gates only when it declares `min_score`. + +Validation proves that configured files parse and that file-only contracts are consistent. Runtime validation also checks installed target availability and preflight observability. It does not execute the skill, contact judge endpoints, or establish outcome quality. + +Judge `prompt_file`, `system_prompt_file`, and `retry_prompt_file` paths are relative to the directory containing `checks.toml`. A judge sees only rendered template variables. Unchanged fixture contents are not implied by `{{trial.workspace_diff}}`; provide test-owned expected facts through an extra CSV column and render `{{case.}}` when the rubric needs them. + +`eval run` records target execution evidence. `eval judge` applies declared judges to completed non-error trials. `eval score` reuses saved trace/workspace evidence for deterministic checks. Missing observations, errored trials, and missing judgments are not passes or zero-quality observations. + +## Isolation and authorization + +Default isolation stages the selected skill in a scratch workspace and suppresses ambient skill discovery where the backend supports it. This is measurement isolation, not an OS sandbox. Inspect the recorded isolation fidelity. + +The authoring agent obtains user authorization for concrete model-backed work. FastSkill accepts explicit runtime/case/trial arguments and has no conversational consent state. Scope does not enforce a monetary ceiling. diff --git a/fastskill/references/eval.md b/fastskill/references/eval.md index 3c8fcd4..878caed 100644 --- a/fastskill/references/eval.md +++ b/fastskill/references/eval.md @@ -2,6 +2,8 @@ This guide matches the FastSkill CLI implementation: configuration is read from `skill-project.toml`, cases from a prompts CSV, optional scoring rules from a checks TOML, and execution uses **aikit-sdk** (`run_agent_events`) with a supported agent CLI. +For designing or repairing those files, use the [authoring workflow](eval-authoring.md). + ## What evals are for Evals run repeatable prompts against a real agent. By default each case runs **isolated**: the skill under test is deployed alone into a per-case scratch workspace, and the agent's user-level skill discovery is suppressed where the backend supports it — so trigger rates measure your skill, not whatever else is installed on the machine. Each run records stdout, stderr, a JSONL trace, and per-case results under a timestamped run directory. Optional **checks** assert properties of that output (skill invocation, substring expectations, file presence, tool-call count). @@ -21,8 +23,10 @@ Evals run repeatable prompts against a real agent. By default each case runs **i If you get this for a key spelled correctly, install that agent's CLI — the `Available:` list reports what was found locally, so it differs from machine to machine and from the list above. -Evals do **not** require embedding or `OPENAI_API_KEY` unless your workflow also uses -`fastskill index rebuild` or `fastskill skill search`. +Deterministic validation and scoring need no model credential. Target execution needs the +selected runtime's authentication; native judging separately needs its configured endpoint, +model, and `api_key_env` credential. That variable may be `OPENAI_API_KEY` if explicitly +selected in the judge configuration. Embedding configuration is unrelated to these prerequisites. ## 1. Configure `skill-project.toml` @@ -169,7 +173,8 @@ Parse/load errors: `EVAL_CHECKS_INVALID`. ### Scoring rules when checks are present - Timeout, or failure to launch the agent at all: case status **`error`** with a message (checks are not used to override that). -- If **`checks` is empty** (no file or file loads zero checks): pass iff exit code is `0`. +- If **`checks` is empty** (no file or file loads zero checks): the implicit `should_trigger` + expectation still applies. Exit code `0` alone does not establish a pass. - If checks are loaded: case **`passed`** only if every **required** check passed; `required = false` results are reported but advisory. - A suite that parses to **zero cases** refuses to run (`EVAL_EMPTY_SUITE`) instead of reporting a vacuous `0/0 PASSED`. diff --git a/fastskill/skill-project.toml b/fastskill/skill-project.toml index 91ab862..b636ff7 100644 --- a/fastskill/skill-project.toml +++ b/fastskill/skill-project.toml @@ -2,7 +2,7 @@ schema_version = "1" [metadata] id = "fastskill" -version = "2.0.0" +version = "2.1.0" name = "fastskill" description = "Package manager and operational toolkit for AI Agent skills. This skill should be used when installing, managing, discovering, bundling, or analyzing skills; configuring repositories; performing semantic search; running skill evaluations or the optimization loop; serving skills over HTTP/MCP; or building marketplace catalogs." author = "Alexandre Oliveira" diff --git a/scripts/test_eval_authoring.py b/scripts/test_eval_authoring.py new file mode 100644 index 0000000..6586b58 --- /dev/null +++ b/scripts/test_eval_authoring.py @@ -0,0 +1,92 @@ +"""Validate the shipped authoring project with the candidate CLI, including its ZIP.""" +import csv +import json +import os +from pathlib import Path +import re +import shutil +import subprocess +import tempfile +import unittest +import zipfile + +ROOT = Path(__file__).resolve().parent.parent +SKILL = ROOT / "fastskill" +CLI = os.environ.get("FASTSKILL_BIN", "fastskill") + + +def cli(cwd, *args): + return subprocess.run([CLI, *args], cwd=cwd, capture_output=True, text=True) + + +class EvalAuthoringTests(unittest.TestCase): + def assert_valid(self, project, cases): + result = cli(project, "eval", "validate", "--json") + self.assertEqual(result.returncode, 0, result.stdout + result.stderr) + data = json.loads(result.stdout) + self.assertTrue(data["valid"]) + self.assertEqual(data["case_count"], cases) + return data + + def test_packaged_example_validates_without_repository_files(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + archive = root / "fastskill.zip" + # Matches publish-skill.yml: only fastskill/ contents, at ZIP root. + with zipfile.ZipFile(archive, "w") as output: + for path in SKILL.rglob("*"): + if path.is_file() and "results" not in path.relative_to(SKILL).parts: + output.write(path, path.relative_to(SKILL)) + with zipfile.ZipFile(archive) as packaged: + packaged.extractall(root / "unpacked") + self.assertNotIn("evals/v2/patterns.json", packaged.namelist()) + self.assertNotIn("evals/authoring/fixtures/defective-total/ground-truth.md", + packaged.namelist()) + data = self.assert_valid(root / "unpacked/examples/invoice-extraction", 3) + self.assertEqual(data["judges"], ["extraction-correctness"]) + for document in (root / "unpacked/references").rglob("*.md"): + for target in re.findall(r"\]\(([^)]+)\)", document.read_text()): + if "://" not in target and not target.startswith("#"): + self.assertTrue((document.parent / target.split("#")[0]).exists(), + f"broken packaged link: {document.name}: {target}") + + def test_contradictory_trigger_is_rejected_and_source_preserved(self): + source = SKILL / "examples/invoice-extraction" + original = (source / "evals/checks.toml").read_bytes() + with tempfile.TemporaryDirectory() as directory: + project = Path(directory) / "project" + shutil.copytree(source, project) + (project / "evals/checks.toml").write_text( + '[[check]]\nname = "skill_invoked"\nexpected = false\n' + 'cases = ["invoice-basic"]\n') + result = cli(project, "eval", "validate", "--json") + self.assertNotEqual(result.returncode, 0) + self.assertIn("EVAL_CHECKS_INVALID", result.stdout + result.stderr) + self.assertIn("invoice-basic", result.stdout + result.stderr) + self.assertEqual((source / "evals/checks.toml").read_bytes(), original) + + def test_invoice_expected_facts_are_independent_of_target_prompt(self): + example = SKILL / "examples/invoice-extraction" + with (example / "evals/prompts.csv").open(newline="") as source: + cases = list(csv.DictReader(source)) + self.assertEqual([row["should_trigger"] for row in cases], ["true", "true", "false"]) + for row, filename in zip(cases[:2], ["basic.txt", "missing-number.txt"]): + expected = json.loads(row["expected"]) + fixture = (example / "fixtures" / filename).read_text() + self.assertIn(expected["currency"], fixture) + self.assertIn(f'{expected["total"]:.2f}', fixture) + self.assertNotIn(str(expected["total"]), row["prompt"]) + self.assertIsNone(json.loads(cases[1]["expected"])["invoice_number"]) + + def test_existing_suite_fixture_validates_and_keeps_user_note(self): + source = ROOT / "evals/authoring/fixtures/existing-suite" + note = (source / "evals/user-note.md").read_bytes() + with tempfile.TemporaryDirectory() as directory: + project = Path(directory) / "project" + shutil.copytree(source, project) + self.assert_valid(project, 1) + self.assertEqual((project / "evals/user-note.md").read_bytes(), note) + + +if __name__ == "__main__": + unittest.main() diff --git a/scripts/test_eval_scripts.py b/scripts/test_eval_scripts.py new file mode 100644 index 0000000..5767492 --- /dev/null +++ b/scripts/test_eval_scripts.py @@ -0,0 +1,111 @@ +"""Exercise preservation and runner-error behavior without model calls.""" +import os +from pathlib import Path +import subprocess +import tempfile +import unittest + +ROOT = Path(__file__).resolve().parent.parent +STAGE = ROOT / "evals/v2/stage.sh" + + +def shell(script, *args): + coverage = os.environ.get("KCOV") + if coverage: + output = tempfile.mkdtemp(prefix="shell-", dir=os.environ["KCOV_OUTPUT"]) + return [coverage, "--bash-method=DEBUG", "--configure=bash-use-basic-parser=1", + "--exclude-region=import sys:PY", + "--include-pattern=/evals/v2/", output, str(script), *args] + return ["bash", str(script), *args] + + +class EvalScriptsTests(unittest.TestCase): + def test_staging_preserves_existing_directory_and_symlink(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + existing = root / "existing" + existing.mkdir() + note = existing / "user-note" + note.write_text("keep this") + link = root / "link" + link.symlink_to(existing, target_is_directory=True) + for dest in (existing, link): + result = subprocess.run(shell(STAGE, "consultation", str(dest)), + capture_output=True, text=True) + self.assertNotEqual(result.returncode, 0) + self.assertIn("already exists", result.stderr) + self.assertEqual(note.read_text(), "keep this") + self.assertTrue(link.is_symlink()) + + def test_staged_project_keeps_payload_and_judge_prompt(self): + with tempfile.TemporaryDirectory() as directory: + dest = Path(directory) / "fresh" + subprocess.run(shell(STAGE, "correctness", str(dest), "1"), check=True) + self.assertEqual((dest / "fastskill/SKILL.md").read_bytes(), + (ROOT / "fastskill/SKILL.md").read_bytes()) + self.assertTrue((dest / "evals/judge-prompt.md").is_file()) + self.assertTrue((dest / "fastskill/references/eval-authoring.md").is_file()) + + def test_subset_runner_failure_is_nonzero_and_preserves_log(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + executable = root / "fastskill" + executable.write_text("#!/bin/sh\necho deliberate-runtime-failure >&2\nexit 23\n") + executable.chmod(0o755) + env = dict(os.environ, PATH=str(root) + os.pathsep + os.environ["PATH"]) + # kcov 38 suppresses output for this unconditional failing child. + # Exercise this assertion directly; the full-sweep test instruments + # the same runner error path with the controllable stub below. + result = subprocess.run(["bash", str(ROOT / "evals/v2/run.sh"), "claude", + str(root / "out"), "1", "consultation"], + env=env, capture_output=True, text=True) + self.assertNotEqual(result.returncode, 0) + self.assertIn("RUNNER ERROR", result.stdout, result.stderr) + self.assertIn("deliberate-runtime-failure", + (root / "out/consultation.log").read_text()) + + def test_unknown_suite_and_missing_judge_credentials_fail_before_runtime(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + result = subprocess.run(shell(STAGE, "../", str(root / "stage")), + capture_output=True, text=True) + self.assertNotEqual(result.returncode, 0) + self.assertFalse((root / "stage").exists()) + env = {k: v for k, v in os.environ.items() if k not in ("AIKIT_LLM_URL", "JUDGE_API_KEY")} + for suites, expected in [(["unknown"], "no such suite"), (["correctness"], "endpoint")]: + result = subprocess.run(shell(ROOT / "evals/v2/run.sh", "claude", + str(root / "out"), "1", *suites), + env=env, capture_output=True, text=True) + self.assertNotEqual(result.returncode, 0) + self.assertIn(expected, result.stdout + result.stderr) + + def test_sweeps_preserve_stages_and_propagate_runner_and_scorecard_verdicts(self): + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + executable = root / "fastskill" + executable.write_text( + '#!/bin/sh\n' + 'if [ "$2" = scorecard ]; then exit "${SCORE_EXIT:-0}"; fi\n' + 'while [ "$#" -gt 0 ]; do\n' + ' if [ "$1" = --output-dir ]; then shift; mkdir -p "$1"; fi\n' + ' shift\n' + 'done\n' + 'echo "1/1 passed"\nexit "${RUN_EXIT:-0}"\n') + executable.chmod(0o755) + env = dict(os.environ, PATH=str(root) + os.pathsep + os.environ["PATH"], + AIKIT_LLM_URL="http://unused.invalid/v1", JUDGE_API_KEY="test-only", + JUDGE_MODEL="test-only") + for index, (suites, run_exit, score_exit) in enumerate([ + (["consultation"], "0", "0"), ([], "0", "0"), + ([], "23", "0"), ([], "0", "9")]): + out = root / f"out-{index}" + result = subprocess.run(shell(ROOT / "evals/v2/run.sh", "claude", str(out), + "1", *suites), capture_output=True, text=True, + env=dict(env, RUN_EXIT=run_exit, SCORE_EXIT=score_exit)) + self.assertEqual(result.returncode == 0, run_exit == score_exit == "0", + result.stdout + result.stderr) + self.assertTrue(list(out.glob(".stage-*/consultation/fastskill/SKILL.md"))) + + +if __name__ == "__main__": + unittest.main() diff --git a/specs/002-eval-authoring-consolidation.md b/specs/002-eval-authoring-consolidation.md new file mode 100644 index 0000000..d39b860 --- /dev/null +++ b/specs/002-eval-authoring-consolidation.md @@ -0,0 +1,130 @@ +# Eval authoring ownership and existing harness assessment + +The installable FastSkill skill lives in `fastskill/`. Eval authoring is a workflow +of that skill: `fastskill/SKILL.md` routes to `references/eval-authoring.md`, the +supporting references, and a copyable `examples/invoice-extraction/` project. +There is no second top-level authoring skill or dependency on a CLI source checkout. + +## Assessment of the existing evals directory + +`evals/` is useful repository test infrastructure, not a competing authoring skill. +Keep it outside the released ZIP so its answers and regression fixtures are not +part of the payload being evaluated. + +| Existing asset | Assessment and disposition | +| --- | --- | +| v1 prompts/checks and pass/fail fixtures | Keep: fast offline parser/scorer regression coverage. Fixture scoring is engine evidence, not a live quality baseline. | +| v2 patterns, generator, consultation/restraint/correctness suites | Keep: explicit case scope and separate metrics avoid conflating consultation with correct answers. Generation and namespace checks pass. | +| vacuity guard | Keep: scans the full packaged payload for answer-pattern leakage. Recheck after adding references/examples. It cannot prove semantic correctness or all forms of leakage. | +| correctness judge and reference answers | Keep: semantic review can catch invalid commands that substring checks accept. Its thresholds are advisory; parsing does not establish judge quality. | +| historical pi baseline | Keep as historical evidence only. It predates the current check/command changes and must not be presented as a current baseline. No full paid sweep was repeated for this relocation. | +| stage.sh | Repair: unconditional deletion of an arbitrary destination could destroy user files or a saved run's project. Refuse existing destinations; each sweep now allocates a fresh stage. | +| run.sh | Repair: subset runtime failures were logged but exited successfully. Propagate the error after retaining results/logs. | +| negctl.sh | Retain as a diagnostic with limitations: it assumes one common skill path across trials and rewrites a synthetic summary. Its output is a mutation control, not an original live run. It does not independently assert the tool-budget metric stayed unchanged. | +| CI and contributor guidance | Repair stale claims that v2 is never validated and Codex cannot supply consultation evidence. Preserve dispatch-only live runs. | +| packaged eval reference | Repair contradictory empty-check scoring and credential guidance. Keep it as the schema reference, linked from authoring. | + +The old authoring assets in gofastskill/fastskill PR #333 move here. Their workflow +fixtures live in `evals/authoring/fixtures/`; executable asset tests live in +`scripts/test_eval_authoring.py`. CLI engine contract tests stay in the CLI repository. + +## Verification boundaries + +The earlier Claude/Codex authoring, continuation, defect/correction, and native-judge +contrasts remain evidence for the carried-over workflow. They do not prove the new +packaging/discovery path. That path must be checked by extracting the release ZIP, +following its local references, validating its example, and exercising installation. +The CLI report retains the original raw-evidence locations and historical results. + +Run on Linux with the candidate FastSkill binary on PATH: + +```bash +python3 -m unittest discover -s scripts -p 'test_*.py' +python3 scripts/check-command-namespaces.py +python3 evals/v2/build.py +git diff --exit-code -- evals/v2/consultation evals/v2/restraint evals/v2/correctness +python3 evals/v2/guard_vacuity.py +``` + +These checks make no model calls. Actual native judge calls require explicit endpoint, +model, and credential configuration; the shipped example retains placeholders. + +## Observed relocation checks + +On server02 with pinned FastSkill 0.9.230: + +- All 12 Python tests passed (four authoring, five shell, three namespace tests). + Namespace checks and regeneration parity passed; the vacuity guard scanned 15 + shipped payload files without finding any of its 12 answer patterns. +- The published ZIP layout installed successfully as FastSkill skill 2.1.0 in a temporary + project. The installed entrypoint and authoring reference were present, and its copied + invoice example validated (three cases, one check, one judge). +- All preserved v2 suites validated: consultation 22 cases, restraint eight cases, and + correctness 12 cases with its judge. The v1 positive fixture passed and negative fixture + failed for missing consultation. No new full live baseline is claimed. +- Six CLI engine contract tests and 13 CLI documentation/related tests passed after the move. +- kcov 38 measured `run.sh` at 54/58 executable Bash lines (93.1%) and `stage.sh` at 19/19 + Bash lines (100%). The four runner continuation lines are not attributed by the collector. + The exact embedded Python body was extracted unchanged and measured separately with + coverage.py: five statements covered, 100%, no branches. It is excluded from the Bash + denominator. Bash branch coverage is unsupported by this collector, not claimed as passing. +- The five shell tests pass with instrumentation; the unconditional failing-child test runs + directly because kcov 38 suppresses its output. The controllable full-run stub separately + exercises the instrumented runtime-error path. Raw coverage is in the sibling CLI evidence + directory under `shell-coverage-v2/` and `embedded-coverage-6f4x9rtk/`. + +The original two-authoring-tool evidence remains in the CLI report. A scoped Claude +Sonnet 4.5 follow-up used the explicitly supplied installed entrypoint with file tools +only. It extended an existing suite with an unrelated negative-trigger case and wrote +`REVIEW.md` describing limitations. Independent caller validation passed for both cases; +the original positive row, checks, fixtures, user note, and installed skill were preserved. +This verifies installed-package routing and suite extension, not automatic discovery or +live target execution. Evidence: `relocation-authoring-result.json` and +`package-smoke-hHkVYA/` in the sibling CLI evidence directory. + +## Changed-file coverage inventory + +| File | Evidence | +| --- | --- | +| `.github/workflows/skill-evals.yml` | Same deterministic test command executed locally; PR CI result reported separately | +| `CONTRIBUTING.md` | Commands/links reviewed against actual workflow and validation output | +| `README.md` | Commands/links reviewed against actual workflow and validation output | +| `evals/authoring/fixtures/defective-total/SKILL.md` | Byte-identical relocation; original workflow/fixture evidence retained in CLI report | +| `evals/authoring/fixtures/defective-total/ground-truth.md` | Byte-identical relocation; original workflow/fixture evidence retained in CLI report | +| `evals/authoring/fixtures/environment-variants.md` | Byte-identical relocation; original workflow/fixture evidence retained in CLI report | +| `evals/authoring/fixtures/existing-suite/SKILL.md` | Byte-identical relocation; original workflow/fixture evidence retained in CLI report | +| `evals/authoring/fixtures/existing-suite/evals/checks.toml` | Byte-identical relocation; original workflow/fixture evidence retained in CLI report | +| `evals/authoring/fixtures/existing-suite/evals/prompts.csv` | Byte-identical relocation; original workflow/fixture evidence retained in CLI report | +| `evals/authoring/fixtures/existing-suite/evals/user-note.md` | Byte-identical relocation; original workflow/fixture evidence retained in CLI report | +| `evals/authoring/fixtures/existing-suite/fixtures/release.md` | Byte-identical relocation; original workflow/fixture evidence retained in CLI report | +| `evals/authoring/fixtures/existing-suite/skill-project.toml` | Byte-identical relocation; original workflow/fixture evidence retained in CLI report | +| `evals/authoring/fixtures/invoice-no-evals/SKILL.md` | Byte-identical relocation; original workflow/fixture evidence retained in CLI report | +| `evals/authoring/fixtures/invoice-no-evals/fixtures/ambiguous-date.txt` | Byte-identical relocation; original workflow/fixture evidence retained in CLI report | +| `evals/authoring/fixtures/invoice-no-evals/fixtures/basic.txt` | Byte-identical relocation; original workflow/fixture evidence retained in CLI report | +| `evals/authoring/fixtures/invoice-no-evals/fixtures/missing-number.txt` | Byte-identical relocation; original workflow/fixture evidence retained in CLI report | +| `evals/authoring/fixtures/invoice-no-evals/fixtures/multiple-currencies.txt` | Byte-identical relocation; original workflow/fixture evidence retained in CLI report | +| `evals/authoring/fixtures/scripted-workflow.md` | Byte-identical relocation; original workflow/fixture evidence retained in CLI report | +| `evals/authoring/fixtures/unnecessary-step/SKILL.md` | Byte-identical relocation; original workflow/fixture evidence retained in CLI report | +| `evals/authoring/fixtures/unnecessary-step/scripted-answers.md` | Byte-identical relocation; original workflow/fixture evidence retained in CLI report | +| `evals/authoring/fixtures/unsupported-order/SKILL.md` | Byte-identical relocation; original workflow/fixture evidence retained in CLI report | +| `evals/v2/README.md` | Commands/links reviewed against actual workflow and validation output | +| `evals/v2/run.sh` | Five shell scenarios; kcov 93.1% lines; runtime failures propagate | +| `evals/v2/stage.sh` | Existing directory/symlink preservation, fresh staging; Bash and embedded Python 100% lines | +| `fastskill/SKILL.md` | Packaged local-link checks; namespace validation; installed-package authoring smoke | +| `fastskill/examples/invoice-extraction/SKILL.md` | ZIP extraction and installation; copied example validation; independent invoice-fact checks | +| `fastskill/examples/invoice-extraction/evals/checks.toml` | ZIP extraction and installation; copied example validation; independent invoice-fact checks | +| `fastskill/examples/invoice-extraction/evals/judge-prompt.md` | ZIP extraction and installation; copied example validation; independent invoice-fact checks | +| `fastskill/examples/invoice-extraction/evals/prompts.csv` | ZIP extraction and installation; copied example validation; independent invoice-fact checks | +| `fastskill/examples/invoice-extraction/evals/review-notes.md` | ZIP extraction and installation; copied example validation; independent invoice-fact checks | +| `fastskill/examples/invoice-extraction/fixtures/basic.txt` | ZIP extraction and installation; copied example validation; independent invoice-fact checks | +| `fastskill/examples/invoice-extraction/fixtures/missing-number.txt` | ZIP extraction and installation; copied example validation; independent invoice-fact checks | +| `fastskill/examples/invoice-extraction/skill-project.toml` | ZIP extraction and installation; copied example validation; independent invoice-fact checks | +| `fastskill/references/eval-authoring.md` | Packaged local-link checks; namespace validation; installed-package authoring smoke | +| `fastskill/references/eval-authoring/examples.md` | Packaged local-link checks; namespace validation; installed-package authoring smoke | +| `fastskill/references/eval-authoring/handoff.md` | Packaged local-link checks; namespace validation; installed-package authoring smoke | +| `fastskill/references/eval-authoring/semantics.md` | Packaged local-link checks; namespace validation; installed-package authoring smoke | +| `fastskill/references/eval.md` | Packaged local-link checks; namespace validation; installed-package authoring smoke | +| `fastskill/skill-project.toml` | Version parity and real ZIP installation as 2.1.0 | +| `scripts/test_eval_authoring.py` | Unittest discovery and execution; deliberate contradiction/runtime failures detected | +| `scripts/test_eval_scripts.py` | Unittest discovery and execution; deliberate contradiction/runtime failures detected | +| `specs/002-eval-authoring-consolidation.md` | Commands/links reviewed against actual workflow and validation output |