feat(skill): integrate eval authoring into FastSkill package

.github/workflows/skill-evals.yml

@@ -7,9 +7,7 @@ on:
       - 'README.md'
       - 'CONTRIBUTING.md'
       - 'CLAUDE.md'
-      - 'fastskill/SKILL.md'
-      - 'fastskill/skill-project.toml'
-      - 'fastskill/references/eval.md'
+      - 'fastskill/**'
       - 'evals/**'
       - 'scripts/**'
       - 'specs/**'
@@ -26,11 +24,8 @@ on:
       agent:
         description: 'Agent CLI to run live evals with'
         type: choice
-        # claude is the default because checks.toml's required `skill_invoked`
-        # check needs structured Skill tool-use events in the trace, which the
-        # codex backend never emits (codex has no Skill tool concept). A codex
-        # live run will always fail the suite; the option remains only for
-        # diagnostic runs of the non-trigger checks.
+        # Both installed runtimes can supply consultation evidence. Check the
+        # installed FastSkill version's runtime observability before a sweep.
         options: [claude, codex]
         default: claude
       trials:
@@ -63,6 +58,8 @@ jobs:
         run: |
           cd fastskill
           fastskill eval validate
+      - name: Validate authoring assets and packaged example
+        run: python3 -m unittest scripts/test_eval_authoring.py scripts/test_eval_scripts.py
 
   # ----------------------------------------------------------------------------
   # Deterministic, always-on: the spec-001 v2 suites. `validate` above reads the

CONTRIBUTING.md

@@ -15,16 +15,18 @@ gofastskill/skill/
 └── fastskill/                    # Shippable skill
     ├── SKILL.md                  # Skill documentation
     ├── skill-project.toml        # Metadata + [tool.fastskill.eval] → ../evals/*
-    └── references/               # eval.md and other reference material
+    ├── references/               # eval.md, eval-authoring.md and supporting references
+    └── examples/                 # Copyable authoring projects included in the ZIP
 ```
 
-There is a single `skill-project.toml` (inside `fastskill/`). Run FastSkill commands
+The main `skill-project.toml` is inside `fastskill/`; copyable examples have their own
+manifests. Run the skill's regression commands
 from the `fastskill/` directory so path resolution finds that manifest and `../evals/`.
 
 ## Development
 
 1. Edit files in the `fastskill/` subdirectory (and `evals/` when extending the suite)
-2. Run `python3 -m unittest scripts/test_command_namespaces.py` and
+2. Run `python3 -m unittest discover -s scripts -p 'test_*.py'` and
    `python3 scripts/check-command-namespaces.py` from the repository root
 3. Open a pull request from a branch off the latest `origin/main` (see `CLAUDE.md`) —
    changes land via PR with required status checks, not by pushing to `main` directly
@@ -99,16 +101,18 @@ a check that contradicts the column is rejected rather than silently resolved. S
 
 - **namespace syntax**, **validate**, and **score-fixtures** run on every push/PR — deterministic,
   no agent, no tokens.
-- **live-eval** is opt-in via *Run workflow* (`workflow_dispatch`), runs the v1 suite against a
-  real agent, and needs `OPENAI_API_KEY` / `ANTHROPIC_API_KEY` secrets.
-
-No CI job runs `evals/v2/` — at roughly $6 and an hour per sweep it belongs on a schedule or a
-manual trigger, and nothing enforces its thresholds yet.
+- **validate-v2** regenerates and validates the three v2 suites and runs the vacuity guard
+  on every PR, without model calls.
+- **live-eval** is dispatch-only and runs the three v2 suites in parallel over Tailscale.
+  It needs the gateway and Tailscale secrets named in the workflow; **scorecard** then applies
+  the configured thresholds. It is not scheduled and does not run on a PR.
+- Authoring assets and the packaged example are checked by `scripts/test_eval_authoring.py`.
+  Authoring test scenarios live under `evals/authoring/fixtures/`, outside the skill ZIP.
 
 `.github/workflows/publish-skill.yml`:
 
 - Detects changes to `fastskill/` directory
-- Packages skill using FastSkill CLI
+- Packages the complete `fastskill/` directory as a ZIP
 - Creates a GitHub release with the packaged skill artifact
 - Uses GitHub App token for release operations
 

README.md

@@ -17,6 +17,11 @@ Read the installed skill with `fastskill skill read fastskill`. The skill docume
 canonical command namespaces introduced by the breaking CLI migration; use a matching
 FastSkill release.
 
+The installed skill includes [guided eval authoring](fastskill/references/eval-authoring.md)
+and a [copyable invoice example](fastskill/examples/invoice-extraction/).
+Repository-only [evals](evals/README.md) measure the FastSkill skill itself; they are not
+included in the released skill ZIP. See the [assessment and ownership record](specs/002-eval-authoring-consolidation.md).
+
 See [CONTRIBUTING.md](CONTRIBUTING.md) for development and release details.
 
 ## License

evals/authoring/fixtures/defective-total/SKILL.md

@@ -0,0 +1,8 @@
+---
+name: defective-total
+description: Add line items from a text invoice and return the numeric total.
+---
+
+# Total an invoice
+
+Add every line item, but deliberately omit the final line item. This defect is intentional test ground truth; a correctly authored eval must detect it rather than adopt it as the expected outcome.

evals/authoring/fixtures/defective-total/ground-truth.md

@@ -0,0 +1,3 @@
+# Test-owned ground truth
+
+For inputs `10.00`, `5.00`, and `2.00`, the independently calculated total is `17.00`. The target instruction deliberately produces `15.00` by omitting the final item. This file is not copied into the target execution fixture.

evals/authoring/fixtures/environment-variants.md

@@ -0,0 +1,9 @@
+# Controlled prerequisite variants
+
+Use these deterministic branches in workflow exercises:
+
+- incompatible CLI: stub `fastskill --version` as `0.8.0`; retain files and mark compatibility unverified;
+- missing runtime: select an unavailable runtime and retain the parse-valid suite;
+- missing judge credential: unset the declared `api_key_env`; do not treat authoring-agent authentication as judge authentication;
+- invalid reference: point `checks` at malformed TOML, capture validation failure, repair it, and rerun validation;
+- unavailable observation: select a backend that preflight identifies as unable to observe a required check; report exclusion/error rather than pass.

evals/authoring/fixtures/existing-suite/SKILL.md

@@ -0,0 +1,8 @@
+---
+name: existing-suite
+description: Summarize release notes into concise bullets.
+---
+
+# Summarize release notes
+
+Preserve version numbers and breaking-change warnings.

evals/authoring/fixtures/existing-suite/evals/checks.toml

@@ -0,0 +1,4 @@
+[[check]]
+name = "command_contains"
+pattern = "BREAKING"
+cases = ["release-basic"]

evals/authoring/fixtures/existing-suite/evals/prompts.csv

@@ -0,0 +1,2 @@
+id,prompt,should_trigger,tags
+release-basic,Summarize fixtures/release.md,true,smoke

evals/authoring/fixtures/existing-suite/evals/user-note.md

@@ -0,0 +1,3 @@
+# User note
+
+Keep this unrelated note byte-for-byte when refining the suite.

evals/authoring/fixtures/existing-suite/fixtures/release.md

@@ -0,0 +1,3 @@
+Version 2.0.0
+
+BREAKING: the legacy endpoint was removed.

evals/authoring/fixtures/existing-suite/skill-project.toml

@@ -0,0 +1,3 @@
+[tool.fastskill.eval]
+prompts = "evals/prompts.csv"
+checks = "evals/checks.toml"

evals/authoring/fixtures/invoice-no-evals/SKILL.md

@@ -0,0 +1,8 @@
+---
+name: invoice-no-evals
+description: Extract known invoice fields into JSON when asked to process a plain-text invoice.
+---
+
+# Invoice extraction
+
+Return `invoice_number`, `date`, `currency`, and `total`. If the invoice number is absent, use JSON null. An ambiguous numeric date must be returned as `ambiguous` rather than guessed. Do not activate for general questions about invoices.

evals/authoring/fixtures/invoice-no-evals/fixtures/ambiguous-date.txt

@@ -0,0 +1,4 @@
+Invoice: INV-8
+Date: 03/04/2026
+Currency: GBP
+Total: 90.00

evals/authoring/fixtures/invoice-no-evals/fixtures/basic.txt

@@ -0,0 +1,4 @@
+Invoice: INV-7
+Date: 2026-09-12
+Currency: AED
+Total: 250.00

evals/authoring/fixtures/invoice-no-evals/fixtures/missing-number.txt

@@ -0,0 +1,3 @@
+Date: 2026-09-10
+Currency: USD
+Total: 18.25

evals/authoring/fixtures/invoice-no-evals/fixtures/multiple-currencies.txt

@@ -0,0 +1,5 @@
+Invoice: INV-9
+Date: 2026-09-09
+Line item: USD 10.00
+Tax: EUR 2.00
+Total: USD 12.00

evals/authoring/fixtures/scripted-workflow.md

@@ -0,0 +1,12 @@
+# Scripted author input for WF1-WF8
+
+1. Start with `invoice-no-evals` and ask for evaluation tests.
+2. Confirm that a missing invoice number must be JSON null.
+3. Confirm that `03/04/2026` is ambiguous and must not be guessed.
+4. Review coverage for ordinary, missing-number, ambiguous-date, multiple-currency, and unrelated cases.
+5. Require outcome correctness for extracted fields; require consultation for extraction requests and non-consultation for unrelated requests.
+6. Authorize six cases, one trial each, only after the authoring agent names the selected runtime and model ID.
+7. Seed an ambiguous-date target failure and retain it as evidence.
+8. Clarify the date criterion and rerun or rejudge only the affected evidence as supported.
+
+The authoring agent must preserve an existing suite's `evals/user-note.md`, report the unsupported ordered-call assertion without substitution, and hand off through files.

evals/authoring/fixtures/unnecessary-step/SKILL.md

@@ -0,0 +1,8 @@
+---
+name: unnecessary-step
+description: Convert temperatures between Celsius and Fahrenheit.
+---
+
+# Convert temperatures
+
+Write an intermediate value to `scratch-temperature.txt`, then calculate the requested conversion and return the answer.

evals/authoring/fixtures/unnecessary-step/scripted-answers.md

@@ -0,0 +1,5 @@
+# Scripted author decisions
+
+- The correct converted temperature is mandatory.
+- Writing `scratch-temperature.txt` is not mandatory and must be treated as advisory adherence, not outcome correctness.
+- If the instruction conflicts with the desired clean workspace, outcome correctness wins and the conflict must be reported.

evals/authoring/fixtures/unsupported-order/SKILL.md

@@ -0,0 +1,8 @@
+---
+name: unsupported-order
+description: Extract text from scanned receipts and normalize their currency values.
+---
+
+# Process receipts
+
+Call OCR first and currency normalization second. The author requires this exact tool-call order, which the FastSkill 0.9.230 deterministic checks cannot represent faithfully.

evals/v2/README.md

@@ -55,6 +55,11 @@ CI runs the three suites on parallel runners for the wall clock — 42 cases at
 
 which is also how to re-fold a sweep already on disk without re-running it.
 
+Each invocation stages into a fresh `.stage-*` directory under the output root. Keep that
+directory while retaining run artifacts: saved judging resolves the original staged project.
+`stage.sh` refuses an existing destination, and `run.sh` exits nonzero on runtime errors
+even when only a subset was selected. Model-quality failures remain reportable via `--no-fail`.
+
 ## Three suites, not fourteen
 
 Correctness used to be twelve single-case suites. A suite's checks applied to all of its

evals/v2/run.sh

@@ -54,9 +54,7 @@ fi
 # one RUNNER ERROR per suite, then EVAL_SCORECARD_NO_RUNS from a scorecard with nothing to read.
 mkdir -p "$OUT"
 OUT="$(cd "$OUT" && pwd)"
-STAGE="$OUT/.stage"
-
-mkdir -p "$STAGE"
+STAGE="$(mktemp -d "$OUT/.stage-XXXXXX")"
 
 # The guard is a precondition, not a formality: without it the numbers are meaningless.
 echo "=== vacuity guard ==="
@@ -79,6 +77,7 @@ if ((${#judged[@]})); then
 fi
 
 echo "=== running ${#suites[@]} suite(s) against '$AGENT', $TRIALS trial(s) per case ==="
+runner_errors=0
 for suite in "${suites[@]}"; do
   skill_dir="$("$HERE/stage.sh" "$suite" "$STAGE/$suite" "$TRIALS" | tail -1)"
 
@@ -98,14 +97,19 @@ for suite in "${suites[@]}"; do
     tail -3 "$OUT/$suite.log" | grep -oE '[0-9]+/[0-9]+ passed' | tail -1 || echo "done"
   else
     echo "RUNNER ERROR (see $OUT/$suite.log)"
+    runner_errors=$((runner_errors + 1))
   fi
 done
 
 if ((score_at_end)); then
   echo
-  exec "$HERE/scorecard.sh" "$OUT"
+  score_status=0
+  "$HERE/scorecard.sh" "$OUT" || score_status=$?
+  if ((runner_errors)); then exit 1; fi
+  exit "$score_status"
 fi
 
 echo
 echo "=== ran ${#suites[@]} of ${#all_suites[@]} suite(s); not scoring a subset ==="
 echo "fold this with the others: $HERE/scorecard.sh <dir containing all three>"
+if ((runner_errors)); then exit 1; fi

evals/v2/stage.sh

@@ -20,13 +20,20 @@ TRIALS="${3:-5}"
 HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 SKILL_SRC="$(cd "$HERE/../.." && pwd)/fastskill"
 
-if [[ ! -d "$HERE/$SUITE" ]]; then
+case "$SUITE" in consultation|restraint|correctness) ;; *)
   echo "stage.sh: no suite '$SUITE' at $HERE/$SUITE" >&2
   exit 1
-fi
+esac
 
-rm -rf "$DEST"
-mkdir -p "$DEST/evals"
+# Refuse existing paths, including dangling symlinks. Staging must never erase
+# user files or overwrite a previous run's project, which saved judging needs.
+if [[ -e "$DEST" || -L "$DEST" ]]; then
+  echo "stage.sh: destination already exists: $DEST; choose a fresh directory" >&2
+  exit 1
+fi
+mkdir -p -- "$(dirname -- "$DEST")"
+mkdir -- "$DEST"
+mkdir -- "$DEST/evals"
 cp -r "$SKILL_SRC" "$DEST/fastskill"
 
 # The whole suite directory, not just the two files the engine always reads: a

fastskill/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: fastskill
-version: 2.0.0
-description: Package manager and operational toolkit for Claude Code-compatible skills. Use this skill when installing, managing, discovering, bundling, or analyzing skills; configuring repositories; running skill evaluations (`fastskill eval validate/run/judge/report/score/scorecard`) or the optimization loop (`fastskill optimization`); serving skills over HTTP/MCP; or building marketplace catalogs. See references/eval.md for eval setup in full.
+version: 2.1.0
+description: Package manager and operational toolkit for Claude Code-compatible skills. Use this skill when installing, managing, discovering, bundling, or analyzing skills; configuring repositories; authoring, repairing, or running skill evaluation suites; using the optimization loop; serving skills over HTTP/MCP; or building marketplace catalogs.
 license: Apache-2.0
 ---
 
@@ -159,6 +159,11 @@ pass/fail rules:
 
 - **[Skill evals guide](references/eval.md)** — setup, prompts CSV, `checks.toml`, agents, artifacts, CI.
 
+For creating or repairing a suite, follow [Eval authoring](references/eval-authoring.md).
+It covers coverage decisions, outcome versus adherence checks, judge calibration, authorized
+pilots, and continuation from saved files. The [invoice example](examples/invoice-extraction/)
+is a copyable project; copy it outside the installed skill before editing or running it.
+
 ## Basic usage
 
 ### Command namespaces

fastskill/examples/invoice-extraction/SKILL.md

@@ -0,0 +1,10 @@
+---
+name: invoice-extraction
+description: Extract invoice fields from plain-text invoices into JSON when the user asks for invoice data.
+metadata:
+  id: invoice-extraction
+---
+
+# Extract invoices
+
+Return JSON with `invoice_number`, `date`, `currency`, and `total`. Use JSON null for a missing invoice number. Do not activate for unrelated questions.

fastskill/examples/invoice-extraction/evals/checks.toml

@@ -0,0 +1,22 @@
+[[check]]
+name = "command_contains"
+pattern = "normalize_currency"
+required = false
+cases = ["invoice-basic"]
+
+[[judge]]
+name = "extraction-correctness"
+cases = ["invoice-basic", "invoice-missing-number"]
+prompt_file = "judge-prompt.md"
+model = "replace-with-approved-judge-model"
+
+[[judge.criterion]]
+name = "correct_fields"
+kind = "bool"
+description = "The JSON values agree with the invoice fixture, and a missing invoice number is JSON null."
+
+[judge_defaults]
+base_url = "https://replace-with-approved-endpoint.example/v1"
+api_key_env = "FASTSKILL_EXAMPLE_JUDGE_KEY"
+temperature = 0.0
+max_tokens = 1024

fastskill/examples/invoice-extraction/evals/judge-prompt.md

@@ -0,0 +1,10 @@
+Review the extracted invoice JSON against the task and recorded workspace evidence.
+
+Task: {{case.prompt}}
+Answer: {{trial.final_answer}}
+Expected facts: {{case.expected}}
+
+{{rubric}}
+
+Reply using this contract:
+{{output_contract}}

fastskill/examples/invoice-extraction/evals/prompts.csv

@@ -0,0 +1,4 @@
+id,prompt,should_trigger,tags,workspace_subdir,expected
+invoice-basic,"Extract the invoice in fixtures/basic.txt as JSON.",true,"outcome,positive",,"{""invoice_number"":""INV-1042"",""date"":""2026-09-01"",""currency"":""USD"",""total"":123.45}"
+invoice-missing-number,"Extract the invoice in fixtures/missing-number.txt as JSON. Use null for missing fields.",true,"outcome,boundary",,"{""invoice_number"":null,""date"":""2026-09-02"",""currency"":""EUR"",""total"":78.90}"
+invoice-unrelated,"Explain the difference between a receipt and an invoice.",false,"trigger,negative",,

fastskill/examples/invoice-extraction/evals/review-notes.md

@@ -0,0 +1,11 @@
+# Invoice extraction eval review
+
+Compatibility: authored for FastSkill 0.9.230; validate against another installed release before use.
+
+Coverage: ordinary extraction, missing invoice number, and unrelated-request non-triggering. The author confirmed that a missing number must be JSON null. Test-owned expected facts are supplied through the `expected` CSV column and rendered into the judge prompt. Field correctness is an outcome expectation. Skill consultation/non-consultation is an adherence/selection expectation imposed by `should_trigger` semantics. Currency-normalization command text is advisory only.
+
+Calibration contrasts: JSON null is acceptable for the missing number; the string `"UNKNOWN"` is unacceptable. These labels are based on the confirmed criterion. They have not been executed against a judge in this example.
+
+Unsupported: ordered OCR-then-normalization calls cannot be asserted by the current deterministic check types. No weaker substitute is claimed.
+
+Status: example files are syntax-validated without contacting the placeholder judge endpoint. No runtime pilot or judge calibration is claimed.

fastskill/examples/invoice-extraction/fixtures/basic.txt

@@ -0,0 +1,4 @@
+Invoice number: INV-1042
+Date: 2026-09-01
+Currency: USD
+Total: 123.45

fastskill/examples/invoice-extraction/fixtures/missing-number.txt

@@ -0,0 +1,3 @@
+Date: 2026-09-02
+Currency: EUR
+Total: 78.90

fastskill/examples/invoice-extraction/skill-project.toml

@@ -0,0 +1,10 @@
+[metadata]
+id = "invoice-extraction"
+version = "1.0.0"
+
+[tool.fastskill.eval]
+prompts = "evals/prompts.csv"
+checks = "evals/checks.toml"
+trials_per_case = 1
+pass_threshold = 1.0
+fail_on_missing_agent = true